Getting computers to handle human language
Natural language processing, usually shortened to NLP, is the field concerned with having computers understand, analyze, and generate the language people use.
Natural language here means English, Japanese, and other languages people use with each other, as distinct from programming languages such as Python that give a machine precise instructions.
Where it already works
- Understanding what a search query means and finding related information
- Translating automatically between languages
- Classifying text and analyzing sentiment or topic
- Converting speech to text and reading the intent of a question
- Letting a chat assistant understand text and compose an answer
- Extracting summaries and key information from long documents
Spam filtering, predictive text, review analysis, and voice assistants all used NLP well before generative AI became widespread.
NLP, NLU, and NLG
NLP covers the whole field. Within it, natural language understanding (NLU) refers to grasping meaning and intent, and natural language generation (NLG) to producing text a person can read.
A chat assistant reads your question with NLU, does whatever processing is needed, and produces the answer with NLG. In real products these are integrated, so they are usually described together as NLP.
How LLMs fit
An LLM is a large model trained on enormous amounts of text. It is one way of doing natural language processing, and it handles translation, summarization, question answering, and generation within a single model.
NLP is not synonymous with LLMs, though. Rules, statistical methods, and small classification models are all still used where they suit the job. For simple processing, a small model is often faster, cheaper, and easier to explain.
Why language is hard
The same words mean different things in different contexts. Ellipsis, irony, jargon, dialect, and typos all occur, and text that reads naturally can still leave the speaker’s intent ambiguous.
Fluent output does not mean full understanding. For anything important, check proper nouns, figures, negation, and surrounding context, and be careful with personal data and skewed training data.