The first and largest stage
Pre-training is the initial stage of model building, where a large dataset gives the model broad general capability. For an LLM, that means repeatedly predicting the next word across an enormous corpus, absorbing how language works and a great deal of world knowledge along the way.
What comes after “pre”
A model fresh out of pre-training is knowledgeable but has not learned how to hold a conversation with a person. Finishing stages follow:
- Fine-tuning: additional training for a purpose, teaching dialogue format or a specialist domain
- RLHF: using human feedback to improve helpfulness and safety
Pre-training builds general education; the later stages teach manners. That framing makes the pipeline easy to remember.
The P in GPT
GPT stands for Generative Pre-trained Transformer. Pre-training is central enough to modern AI that it is named in the acronym.
Why this approach won
AI development used to mean training a separate model from scratch for each purpose. Pre-training made it possible to grow one large base and reuse it across many uses, which transformed development efficiency. Such a base is also called a foundation model, and nearly every AI service in the news today is built as a huge pre-trained model plus purpose-specific finishing.
Pre-training also consumes enormous compute, with frontier runs reported to cost hundreds of millions of dollars. That cost is the background to the funding rounds and data center construction that fill AI business coverage.
When you see “pre-trained model”, read it as a model that has finished its general education and only needs finishing for a specific use.