Pretraining
The first and most expensive stage: learning to predict the next token across a very large body of text.
Also written: pre-training, base model.
A model starts as random numbers and reads an enormous amount of text, adjusting itself each time to predict the next token slightly better. This runs for months on thousands of accelerators and accounts for nearly all of what a model knows.
What comes out is a base model: fluent, knowledgeable, and not useful. It completes text rather than answering questions, and it has no notion that it should be helpful. Everything that makes it feel like an assistant happens afterwards, and cheaply by comparison.