An LLM doesn’t arrive as a helpful assistant. It is built in stages, and each stage shapes how it behaves when you talk to it.
Step 1: Collect and clean text
Training starts with a huge amount of text. Before it’s used, the data is cleaned: low-quality, duplicated and toxic material is removed, which improves results. Sometimes a trained model is even used to help clean data for the next one.
Two growing concerns: the web now contains a lot of AI-written text that can lower quality, and for the biggest models there may not be enough good human-written text, so synthetic data is sometimes used.
Step 2: Turn text into tokens
Models work with numbers, not letters. A tokenizer splits text into pieces and gives each piece a number. A popular method, byte-pair encoding, starts with single characters and repeatedly merges the pairs that appear together most often, until it has a vocabulary of the target size. Common words end up as one token, rare words as several.
Step 3: Pretraining
The model reads enormous amounts of text and practices one task: predict the next token. It makes a guess, compares it with what actually came next, adjusts slightly, and repeats billions of times. This is where it absorbs grammar, facts, writing styles and patterns of reasoning. At this stage it is a very good text continuer, not yet a helpful assistant.
Step 4: Fine-tuning
Next, the model is shaped to behave well:
- Instruction tuning: it learns from examples of questions paired with good answers.
- Human feedback (RLHF): a separate model is trained to predict which answers people prefer, and the LLM is then adjusted to produce answers that score well.
- Some developers also use written principles to guide behavior (an approach called constitutional AI).
Step 5: Make it a chat
A chat is just formatted text. Your message is marked as “User”, the model continues after “Assistant”, and a hidden system prompt at the start sets the overall instructions and role.
Why it costs so much
Training the 1.5-billion-parameter GPT-2 in 2019 cost around $50,000. Training PaLM (540 billion parameters) in 2022 cost around $8 million. Training also costs far more per token than simply answering a question, which is why most people use models that someone else has trained.
What this means for you
- The model’s knowledge comes from pretraining and stops at some point in time.
- Its manners and helpfulness come from fine-tuning, so different assistants built on similar models can feel very different.
- Bad or biased training data can lead to unreliable answers.
Further reading: the Wikipedia article on large language models.