ChatGPT arrived in late 2022 and seemed to come from nowhere. In reality, it was the visible tip of a long road.
Before transformers: counting words
In the 1990s and 2000s, language models were statistical: they counted which words tended to follow which. A well-known 2001 model trained on 300 million words was state of the art for its time. As internet access spread, researchers started using the web itself as a giant pile of training text.
The neural turn
Around 2000, researchers began trying neural networks for language, and the idea took off after deep learning’s breakthrough in image recognition around 2012. Two milestones:
- 2013, word embeddings (Word2Vec): words became lists of numbers, so similar words sit close together.
- 2016, neural machine translation: Google replaced its older translation approach with deep neural networks.
2017: “Attention Is All You Need”
A research paper introduced the transformer, built on the attention mechanism, which lets a model look at the relationships between all the words in a text at once, however far apart they are. Almost every modern LLM is built on it.
2018–2020: two families, one winner
- BERT (2018) learned by filling in missing words. It became extremely popular in research.
- GPT models learned by predicting the next word. GPT-2 (2019) made headlines because its makers first hesitated to release it, fearing misuse. GPT-3 (2020) showed that a model could pick up a task from just a few examples in the prompt.
Over time, GPT-style models got so good at following prompts that research use of BERT-style models declined.
2022–2023: everyone gets one
ChatGPT put a conversational interface on a model fine-tuned to follow instructions, and millions of people tried it. In 2023, GPT-4 added the ability to work with images as well as text.
2024–2025: reasoning and open weights
- Reasoning models such as OpenAI’s o1 were trained to work through problems step by step before answering.
- Open-weight models (LLaMA, Mistral and others) let anyone download and study the model. In January 2025, DeepSeek-R1, a 671-billion-parameter open model, matched o1’s performance at a much lower price per token.
The takeaway
Nothing here happened overnight. Better ideas (attention), more data, more computing power and better fine-tuning each added a piece. And “large” has no official threshold: it simply means much bigger than what was practical before.
Further reading: the Wikipedia article on large language models.