“Language model” sounds like it only works with words. But the same core idea, predicting what comes next in a sequence, turns out to work on many kinds of data.
Multimodal models
A modality is a type of input or output: text, images, audio, video. Since 2023 many LLMs have become multimodal. GPT-4o, for instance, can process and produce text, audio and images.
How does a text model “see”?
A common recipe:
- Take a trained LLM and a trained image encoder.
- Add a small network that converts an image into vectors the same size as text tokens, effectively “image tokens”.
- Mix image tokens and text tokens in one sequence.
- Fine-tune on pairs of images and text.
To the model, a picture becomes just another kind of token.
Code is a language too
Programming languages are text, so LLMs handle them without special tricks. They can write code from a plain-language description (sometimes called vibe coding), explain code, or translate between languages. They started as code-completion helpers and are moving toward writing whole programs, in tools like GitHub Copilot.
Biology as a language
Proteins are chains of amino acids, and DNA and RNA are sequences of letters. Transformer models can learn patterns in them much like grammar in a sentence.
- Meta’s ESMFold predicts protein structures about an order of magnitude faster than AlphaFold2, and was used to predict 772 million metagenomic structures.
- Language models have also been used to design new proteins unlike any found in nature.
A surprising side effect: compression
Because a good next-token predictor knows what’s likely, it can compress data. In a 2023 study, DeepMind’s Chinchilla, trained mostly on text, shrank image data to 43% of its size, beating PNG’s 58%.
What you can try today
- Upload a photo of a plant, a receipt or a chart and ask what it shows.
- Ask an assistant to explain a screenshot of an error message.
- Describe a small spreadsheet task in plain words and ask for the formula or script.
Further reading: the Wikipedia article on large language models.