Most people use LLMs through a website or app. But some models can run entirely on your own computer, with no internet connection and no data leaving your machine.
Two ways to run a model
- Hosted: the model runs in a data center, and you reach it through a chatbot, an app or an API. Services are usually priced by the number of tokens you send and receive.
- Local: you download the model and run it yourself. This is only possible with open-weight models, where the trained model files are publicly available.
Why run locally?
- Privacy: your text stays on your machine.
- Offline use: it works without internet.
- No per-token fees and freedom to experiment.
The trade-off: the biggest, most capable models are far too large for a normal computer, so local models are usually smaller.
Will it fit on my computer?
A simple rule of thumb: at standard 16-bit precision, each billion parameters needs about 2 GB of memory. So a 7-billion-parameter model needs roughly 14 GB, while models with over 100 billion parameters are out of reach for most consumer devices.
The model is held in your RAM, your graphics card’s memory (VRAM), or unified memory on some laptops. If it doesn’t fit in the graphics card, it can be split between the card and system memory, at the cost of speed.
Quantization: making models smaller
Quantization stores the model’s numbers at lower precision, shrinking its size so larger models fit on ordinary hardware. You lose a little quality, but often much less than you save in memory.
Tools that make it easy
- Ollama: download and run models with simple commands.
- LM Studio: a desktop app with a friendly interface.
- llama.cpp: the engine many tools are built on, designed for ordinary CPUs and GPUs.
- MLX: Apple’s framework for Apple silicon Macs.
- Open WebUI: a chat-style web interface for local models.
Try it
- Install LM Studio or Ollama.
- Pick a small model (a few billion parameters, quantized).
- Ask it to summarize a text or draft an email.
- Compare the speed and quality with the hosted assistant you normally use.
Further reading: the Wikipedia article on large language models.