Blog · AI
Running local LLMs on Apple silicon: start with memory
A simple rule of thumb for which models fit on which Mac, and how to leave room for the context window.
On a Mac, the GPU can use most of the unified memory, which is why a Mac Studio can run models that would need several graphics cards elsewhere. The first question is always the same: does the model fit?
A rule of thumb
For a model quantized to 4 bits, plan for roughly 0.5 to 0.6 GB of memory per billion parameters, plus room for the context window (the KV cache) and the system. At 8 bits, roughly double it.
- 7–8B models at 4-bit: comfortable on 16GB.
- 13–14B models at 4-bit: better on 24GB.
- 30–34B models at 4-bit: look at 36GB or more.
- 70B-class models: a Mac Studio with large memory.
These are estimates. Long contexts and several users at once need noticeably more.
Tools that work well
- llama.cpp and Ollama for quick serving with Metal acceleration.
- MLX from Apple for running and fine-tuning models natively.
Serve it to your team
Run the model server on the cloud Mac and reach it over an SSH tunnel or a private network, rather than opening the port to the internet. That keeps prompts and data inside your own setup.
