How to Run an LLM Locally: Ollama vs llama.cpp, Hardware Needs and Quantisation Explained

Running a large language model on your own computer used to mean compiling research code and hoping your graphics card was supported. Today two free, open-source tools make it a fairly routine job: Ollama, which packages models behind a simple command and a local API, and llama.cpp, the C/C++ inference engine that Ollama itself builds on. With either one, prompts and answers stay on your machine, and you can keep working without an internet connection once a model is downloaded.

This guide covers what you need, how the two tools differ, how to choose a model size your hardware can actually handle, and the settings worth knowing on day one.

Why run a model locally

There are three common reasons. The first is privacy: Ollama's own documentation says that when you run models locally it does not see your prompts or data. The second is cost and control, because there are no per-token bills and no rate limits beyond your own hardware. The third is availability, since a local model keeps working on a plane or behind a strict firewall.

The trade-off is capability. The models that fit on a laptop are smaller than the largest hosted systems, so they are best for drafting, summarising, coding help, private document search and experimentation rather than the hardest reasoning tasks.

Ollama or llama.cpp?

Ollama lists llama.cpp as a supported backend, so the two are closely related. A good rule is to start with Ollama and move to llama.cpp when you want to fine-tune performance or use a model file Ollama does not package.

Step 1: Check your hardware

Memory is the limit that matters most. A model has to fit in your GPU's video memory, or in system RAM on machines that share memory between CPU and GPU, such as Apple silicon Macs.

GPU support in Ollama:

• Nvidia: cards with compute capability 5.0 or newer and driver version 550 or later. Older cards in the 5.0 to 6.2 range need driver 570 or later.
• AMD: supported Radeon, Radeon PRO, Ryzen AI and Instinct parts through the ROCm library, which needs ROCm v7 drivers on Linux. Vulkan adds support for more GPUs on Windows and Linux.
• Apple: GPU acceleration through Apple's Metal API.

llama.cpp covers similar ground. Its maintainers describe Apple silicon as a first-class target, it has custom CUDA kernels for Nvidia GPUs, supports AMD through HIP, and offers Vulkan and SYCL backends. It can also split a model between CPU and GPU when the model is bigger than your video memory, which is slower but lets larger models run.

You can run on CPU alone. It works, but expect noticeably slower responses.

Step 2: Understand quantisation

Models are usually published with each weight stored as a 16-bit number. Quantisation stores weights with fewer bits, which shrinks the file and memory use at some cost to quality. Most local models are distributed in GGUF, a binary format created by Georgi Gerganov, the developer of llama.cpp, that stores both the model's tensors and standard metadata in one file and is designed for fast loading.

Hugging Face's GGUF documentation lists the common quantisation types and how many bits each uses per weight:

You can estimate the size of the weights with simple arithmetic: number of parameters multiplied by bits per weight, divided by eight, gives bytes. A 7-billion-parameter model at Q4_K's 4.5 bits per weight works out to roughly 3.9 GB. The same model at 16 bits is about 14 GB. On top of that you need extra memory for the context window, the running conversation the model keeps in memory, and for your operating system and other apps.

As a practical starting point, leave a few gigabytes of headroom beyond the file size. If a model will not fit, choose a smaller model before choosing a more aggressive quantisation, because very low-bit versions of a bigger model can perform worse than a well-quantised smaller one.

Step 3: Install and run your first model with Ollama

1. Install. On macOS and Linux, Ollama's official installer is a one-line shell script from ollama.com, or you can download the app manually. Windows has a PowerShell installer and a manual download. An official ollama/ollama Docker image is also available.
2. Run a model. Type ollama run followed by a model name from the Ollama library. The first run downloads the model; after that it loads from disk.
3. Check where it is running. ollama ps lists loaded models, their size and whether they are on the GPU or CPU. If you see CPU when you expected GPU, check the driver requirements above.
4. Use the API. Ollama serves a REST API at http://localhost:11434, with endpoints such as /api/chat and /api/generate. Official Python and JavaScript libraries wrap it, so you can plug a local model into scripts and apps.

Step 4: Or run it with llama.cpp

llama.cpp offers an installer script, prebuilt binaries from its releases page, Docker images, or a source build. Once installed, it can download and run a GGUF model directly from Hugging Face with its command-line tool, and llama serve launches an OpenAI-compatible API server with a built-in web interface. Because many apps already speak the OpenAI API format, that compatibility makes it easy to point existing tools at a local model.

Settings worth knowing

• Context window. Ollama uses a default context window of 4,096 tokens. You can raise it with the OLLAMA_CONTEXT_LENGTH environment variable, with /set parameter num_ctx inside a chat, or with the num_ctx option in an API call. A bigger window lets the model read longer documents but uses more memory.
• Keep-alive. By default Ollama keeps a model in memory for five minutes after use so follow-up requests are fast. ollama stop unloads it immediately, and OLLAMA_KEEP_ALIVE or the keep_alive API parameter changes the timeout.
• Busy servers. If too many requests arrive at once, Ollama returns a 503 error. Its documentation explains how to tune queue limits if you share one machine between several users.
• Local-only mode. Ollama also offers cloud-hosted models. If you want everything to stay on your machine, its documentation describes how to turn off cloud features, after which the logs show that cloud is disabled.

Keep it private and secure

By default Ollama listens only on 127.0.0.1 port 11434, which means only your own computer can reach it. You can change that with the OLLAMA_HOST setting, for example to share a model across a home network, but think before you do. The local API has no login of its own, so anyone who can reach the port can use your model and your hardware. If you need remote access, put it behind a reverse proxy with authentication or a private network, and never expose the port directly to the internet.

Also treat model files like any other software download. Get them from the official Ollama library or from well-known publishers on Hugging Face, and check the model's licence before using it commercially, since open-weight licences vary.

Quick checklist

• Confirm your GPU and driver meet the requirements, or accept slower CPU-only speeds.
• Estimate memory: parameters times bits per weight divided by eight, plus headroom.
• Start with a 4-bit or 5-bit quantised version of a small model.
• Install Ollama for the fastest start; use llama.cpp for deeper control.
• Run ollama ps to confirm the model is on the GPU.
• Raise the context window only as far as your memory allows.
• Keep the API on localhost unless you add authentication.