How to run local LLMs for coding: hardware, setup, context

To run local LLMs for coding, install Ollama (or LM Studio, or llama.cpp), download an open-weight coding model that fits in your GPU memory, raise the context length to at least 64k tokens, and point your coding agent at http://localhost:11434. Your code never leaves your machine, and there is no per-token bill.

The context step is the one people miss. Ollama’s default is often 4k tokens, too small for an agent, which then forgets your instructions halfway through a task.

What it looks like

On a machine with a 24 GB GPU:

ollama pull qwen3.8:27b
ollama launch claude --model qwen3.8:27b

The second line starts Claude Code with the local model instead of Anthropic’s. You give it a task as usual; every request goes to your GPU. Set the context length first, as shown below, or the agent will run out of room.

What hardware do I need?

Memory decides almost everything. The model has to fit in GPU memory (VRAM), or in unified memory on Apple Silicon Macs. If it spills into normal RAM, it still runs, but much slower.

A useful rule from Ollama’s own download sizes: a 4-bit model needs about 0.7 GB per billion parameters, plus room for the context.

The context needs memory too, and a 64k context can add several gigabytes: about 2 GB for qwen3.5:9b, 4 GB for qwen3.8:27b. In practice:

• CPU only: fine to try things out, too slow for agent work.
• 8 to 12 GB GPU: qwen3.5:9b, short focused tasks. On 8 GB it only fits with a smaller or quantized context.
• 16 GB GPU: qwen3.5:9b-q8_0, the same model at 8 bits, with a full 64k context.
• 24 GB GPU: qwen3.8:27b. Usable for real agent tasks.
• 48 GB or more of unified memory, or several GPUs: room for 8-bit weights, longer contexts and bigger models.

Best Ollama models for coding has the pick for each memory size, with the context memory worked out per model.

Speed matters as much as fit. An agent task takes dozens of turns, so a model that answers slowly makes every task drag.

Which local model is best for coding?

For agent work, the model must support tool calling: that is how it reads files and runs commands. On ollama.com, look for the tools tag on the model page. A model without it can talk about code but can’t act on it.

Current picks, all tagged for tools: qwen3.8:27b for a 24 GB GPU, qwen3.5:9b for 12 to 16 GB, and qwen3-coder:30b if you want short turns without thinking (with a 64k context it needs more than 24 GB). Try your pick on a real task. New open-weight models come out often; best Ollama models for coding keeps the list by GPU size.

How do I set it up with Ollama?

1. Install Ollama from ollama.com for your OS.

2. Pull a model:

   ollama pull qwen3.8:27b

3. Try it directly:

   ollama run qwen3.8:27b "Write a Python function that parses ISO dates"

4. Check the server is up. It listens on port 11434:

   curl http://localhost:11434/api/tags

   This lists the models you have pulled.

5. Raise the context length (next section).

6. Connect your agent (the section after that).

Why does my local agent forget things or stop making sense?

Almost always the context length. Ollama picks a default from your GPU memory: 4k tokens under 24 GB of VRAM, 32k from 24 to 48 GB, and 256k above that. Its own docs say agents and coding tools need at least 64k. The system prompt and tool definitions alone can fill much of 4k before the agent reads any code.

Three ways to raise it:

• Ollama app: move the context length slider in Settings.

• Server setting: on Windows, quit Ollama, run setx OLLAMA_CONTEXT_LENGTH 64000 in PowerShell and start Ollama again. On macOS or Linux: OLLAMA_CONTEXT_LENGTH=64000 ollama serve

• Per model, for agents that use the OpenAI-compatible endpoint, which has no context setting of its own. Create a file named Modelfile:

  FROM qwen3.8:27b
  PARAMETER num_ctx 64000

  Then build a new model name and use that name in your agent:

  ollama create qwen3.8-64k -f Modelfile

A bigger context uses more memory. After a change, run ollama ps while the model is loaded and check the PROCESSOR column. It should say the model runs fully on the GPU.

How do I connect Claude Code, Codex, OpenCode or another agent?

Claude Code

ollama launch claude

Or set it up yourself:

export ANTHROPIC_AUTH_TOKEN=ollama
export ANTHROPIC_API_KEY=""
export ANTHROPIC_BASE_URL=http://localhost:11434
claude --model qwen3.8:27b

In PowerShell on Windows:

$env:ANTHROPIC_AUTH_TOKEN = "ollama"
$env:ANTHROPIC_API_KEY = ""
$env:ANTHROPIC_BASE_URL = "http://localhost:11434"
claude --model qwen3.8:27b

Codex

ollama launch codex

Or: codex --oss -m qwen3.8:27b

OpenCode

Add Ollama as a provider in opencode.json in your project, or in ~/.config/opencode/opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama (local)",
      "options": { "baseURL": "http://localhost:11434/v1" },
      "models": {
        "qwen3.8:27b": { "name": "Qwen3.8 27B" }
      }
    }
  }
}

Then pick the model with /models in OpenCode. Use the name of the 64k model you created above if you made one. OpenCode with local models covers this setup in detail.

Any agent with an OpenAI-compatible setting: base URL http://localhost:11434/v1/. The agent may insist on an API key; any value works, since Ollama ignores it locally.

Contenox: open Backends → Providers, choose Ollama / Ollama Cloud, keep the address http://127.0.0.1:11434 and select Connect. Your pulled models then appear in the Model chip, next to any cloud providers you connected, so you can switch per session. See Ollama in Contenox.

Is a local model good enough?

For some work, yes. Be honest about the rest.

Where local models do well: focused, clearly described tasks. Writing tests for a given function, small refactors, explaining a module, boilerplate.

Where they fall short of the top hosted models:

• Long tasks. Smaller models lose track of instructions over many turns.
• Tool calls. More malformed or repeated calls.
• Hard debugging across many files. This is where the gap is largest.

The hardware is not free either: a GPU with enough memory is a real purchase, and it usually runs slower than a hosted model. Run local models because of privacy, offline work or very high volume, not to save money on light use.

A setup that works for many people: a local model for routine and private work, a hosted one for the hardest problems. See how to choose a model for coding.

Other ways to run models locally

• LM Studio: a desktop app with a model browser and a local server mode. Good if you prefer clicking to typing commands.
• llama.cpp: the engine underneath many local tools. Its llama-server gives you more control over every setting.
• vLLM: for a shared GPU server that several people use at once.

Related guides

• How to choose a model for coding: test a local model against hosted ones on your own tasks
• AI coding costs: when local is actually cheaper
• How to keep secrets away from AI agents: local models help, but the agent can still read your files
• Best Ollama models for coding: the pick for each GPU size
• Best Ollama harness: which agent works best with small local models
• What is an AI coding agent?