Ollama and local runners

01 · Concept

Concept

Two people run the same pinned 17.1 GB Qwen3.8-27B Q4_K_M GGUF on the same laptop, ask the same question, and get answers with visibly different temperament — one crisp and literal, the other discursive and prone to inventing detail. Neither edited a config file. The difference is that one of them is running the engine from lesson 8.7 directly with explicit flags, and the other is running it through a convenience layer that supplied sampling parameters on their behalf. This lesson is about that layer: what it genuinely gives you, and the single configuration habit that keeps it from quietly changing your model.

A local runner such as Ollama is not an inference engine. It is a distribution, configuration, and lifecycle layer wrapped around one — in this case the llama.cpp and ggml stack. It contributes three things the engine deliberately does not. First, a registry: models are addressed by name and tag rather than by a URL to a specific file, so pulling a model is one command and the runner resolves which quantization variant to fetch for your hardware. Second, a Modelfile, a small declarative recipe that binds a GGUF to a chat template, a system prompt, stop sequences, and sampling parameters, and can be built into a new named model that inherits from another. Third, a local server: a long-lived process on the loopback interface exposing a chat API, including OpenAI-compatible routes, that loads a model on first request, keeps it resident for a keep-alive interval, and evicts it to make room for another. That last piece is why a runner feels different from the engine — you stop thinking about processes and start thinking about model names.

The cost of that convenience is that defaults now exist where previously there were only your flags, and defaults you did not choose are the ones that bite. Qwen3.8-27B publishes two sampling presets on its model card. Thinking mode, the default behaviour, expects temperature 1.0, top_p 0.95, top_k 20. Instruct mode — the non-thinking path — expects temperature 0.7, top_p 0.80, top_k 20. These are not stylistic suggestions; they are the settings under which the vendor’s reported behaviour was produced. A registry entry ships whatever its packager put in the Modelfile, and there is no mechanism guaranteeing that matches the card. If you deploy an assistant on top of unexamined defaults, you are evaluating a configuration nobody documented.

So pin it. Write a Modelfile that inherits from the pulled tag and states every parameter you care about, then build a named local model and use only that name from your application.

FROM qwen3.8:27b
PARAMETER temperature 0.7
PARAMETER top_p 0.80
PARAMETER top_k 20
PARAMETER min_p 0
PARAMETER repeat_penalty 1.0
PARAMETER presence_penalty 1.5
PARAMETER num_ctx 16384

Build it with ollama create qwen3.8-27b-instruct -f Modelfile and point your client at qwen3.8-27b-instruct. The sampling configuration is now a versioned artifact in your repository rather than a property of whichever machine happens to be serving.

Walk through what each line buys. Alongside temperature, top_p and top_k, Ollama exposes min_p, repeat_penalty and presence_penalty. Pin all six sampling options as shown and verify the effective request and backend, so the configuration follows the card’s instruct preset — and if you want thinking mode instead, you build a second named model with 1.0 and 0.95, rather than toggling values by hand and forgetting. The num_ctx line is the one people omit, and it is the most consequential. Ollama currently defaults to 4K below 24 GiB VRAM, 32K from 24 to 48 GiB, and 256K at 48 GiB or more, against this model’s 262,144-token native context. That default is not laziness: the window is a memory promise, because context length times 64 KiB per token from lesson 7.2 is KV cache the runner must reserve. At 16,384 tokens the cache is 1 GiB. On the lesson’s 24 GiB-class machine that is a deliberate cap below the current 32K default, chosen to leave headroom beside the pinned 17.1 GB Q4_K_M artifact. At 262,144 tokens it is 16 GiB, which is not. Choose the window your workload actually uses and pay for exactly that.

Two smaller traps follow from the layering. Per-request options sent through the API override Modelfile parameters for that request, which is useful and also means a stray client-side default can undo your careful pinning — audit what your SDK sends when you pass nothing. And this model ships a thinking mode whose behaviour is controlled by the chat template and a per-request toggle rather than by sampling alone; the model card’s own switch is enable_thinking, and the runner exposes its own equivalent flag, so check the current documentation for the name rather than assuming that lowering the temperature disabled reasoning. It did not. It only made the reasoning less varied.

The durable habit is small: treat the runner as packaging, never as policy. It decides where the file lives and how the server behaves; you decide the sampling preset, the context window, and the system prompt, and you write those decisions down in a Modelfile that ships with your code. Lesson 8.12 puts this stack beside the server-class ones and asks when the convenience is worth its ceiling.