Running models locally

01 · Concept

Concept

Can your machine run Qwen3.8-27B? The honest answer is a ladder, not a yes or no — and the rung you land on is set by the quantization level you accept, using exactly the arithmetic of lessons 7.9 and 7.10.

At the top, bf16 weighs about 54 GB; int8 is approximately 27 GB before format overhead. At the pinned community revision, near-4-bit text-model files span 15.7–17.9 GB (14.6–16.7 GiB), with Q4_K_M at 17.1 GB (15.93 GiB). Lesson 7.10 owns those exact artifact figures. The model card does not state a 24 GB VRAM minimum, so this lesson treats 24 GB as a concrete course scenario whose complete resident budget must be measured.

The classic wrong turn is to multiply the rounded 27B label by half a byte and conclude that a 16 GB card works. The exact pinned Q4_K_M text file is already 15.93 GiB, before the separate multimodal projector, KV cache, DeltaNet state, runtime workspace, or display use. Unless the cache is quantized too, it accrues in bf16 regardless of weight precision. File size is an input to capacity planning, not the resident-memory answer.

Size the 24 GB scenario step by step. Q4_K_M text weights occupy 15.93 GiB on disk. At 8,192 tokens, KV costs 512 MiB and the reference float32 DeltaNet matrix state adds 144 MiB per sequence. Runtime workspace, compute buffers, allocator behavior, the optional separate projector, and display use remain backend-dependent, so load the exact stack and inspect realized memory rather than claiming a universal total. At 65,536 tokens the KV term alone reaches 4 GiB; at the native 262,144-token window it reaches 16 GiB. Configure the context you will actually support.

Capacity decides whether the model loads; bandwidth decides whether it is pleasant. Each decoded token moves a large fraction of the resident weight bytes through memory, so local generation speed tracks memory bandwidth far more than compute specs — one reason unified-memory machines punch above their apparent class, and the exact mathematics of lesson 9.1’s roofline treatment. Remember too that this is a multimodal model: the vision tower is part of the checkpoint, and image inputs add encoder computation and buffers on top of text-only sizing.

Where does it actually run? The community quantization ecosystem for this model spans llama.cpp and its GGUF format, the runners that wrap it such as Ollama and LM Studio, and MLX on Apple silicon. One check outranks every download: the runtime must implement Qwen3.8-27B’s hybrid architecture — Gated DeltaNet layers beside full attention — in its kernels. Support for brand-new layer types arrives in specific releases, not by osmosis; verify before pulling 17.1 GB. From here the course goes hands-on: lesson 8.7 works llama.cpp and GGUF properly, lesson 8.9 runs the model through MLX, and lesson 9.6 explains why Apple’s unified memory changes the local calculus.

The durable planning order: pick the quantization rung your memory affords; add cache at the context you will really use, plus state and workspace; confirm the runtime supports the architecture and consumes the packed format natively; then measure bandwidth-bound reality. Local inference is rewarding precisely because every layer is visible — but that also makes you the serving operator, and this lesson’s arithmetic is your capacity plan.

Case study: Kimi K3 in C and storage-backed inference

Keeping all weights resident is not the only execution strategy. Kimi K3 in C implements CPU inference in C99: it loads selected MoE experts in native MXFP4 and streams dense trunk layers within a configurable memory budget. The architecture makes the tradeoff visible: fewer resident weights can mean more storage reads per token. This is a separate model and runtime from the course’s Qwen example.

Read the measurement data and its limits alongside the headline. Peak RSS measures the process’s resident RAM; the checkpoint and packed trunk still require terabytes of disk space. The published memory ladder varies budgets on a server, rather than benchmarking a collection of laptops. Keep its hardware, prompt, threads, storage and cache conditions attached to any timing claim. These are engine measurements, not task-quality scores.

For a code-reading exercise, follow src/io/k3_trunk.c and src/cache/k3_cache.c, then inspect the tests that need no model weights. Fixture correctness and practical full-model performance are separate checks; the former is a useful starting point before a large download.