Running models locally

1

Concept

Running a model locally can reduce dependence on a network service, keep prompts on a controlled device, support offline work, and make experiments reproducible. It does not mean “download any model and double-click.” A successful setup aligns the checkpoint, runtime, hardware, memory budget, and workload.

Start with compatibility. Identify the architecture, tokenizer, chat template, context configuration, and license. Then choose an artifact the runtime actually supports: original framework tensors, a safetensors checkpoint, GGUF tensor types, or another documented format. A matching filename is not enough; model metadata, tensor shapes, positional encoding, multimodal projectors, and special tokens must agree.

Estimate weight memory from parameters, average stored bits, and metadata. Quantized models include scales and sometimes selected higher-precision tensors, so dividing the parameter count by a bit width is only a lower-order estimate. Runtime loaders may map the file, copy tensors, dequantize some layers, or split work between CPU and accelerator. Measure resident memory after load.

Then add dynamic memory. KV cache grows with retained context, layers, KV heads, head dimension, precision, and concurrent sequences. Activations and scratch workspaces depend on kernels and batch shape. Multimodal models add image or audio encoders and input buffers. The operating system and display also need headroom. A model whose file barely fits can fail on the first long prompt.

Capacity and speed are separate. Autoregressive decode often moves a large fraction of weights for each token step, so usable memory bandwidth strongly influences generation speed. Compute capability matters for prompt processing and specialized kernels. CPU vector instructions, GPU backend support, unified versus discrete memory, transfer links, power limits, and cooling all affect sustained performance.

Offloading extends capacity by placing layers or tensors across accelerator memory, system RAM, and sometimes storage. FlexGen, published in 2023, studied scheduled offloading for high-throughput generation on limited GPU memory. Offloading is not free: every boundary adds transfers, and storage is much slower than RAM. A configuration that technically runs can be unusable for interactive work.

Choose the runtime around the device and artifact. llama.cpp targets portable local inference across CPU and several acceleration backends and uses GGUF in its ecosystem. Other runtimes specialize in vendor GPUs or framework-native models. Confirm native kernels for the selected quantization; a fallback that expands weights can erase memory or speed benefits.

Benchmark the intended task, not a screenshot. Record runtime and model versions, device, thread count, backend, context, prompt and output lengths, batch settings, cache dtype, and thermal state. Measure load time, TTFT, output tokens per second, peak memory, energy if relevant, and answer quality. Warm and cold runs answer different questions.

Context settings deserve skepticism. A runtime may allow a large configured window while quality degrades beyond the model’s trained or adapted range. More context also consumes cache and prompt time. Use retrieval or deliberate context selection rather than treating the maximum as a target.

The durable planning order is: establish compatibility and rights; size weights plus dynamic state and headroom; verify backend kernels; estimate bandwidth and transfers; benchmark representative prompts; then secure the local data path. Local inference is rewarding precisely because every layer is visible—but that also makes you the serving operator.