Qualcomm Hexagon and the edge

01 · Concept

Concept

A product manager asks the only question that matters for a mobile roadmap: can we ship Qwen3.8-27B in the app, so it runs on the phone with no network round trip? This lesson answers no, and the reason the answer is worth a whole lesson is that arriving at a well-supported no is an engineering result. Most edge disappointments come from teams who never did this arithmetic and discovered the ceiling three months into an integration.

Start with the memory budget, which is not the number on the spec sheet. A current flagship handset ships with something in the region of 12 to 16 GB of DRAM, but an application does not own that machine. The operating system, the compositor, the camera stack and every backgrounded app share it, and the OS reclaims memory from foreground applications under pressure. A realistic sustained budget for one application’s model weights is a few gigabytes — take 3 GB as a stated working assumption for the rest of this lesson, and adjust it for your own platform. Now bring the ladder from lesson 7.9 and the artifact sizes from lesson 7.12:

The classic wrong turn arrives immediately: quantize harder. Two bits instead of four, surely, halves it into range. Do the arithmetic rather than the hoping:

before the group scales and mixed-precision tensors that lesson 7.10 showed always ride along, and therefore still more than twice the budget. The move fails on size, and it fails harder on quality — lesson 7.9’s degradation curve is gentle from 16 to 8 to 4 bits and steep below that. Push the logic to its conclusion and it becomes absurd: fitting 27 billion parameters into 3 GB requires

under one bit per parameter, which is not a quantization setting but a research programme.

Solve the equation the other way and it becomes a design brief. At 4 bits, a 3 GB weight budget affords about 6 billion parameters, and that is before the KV cache, the runtime workspace and the tokenizer. Do not reuse lesson 7.2’s 64 KiB per token here: that value belongs to Qwen3.8-27B’s 16 caching layers. For the chosen 3-to-4-billion-parameter model, derive KV bytes as layers times KV heads times head dimension times two tensors times bytes per element, then multiply by 8,192 tokens. A sober target is therefore a model in the 3-to-4-billion-parameter range, which is precisely what lesson 6.11 builds: a distilled sibling, a genuinely smaller network trained against the large model’s outputs, inheriting behaviour rather than weights. Quantization reduces bytes per parameter and bottoms out; distillation reduces the parameter count, which is the only term that gets you through the door.

Bandwidth confirms the verdict independently, which is the reassuring part of a negative result — two unrelated arguments agreeing. Mobile LPDDR bandwidth is an order of magnitude below desktop unified memory and two below data-center HBM; take 60 GB/s as a stated order-of-magnitude assumption rather than a vendor quote. Using artifact size only as a proxy, lesson 9.1’s division gives about 3.5 tokens per second for the pinned 17.1 GB artifact and roughly 30 for a 2 GB one. Even in a fantasy where the large model fitted, it would be unusable.

So what does the silicon actually offer? Qualcomm’s Hexagon NPU is a fixed-function-leaning accelerator tuned for quantized integer arithmetic at low power, and Qualcomm’s developer documentation exposes it through AI Engine Direct — the QNN stack — with AI Hub as the compile-and-validate path that targets a specific device family. The mental model that matters is this: the NPU is fast and efficient only for graphs it fully supports. NPUs prefer static shapes, quantized weights and a fixed operator set. Anything outside that set falls back to the CPU or GPU, and a graph that falls back on a hot path has spent its power advantage while keeping its integration cost. Qwen3.8-27B is a hybrid architecture whose Gated DeltaNet layers are new; vendor kernel coverage for brand-new operators arrives in specific toolchain releases, exactly as lesson 7.12 warned about local runtimes. Verify operator coverage on your target device before committing to an architecture, not after.

The durable lesson is the shape of the answer. Sizing produced a clean, defensible no, and the no pointed directly at the right yes: hybrid deployment, where a small distilled model handles latency-sensitive and offline work on the Hexagon NPU, and the 27B is called over the network when the task genuinely needs it. That split is a cost and reliability decision as much as a technical one, and lesson 9.10 gives it a price.