The Trunk Tax

The tax

A safetensors file starts with a header that lists every tensor, its type, and its shape. Reading the headers of all 96 shards of Kimi K3 costs about 60 MB of range requests against a 1.56 TB checkpoint — and settles where decode time actually goes.

Per token, decode touches 16 of 896 experts (25.8 GB of MXFP4) and the whole 111.2 GB BF16 trunk. The bandwidth statement is batch-1; the capacity consequence — shipped K3 cannot fit an 8×B200 node, and only a quantized trunk plus KDA's 27 KB/token KV makes it fit — survives batching.

The refund, measured

Quantizing only the trunk converts the accounting into speed. On OLMoE-1B-7B the full expert-precision × trunk-precision matrix gives 1.23× / 1.36× / 1.42× at 8 / 6 / 4-bit trunks (medians of 12 shuffled, cooled runs with bootstrap CIs; single-pass numbers proved wrong by up to 37% and never touch the headline). On gpt-oss-20b the repaired 6-bit trunk measures 37.9 → 49.9 tok/s, 1.32×, with perplexity held on two corpora and a paired per-chunk CI that includes zero. The same refund replicates on CUDA: the public gpt-oss GGUF variants — identical MXFP4 experts, only trunk precision differing — run 62.1 / 84.2 / 92.5 tok/s on an L4. And the quality side holds at frontier scale: on official Qwen3.5-122B weights, an 8-bit trunk moves perplexity by nothing measurable.

Speedup over each model's shipped pattern versus trunk width. On today's datacenter engines the software floor we measured in BRL-2026-11 bounds the trunk-FP8 refund to roughly 1.11× until the floor is removed; the lean-stack numbers are the ceiling the engines are leaving on the table.

The fragile tensor

The naive version of this refund fails on gpt-oss: whole-trunk 8-bit at group size 64 costs +53% chat perplexity, and 4-bit destroys the model outright. Holding one tensor class at BF16 at a time isolates the culprit completely — attention recovers nothing, the output head recovers nothing, restoring the embedding table recovers everything. A kernel-free control reproduces the damage with pure floating-point arithmetic, so this is the quantization math itself, not an implementation.

Why this tensor: gpt-oss's embedding rows are extreme outlier carriers — median absmax/RMS of 14.1 against 3.6–4.0 for every other checkpoint measured, at both 20B and 120B scale. Under min–max grouping the outlier sets its group's scale and every neighboring weight is coarsened by an order of magnitude relative to its own size; halving the group confines the blast radius, which is why group-32 alone repairs it. The statistic computes from any checkpoint in seconds, separates fragile from immune with no overlap across eight checkpoints including K3 (immune, 4.0), and ships in the repository as a screen.

In the wild

The most-downloaded MLX build of gpt-oss-20b — 348,084 downloads — quantizes embeddings exactly this way, and measures chat perplexity 21.0 against the vendor pattern's 13.8. Healing the public artifact in memory, restoring only the original embedding rows fetched by byte range, recovers it to 13.43 — within 0.004 of our independently rebuilt repair. One tensor separates the damaged build from clean. A config audit of 103 quantized MLX MoE builds finds the risky default in 70 of them; family-by-family measurement finds actual damage in exactly one of six checkpoints tested.

The metric that fails in both directions

Here is the part that should worry anyone shipping or consuming quantized models. That +53% regression is invisible on wikitext — the corpus class the ecosystem evals against preferred the damaged build. It is invisible to a full 1,172-item ARC eval. It is invisible in greedy generation, where damaged and healed builds walk near-identical paths. And it is invisible to a blind frontier-model judge over 300 sampled generation pairs, which scored the healed build at a coin-flip 51%. The regression is real, mechanistically attributed, and by every user-facing probe we could construct, practically latent. Teacher-forced perplexity — the default quality metric for quantized builds — missed real distributional damage in one direction and overstates user harm in the other. The field's QA for quantized artifacts is measuring something users don't experience, and missing something they ship.

Reproduce it

Everything is public: the header-only census tool that reads any Hugging Face checkpoint's byte anatomy without downloading it, every measurement harness, every raw cell, and the paper itself. The per-build fragility test takes minutes on a laptop.

github.com/brainsless/the-trunk-tax · paper PDF