A Measured Decomposition of the Trillion-Parameter Serving Step.

A Measured Decomposition of the Trillion-Parameter Serving Step

A lossless single-stream record on Kimi-K2.6, and the measured physics of the decoding step.
Mohammad Alsufi and Connor Boone, and the Brainsless Research Lab AI Systems Research Group. Brainsless Research Lab, Technical Report BRL-2026-11, July 2026.

Abstract

We serve Kimi-K2.6, a 1T-parameter mixture-of-experts model with 384 routed experts, at 511.6 tokens per second single-stream on four B200 GPUs — a record set at 505.9 (depth 7) and raised to 511.6 (depth 6) by blind re-runs of the unmodified public release, gates fixed before the runs — measured at the public leaderboard's own workload (10,000-token inputs, 2,048-token outputs, temperature 0.6), sixteen novel prompts per cell, prompts hash-pinned before any measurement ran. Replication moved this record up, not down. On the day the record was set, the fastest reading on Artificial Analysis' fourteen-provider leaderboard for the same model was Crusoe's 438.1 from undisclosed hardware (505.9 stood 15.5% above it), followed by Fireworks 381.2, CoreWeave 261.8 from a 72-GPU rack, Nebius 222.4, and Together 218.2; at the raised cell's pin that leader read 451.0, with 511.6 standing 13.4% above it. 505.9 was the first single-stream GPU reading above 500 on this model. Per accelerator ours is 127.9 tokens per second per GPU, roughly 3× the best disclosed configuration, and it comes from within one GPU of the physical floor: the 554 GiB checkpoint does not fit on three B200s. The median first token on a 10,000-token prompt arrives in about 240ms measured at the server; the fastest provider's measured latency at the same workload is 11 seconds over the network.

The path to the number is the science of this report: a millisecond-by-millisecond decomposition of the serving step at one trillion parameters. The bare forward pass costs 6.6ms at 10k context; verifying a drafted token costs 0.397ms, measured with a weights-free drafter run at two widths so the proposer's own overhead cancels — the first measured verify-cost curve on a fine-grained 1T MoE, and the direct measurement of the verify tax this report is named for. The tax is real and small; the draft pass and engine overhead were the expensive parts, and doubling the node to eight GPUs changed nothing (bare forward 6.54 against 6.60ms), because the step was roughly 95 percent software against a per-GPU memory-traffic floor near 0.3ms. Moving the same model, same public draft head, and same protocol to the engine the decomposition pointed at cut the step from 12.24ms to 10.10ms at identical acceptance and produced the record. We also measured both live drafter families, EAGLE-3 and DFlash, on the same 1T target under identical conditions — public head-to-heads stop at 120B — and the published small-scale ordering reverses, which the verify tax prices exactly. Along the way our warm-start diagnostics exposed a training defect with consequences beyond this lab: a one-line patch had shifted the distillation teacher one transformer layer early, invisible to from-scratch training and fatal to warm starts. Replaying the same protocol against a provider's production API read 118–169 tok/s against that provider's 381.2 leaderboard median, so the field comparison here is stated against the board's numbers as published. Every table re-runs from the public release: one command, one rented four-GPU node, about $15 — and blind re-runs of that release reproduced every n=16 cell within the stated node-draw envelope, raising the record to 511.6 (depth 6). The record configuration carries no trained component, so the number is this configuration's floor; our own trained head reads +5.8% over the public head in a same-session pair and crossed 500 on a below-reference node. Artifacts and costs are itemized in the compute statement.

1. What we serve

We serve Kimi-K2.6, the open trillion-parameter model, on four rented B200s. The record-setting cell: 505.9 tok/s on agentic tool traffic, the public 3B MLA head drafting at depth 7 on SGLang v0.5.14, measured as the interpolated median of sixteen streamed requests at 9,900–10,400 input tokens and 2,048 output tokens each (artifact: brl11_stage600r.json). The same session's depth-6 cell reads 487.2 and its math cell 419.2; a second node the same day reads 498.5 at depth 6 (artifact: brl11_stage600sg.json). The no-speculation baseline on this protocol is 150.9.

The field, pinned the same day from the Artificial Analysis leaderboard at the same 10k-input workload (all external): Crusoe 438.1 from undisclosed hardware (peak 449); Fireworks 381.2, count undisclosed; CoreWeave 261.8 on a GB300 NVL72 rack; Nebius 222.4 and Together 218.2 (FP4) follow, GMI's FP8 build reads 40.2, and the board's remaining entries carried no speed reading at the pin. NVIDIA's own published single-user record on the nearest comparable open MoE (DeepSeek-R1 671B) is 340 tok/s lossless on eight B200s, 368 with relaxed acceptance at a measured 2.8-point MMLU-Pro cost.

On 2026-07-06 we replayed this report's record protocol against the production API of Fireworks, the second-fastest provider on the pinned board and the fastest whose serving is purchasable through a public self-serve API: the same sixteen SHA-pinned 10k-token prompts, 2,048-token outputs, temperature 0.6, the same per-request streaming statistic, sent from a standard paying account over the public internet. The production kimi-k2p6 endpoint returned an interpolated median of 118.4 tok/s (n=16); with reasoning off it returned 149.2 (n=8), and the provider's kimi-k2p6-turbo router returned 168.9 (n=8). Every request completed its full 2,048 tokens, and every per-request row is in the artifact (fireworks_live_replication.json). The leaderboard credited the same provider 381.2 at the pin — 2.3–3.2× the production readings. We measured one provider, from one client region, on one day, and we do not know what configuration served either measurement; explanations consistent with the gap include regional routing, time-varying load, tiered or dedicated deployments, and the possibility that the endpoint the leaderboard measures is configured differently from the production API sold to customers — our data cannot distinguish among them, and we attribute nothing. What the replication establishes is the part that matters for reading the standings: a leaderboard median is a property of the specific endpoint and conditions the leaderboard measures, we could not reproduce it from the production API, and the gap is more than an order of magnitude larger than the board's own day-to-day movement (its leading entry moved 438.1 to 451.0, about 3%, across the two days we pinned it). The comparison in this report is stated against the board's numbers as published. Our own number reproduces from a published protocol on disclosed hardware.

The comparison is sometimes framed as apples to oranges: a controlled benchmark on one side, medians of endpoints under production load on the other. The replication above is the answer to that framing. When we replayed the identical workload against the same provider's production API — the endpoint a customer actually buys — it read 118–169 tok/s, not the 381.2 the board credits. A board median is therefore not a measurement of "live serving" in any general sense; it is a reading of one endpoint under one set of conditions, and it did not survive outside them. There is exactly one number in this comparison that anyone can re-measure — ours — and the code to test both, our cell and theirs, is in the release. That fragility cuts both ways, and the claim is scoped to survive it: the field table is a ranking over published measurements, not over provider capabilities. We claim the fastest measured number; whether a funded team could match it is exactly what the release lets them show, at $15, and this report's own thesis — the step is mostly software, the components public — says the attempt would land close. Independent of the field entirely, the number stands against physics: 3.39× the undrafted floor on the same node and protocol. Any controlled cell published above 511.6 will be cited in this report's next revision.

The record itself has been replicated. On 2026-07-06 the protocol was re-run three times from the public release on fresh node draws, blind: gates fixed before the runs, no configuration touched, one of the runs clean-room from the public materials alone — the repository located from this report, the prompts rebuilt to the pinned SHA, the published medians re-derived from the raw artifact rows, the node rented fresh. Every n=16 cell landed inside the stated node-draw envelope: tool depth-6 cells read 511.6, 484.4, and 474.2 on anchors of +0.9, −5.7, and −4.3% against reference; the depth-7 cell read 486.5 with a first-eight median of 538.0; math read 405–422 against the published 419.2. Within these sessions individual requests peaked at 568.0 tok/s (per-request rows in the artifacts). The best replication cell (511.6, depth 6, n=16) exceeds both the record-setting cell (505.9, depth 7) and its like-for-like depth-6 counterpart (487.2), and raises the record. Replication moved this record up, not down — and a record whose replications read above it is a floor: every component in the configuration is stock and public, and what moves the number from here is the trained head. Artifacts: brl11_repl_r1.json, brl11_repl_cleanroom.json, brl11_repl_fovea.json.

Three scoped claims follow. Absolute: 505.9 sat 15.5% above the best GPU provider median at its pin and 12.7% above that provider's peak; the raised 511.6 stands 13.4% above the leader at its own pin (451.0). Per accelerator, nothing with a stated GPU count is close: 511.6 on four B200s is 127.9 tok/s per GPU, against 42.5 for NVIDIA's eight-GPU lossless record configuration. And four B200s is within one GPU of the physical floor: the checkpoint alone is 554 GiB (595 GB) at load, more than the 576 GB of HBM on three 192-GB B200s.

Latency at the same workload: our median first token on a 10k-token prompt is about 240ms measured at the server. Artificial Analysis measures the providers over the network at 11.0–21.9 seconds for the same input length. A gap north of 30× survives any reasonable correction for network and queueing.

The economics are recorded in the session artifacts at the node's $24/hour rate. At the record cell's rate, a 2,048-token tool response is 4.0 seconds of decode. Single-stream serving on this node prices at $13.18 per million output tokens at the record-setting cell's rate (505.9; $13.03 at the raised cell) against $44.15 undrafted (150.9 tok/s, same protocol); at 32 concurrent streams a prior session on the earlier engine recorded $7.32 per million. On specification alone, four B200s are half the 1,000W-TGP power envelope of the eight-GPU record configuration; we have not measured wall power and print no joules number until we have.

2. The measurement the field publishes around

Speculative decoding is the reason every fast deployment of a large model is fast. A small drafter proposes several tokens; the large target verifies them in one step; accepted tokens are speed, and the output distribution is unchanged. The two live families differ in how they guess: EAGLE-3 drafts left to right, each token conditioned on the previous one; DFlash drafts a block of eight in parallel, conditioned on the target's hidden states.

As of July 2026 the public record cannot say which family is faster on a trillion-parameter model. The DFlash paper compares the families at 4B–30B; NVIDIA's comparison reaches gpt-oss-120B; DeepSeek's DSpark report compares them to 14B. At the scale where deployments actually run, they publish one family each and no throughput. SPEED-Bench contains no block-diffusion drafter.

We measured it, twice. First at the released budgets on short novel prompts (sixteen per domain, temperature 0.6, one session, single stream), where the published small-scale ordering — block-diffusion ahead — reverses: EAGLE-3 takes general chat 243 against 174–185, tool calls 289–337 against 218–258, math 324 against 278, code 290 against 233–255, while the block family holds the higher accepted length in every one of those cells (artifact: brl11_verify_tax_session.json). Then at the leaderboard workload, where long inputs raise acceptance for both families and the ordering holds. The better guesser losing everywhere is the central fact of drafting at this scale, and it has a mechanism.

3. The verify tax

On a dense model that table would be impossible: verifying nine tokens costs nearly the same as verifying four, because the weights stream through the GPU once per step either way. A 384-expert MoE breaks that arithmetic. Each verified token routes to its own eight experts, the union of experts a step touches grows with the number of tokens in it, and every touched expert's weights stream from HBM. Dividing accepted length by measured throughput gives the implied step time: EAGLE-3's 4-token verify costs 8.0–9.3ms across domains; DFlash's 9-token verify costs 11.5–16.6ms. The block family's acceptance advantage, pooled across the two measurement sessions, is a factor of 1.1–1.4; its step-cost penalty is a factor of 1.3–1.8. The product decides every row.

The tax appears before the first token — batch-1 median time-to-first-token is 1,160ms for DFlash against 256ms for EAGLE-3 at 8K context, because the block is drafted and verified before anything streams — and it compounds under load: DFlash per-stream decode falls 156 → 60 → 18 tok/s at 1, 8, and 32 streams, and at 32 streams the node sustains 910 tok/s aggregate with EAGLE-3 against 845 undrafted and 499 with DFlash (recorded per-token cost $13.36/M against EAGLE-3's $7.32/M), so a drafter can be net-negative for fleet throughput while winning batch-1 latency. Concurrent work measured the same expert-union effect on MoEs up to 45B and built draft-length controllers on it; these numbers put it at 1T.

The tax is also a lever. If longer drafts carry a verify cost, they should be bought exactly where acceptance is high enough to pay — and at the 10k workload it is. From the baseline run's measured per-position acceptance, the run's own live forecast priced k=5 at ~398 tok/s on math, from the baseline node, before the arm ran. A second node the same day read 10% slower on its health cell, and its math k=5 measurement, 357.6, matches the forecast at that deficit; on tool traffic, where acceptance ran higher still (τ 4.96 EAGLE-3, 4.79 Fovea, with local-argmax reduction), the same draft length lifted the cell from 368 to the 396–398 band. The forecast survives contact with the measurement, which is what a mechanism is for.

4. Fovea

Fovea is our drafter program: heads we train ourselves, on conversations our own served target regenerates, so the drafter learns the distribution it will actually draft for — online drafter adaptation applied at 1T. The name is the fovea, the pit in the retina where vision is sharpest. Two heads exist.

Fovea-1, a fine-tune of NVIDIA's block-diffusion head on 5,859 general conversations, matches or beats its vendor parent in every comparable cell: +6% general chat, +4% tool-think, +10% code, +4% math at n=8, and a tie on tool no-think (artifact: brl11_verify_tax_session.json). It proves the training recipe moves served performance; the verify tax caps what its family can win.

Fovea-E is the routing conclusion executed: a warm-start of the public EAGLE-family head, trained on 2,373 hard-example tool and math conversations regenerated by our target at temperature 0.6, mode-matched to serving. It produced the earlier stack's cells; on the record stack the public head sets the number, which is deliberate, so the record reproduces from an off-the-shelf checkpoint. Fovea's aim is the narrower traffic served in our own production, where a head matched to the distribution has more headroom than a general one, and the acceptance gains that price the next ceiling.

5. The teacher one layer down

Fovea-E's training pipeline carried a defect whose detection is the finding. The warm-started head loaded bitwise-correctly, then trained as if from scratch: near-random distillation loss, accepted length 0.8–1.4. The autopsy exonerated the checkpoint pipeline by reproducing the trainer's exact load path on CPU, and found the defect upstream of the optimizer: a one-line patch inherited from an earlier training program of ours (justified there by a claim we now know is false — this model has no MTP layer) combined with the engine capturing hidden states only at layer input. The distillation teacher was therefore computed from the hidden state one transformer layer before the model's real output.

A from-scratch draft learns the shifted teacher invisibly: training curves look normal, and the defect surfaces only as unexplained served-acceptance collapse. A genuine warm head approximates the true output distribution, so against the shifted teacher it reads near-random while visibly beating from-scratch initializations; that signature caught it. Two consequences. For this report: our earlier from-scratch drafts' served collapse, previously read as pure entropy-floor evidence, is partially attributable to this defect; the floor argument in Section 8 therefore rests only on the public heads. For the field: anyone distilling drafters against a served MoE should verify the teacher's final-hidden capture against the serving engine's, and a warm-start canary is the cheapest detector we know of — it converts a silent data defect into a loud one. With the capture corrected, the same course trained cleanly (held-out accepted length 2.59; artifact: fovea_e_result.json) and produced our strongest head.

6. The configuration ledger

Findings from the sweeps that model cards do not mention. Artifacts: fovea_gauntlet.json, brl10_k26_full_v2.json, brl11_record.json.

Draft length is a workload decision, priced by the tax. At short contexts, k=5 loses: health cells read 265–294 against 335 at k=3, because two extra verify tokens cost more than short-prompt acceptance repays. At 10k context the same k=5 wins the record. Per-position acceptance measured at the target workload is the deciding instrument, and it forecast the outcome before the spend.

KV-cache dtype is domain-dependent. Native precision against fp8 moved math +34% in one session while costing tool throughput 6% and code 19%. Pick per traffic.

Speculation levies a capacity tax. At the same memory utilization, the KV cache holds 484K tokens undrafted, 336K with EAGLE-3, 295K with DFlash: drafting spends concurrency and context headroom, not just step time.

Node draws move throughput ~10% (two same-day nodes read 294 and 265 on an identical health cell). Same-session pairs are the only controlled comparisons in this report, every arm re-anchors, and the record cell's own anchor is printed with it in the artifact.

The engine's flag surface shifts under you. The attention-backend environment variable our earlier sessions set was removed upstream months ago and did nothing; a correctness patch for an NVFP4 initialization defect (nondeterministic corrupted output at batch 1) postdated our image and had to be applied at runtime; capture sizes must be multiples of (1+k) or concurrency runs uncaptured. Every flag in the record run was verified against the engine's source before launch, and the run fail-closes on a manifest.

7. What the proxy missed

Before spending on training we tested whether a laptop-scale proxy could predict whether a drafter course would move acceptance. A standalone 1.7B proxy, fine-tuned on the same kind of data under three instruments of increasing care, converged on +3 points, saturating — a do-not-fund verdict. The served heads answered otherwise: Fovea-1 beat its parent in every cell, and Fovea-E took the record. The proxy measures text-only imitation; the real heads read the target's hidden states at every layer, and what a course teaches that channel is invisible to a text-only rehearsal. Data decisions for hidden-state-conditioned drafters are decided by training the conditioned drafter and serving it.

8. Two ceilings, priced

Open-ended chat did not reach these numbers, and the anatomy says why in two numbers. Acceptance on chat is entropy-bounded: every functioning public general head we measured sits at τ 2.2–2.9 on chat, against 3.5–5.0 on tool and math at the 10k workload and released block budgets (short-prompt tool and math cells run 2.6–3.8). And the verify tax re-prices every added draft token, so the block lengths that pay on low-entropy traffic do not pay on chat: at the block lengths we measured, matching the cheap verifier's chat throughput would need τ above that entropy band. The conclusion is a routing rule, arithmetic rather than advice: serve each traffic domain with the family and draft length that wins it. The record cell is that rule executed — our product's traffic is agentic tool calls, so that is the cell we trained for and the cell we hold.

Also in the limits column: cross-session throughput drifts by tens of tok/s and node draws by ~10%, so same-session pairs are the only controlled comparisons here; the k=5 chat and code cells were not measured at the 10k workload; and the verify tax is now a direct measurement — 0.397ms per verified token from a weights-free drafter differential (brl11_stage600e1.json) — replacing the τ ÷ throughput inference this report first shipped with.

9. Related work

EAGLE-3 and DFlash define the families measured here; NVIDIA's Kimi-K2.6 cards publish each head separately, at incomparable draft budgets and without throughput. Controlled cross-family comparisons exist at 4B–30B (the DFlash paper), 120B (NVIDIA), and 4B–14B (DSpark); this report contributes the 1T case and finds the ordering reversed. On the verify-cost side we build on Cascade, EVICT, and MoE-Spec, which measured the expert-union effect to 45B and control draft length against it; DSpark ships confidence-scheduled verification in production; D-Cut implements post-draft truncation for DFlash in a pending patch. None publish the batch-1 anatomy at ≥600B that their mechanisms depend on. Draft-length adaptation for autoregressive drafters goes back to SpecDec++ and EAGLE-2; online drafter adaptation is established at small scale. Fovea applies it at 1T with target-regenerated data and same-session served comparisons, and adds the teacher-capture defect class (Section 5) to the training literature.

10. What happens next

The record was set with two proven levers held off and two domains not yet measured at the record draft length, so the number has measured room above it. Reaching that headroom is engineering with named parts: turn the allreduce and kernel-backend levers back on under proper proofs, and run course four of Fovea-E on the record run's own lowest-acceptance rows, the same hard-example loop the leading deployment describes. Fovea opens up: the weights are derivatives of openly licensed heads trained on outputs of an openly licensed model, and we intend to publish both checkpoints once license review closes, so every table here can be re-run by anyone. The verify tax gets its direct observation: the unique-experts-per-step counter leads the next session. And the anatomy continues toward serving this class of model from hardware that costs a tenth as much — the subject of the next report.

Compute statement

The record campaign behind this report ran $125.70 of rented serverless GPU time inside a $300 budget: stage A ~$25.20, the verify-tax differential ~$26, the TP8 null $41.50 (8×B200 at $50/hr; every other session 4×B200 at $24/hr), the SGLang smoke $11.10, the record session $12.90, and the depth ladder $9.00. The Fovea training courses this report reuses were run and priced in the prior edition. The record-session figure includes node boot, health checks, and the aborted first draw; the artifacts' in-run estimates count measured cells only. Artifacts: brl11_stage600r.json (record cells), brl11_stage600a.json (baseline, k-ladder, per-position conditionals), brl11_stage600e1.json (verify-tax differential), brl11_stage600b.json (TP8 null), brl11_stage600sg.json (SGLang cross-engine), brl11_stage600d.json (depth ladder), fovea_verdict.json / fovea_gauntlet.json (Fovea-1 pairs, config sweeps), fovea_e_result.json (Fovea-E course). Record prompts SHA-256-pinned; raw output text stored per request for independent tokenizer normalization (measured o200k/native ratio 1.004–1.008).