Local models · dual DGX Sparks · TP2

How the household models actually score

Not a synthetic leaderboard. This is every Sparks build we have measured on Léo Laporte’s Hermes agent harness — real tools, mail and calendar cases, buried-needle reads, fairness under load, and decode speed — as of 2 October 2026. Since the last snapshot: GLM-5.3-Flash moved to Mia’s TensorFold serving kit and roughly doubled its speed; a ladder of fidelity knobs and two other GLM builds were measured against it; and an overnight rerun showed how noisy the corpus total really is.

Columns are model + quant + recipe. Rows are tests. A dash means we did not run that test on that build, never a fail. Gold cells are the best measured value in the row.

How to read this

Every number comes from the same dual NVIDIA DGX Spark cluster, tensor-parallel 2. The agentic corpus is 78 cases at effort low with a 4,096-token floor. Where a build ran more than once, the cell shows min–max across arms. Measured on 2 Oct, run-to-run noise on the total is up to six cases. “Carried” means the number is from an earlier run of the same weights on this recipe lineage.

GLM · Mia TensorFold resident

GLM-5.3-Flash EXL3 4 bpw (stock weights) on Mia’s TensorFold serving kit, defaults: quantized dense layers, fp8 KV cache, DFlash2 drafter, four slots, 262K context. Resident since 1 Oct.

GLM · rung H

The same weights on Mia’s vLLM EXL3 lane with dense FP8 (PR 281). Resident 28 Sep → 1 Oct; three corpus runs.

GLM · TF dense bf16 / + KV bf16

The TensorFold kit with the dense layers in bf16, then also the KV cache (context drops to 196K). Fidelity rungs, 1 Oct.

GLM · jayleaton TensorFold

A community TensorFold build on an abliterated checkpoint (30 Sep / 1 Oct). Weights since deleted.

GLM · sparkglm NVFP4

A vLLM build on NVIDIA’s NVFP4 checkpoint, 1 Oct.

FN · myllmbox

Qwen3.8-Flash-Next on the myllmbox cluster recipe (vLLM, MTP 5). Resident 28 Sep → 1 Oct.

GLM · Fireworks

GLM-5.3-Flash as served by Fireworks: the hosted reference.

GLM · Mia c1b7d4c → d0b9608

GLM-5.3-Flash EXL3 4 bpw on Mia’s current main (fair mixed-prefill scheduler upstream), rung E settings carried. Resident 22 → 27 Sep; HEAD d0b9608 kept 23 Sep. Quick decode bench only (marked †); quality carried from the lineage.

FN · R4 d2f54b7

Qwen3.8-Flash-Next NVFP4, Mia dual recipe d2f54b7: three MTP draft tokens, bf16 KV cache, async scheduling off. Winner of the 19–20 Sep overnight ladder; resident 20–22 Sep, then parked.

MiMo-V2.6 · tonyd2wild

XiaomiMiMo MiMo-V2.6-Flash (309B, 15B active) on tonyd2wild’s two-Spark vLLM recipe with a DFlash drafter. Bake-off stopped 22 Sep after the speed, hammer and storm stages; corpus, needles and vision never ran.

GLM · rung E

GLM-5.3-Flash EXL3 4 bpw, fair-v5 rung C plus upstream main with InstantTensor. Resident 16–19 Sep, parked for the Qwen ladder, superseded 22 Sep.

GLM · Mia (HEAD → rung C)

GLM-5.3-Flash EXL3 4 bpw. Mia recipe lineage from 9 Sep through fair-v5 rung C (15 Sep); the quality tables carry this lineage through to rung E.

GLM · rung F (coop kernel)

Same GLM weights with Mia’s cooperative MoE decode kernel overlaid on stock main 56d0bdf, gated and parked 17 Sep.

DSV4F · Vision-Exp r0b0tlab

DeepSeek-V4-Flash Vision-Exp on r0b0tlab’s vLLM build, baked off and parked on 17 Sep with no swap.

GLM · Reederey

Same weights, Reederey87’s self-built ExLlamaV3 1.4.7 recipe. Bake-off 10 Sep, not adopted.

GLM · Mia c190db1

The 1–7 Sep resident (MTP k=2). Four corpus arms on an older harness.

DSV4.1 · EXL3

DeepSeek-V4.1-Flash EXL3 2.9 bpw, shipped recipe, text-only. 13 Sep.

FN · NVFP4

Qwen3.8-Flash-Next, NVIDIA NVFP4 checkpoint, Mia dual recipe. 13 Sep.

FN · FP8

Qwen3.8-Flash-Next, Qwen’s official FP8 checkpoint. Resident for 1.5 h on 16 Sep, then parked.

Table H — the latest window, 28 Sep → 2 Oct

Rung H, the TensorFold ladder, two other GLM builds, the Qwen myllmbox recipe, and the hosted reference. Corpus at effort low unless marked. Where a build ran more than once, each run is shown. The previous GLM build scored 62, 59 and 56 on three clean runs, so corpus totals within about six cases are noise; the per-case rows are where real gaps show. The strip below is aggregate tokens per second across four streams.

Speed for the resident build comes from the 2 Oct overnight probe, when nothing else was using the model; a 1 Oct probe that shared the model with the household agent read 58 on code.

Table A — agentic corpus, 78 cases

Pass / cases, by suite. Bar ≥ 59 on the total. Visual strip uses the high end of any range.

Qwen R4 (20 Sep) is the first non-GLM build over the bar: best on the board for agent basics (23/28) and mail and calendar (5/5), level on general knowledge. It gives the points back on precision traps and on the robustness suite: two needles and determinism missed. MiMo never reached the corpus. The official FP8 checkpoint (16 Sep) did not change Qwen’s shape: +1 total, a reshuffle of the same suites.

Table B — the cases that separate the models

Pass / fail, or k/n across arms. This is where GLM’s quality lead actually lives.

† c190db1 arms ran on harness b24e30c before the needle refit; the 200K needle there is not comparable.

Table C — other quality and reliability gates

Reference points on the same coding axis: frontier pair (Kronk / Pi) = 100; Mac mini qwen3-coder = 53.5. Effort vocabularies differ by build — a rejected medium or high is a 400, not a quality fail. The Qwen R4 and MiMo hammers ran at effort medium. MiMo accepts every effort value, but none of them turns thinking on. The storm guard is a small proxy that ends the reply after the first tool call. It is a band-aid outside the model, not a fix.

Table D — speed and capacity

Tests down, models across. Decode is tokens/s. Prefill is wall time to first content. Fairness bar is a short request waiting behind a long read, target < 10 s. Newcomer bar is first token of a 5.5K prompt ten seconds into another thinking reply, target < 20 s.

Aggregate tokens per second with eight streams in flight. Qwen R4 now leads every decode row on the board. Rung F’s cooperative kernel lands 2–6 % under rung E on every decode row — single-stream 27.4 against 27.9 no-think, 32.0 against 34.1 thinking — because rung E already carries PR 182’s fast thin-decode path that the overlay replaces.

† Quick bench on the resident GLM builds, not the gate harness: three runs of 500 tokens at temperature 0, 22 Sep (c1b7d4c) · 23 Sep (d0b9608). Rung E’s third single-stream figure is its 19 Sep control run. MiMo’s launcher sets no long-prefill threshold, which explains its fairness stalls; the rung that would have added one never ran. ‡ Reederey 2b5dfdc single-stream thinking figure carried from the 10 Sep session summary; approximate. Only fair-v5 GLM (rung C) is under 5 s behind a long read and under 15 s for a newcomer during decode. Qwen FP8 sits right at the 10 s fairness line.

Table G — Qwen3.8-Flash-Next overnight ladder, 19–20 Sep

Mia dual recipe d2f54b7, one change per boot, the same 40-minute gate on each rung: a decode sweep, smoke tests, fairness on frozen inputs, and the hammer. A rung passes if the gate passes, every fairness median is under 10 s, and thinking decode is at least 0.98× the current winner.

The upstream “MTP 4 + async” package did not deliver its claimed +16 % on this pair: roughly zero on one stream, and worse at ×8. The levers that worked were bf16 KV (+9 %) and three draft tokens instead of four (+5 % single, +11 % ×4). R3 and R4 tie on one stream; R4 won on the simpler config. “Carried” cells are rung E’s 16 Sep gate.

Table E — late-August GLM serving lineage

Older harness; corpus at high effort; rob-01 at 2048 and 4096. Kept so the September board has a floor.

Table F — checked, but no comparable numbers

Reading the board — 2 October 2026

• The order is intelligence first, then reliability, then speed, with one refinement from late September: a small measured quality gap can lose to a large speed win, and GLM needed more than 50 tok/s on prose to return as the main model.
• Resident: GLM-5.3-Flash on Mia’s TensorFold kit (since 1 Oct). 51 prose, 79 code, 74 tool calls, 105 across four streams, with the fewest tool-call loops on the board and an 11-call agent chain in 53.6 s.
• The corpus total is noisier than we thought. Rung H scored 62, 59 and 56 on three clean runs. The TensorFold build’s 58 (twice) sits inside that band, so the total alone does not separate them.
• The real difference is four cases, and it is consistent. The TensorFold build missed the 200K needle, recall of an earlier turn, refusal shape and number decoys on both runs; rung H passed all four every time. The bf16 dense rung gets the three short cases back but still misses the needle, which points at the kit for the needle and at its quantized dense layers for the rest (one run each, not proven).
• Fidelity knobs: bf16 dense and bf16 dense + KV both scored 59, inside the noise, for about 40 % less speed. Higher effort never beat low.
• Hosted reference: Fireworks’ GLM scored 59, inside the same band, so the local quants have not lost measurable intelligence; only speed differs.
• Earlier on this board (25 Sep): Qwen R4 matched GLM’s total but lost the precision and needle rows and was replaced after two days in the seat; MiMo-V2.6-Flash was stopped after tool-call storms; tables A–G below keep those builds.