The one speedup in local AI that doesn’t change the answers

In inference, everything costs speed: bigger models, more memory, fewer tokens per second. Speculative decoding is the exception, and it got quietly better again this week. A small draft model races ahead of the big one, guessing the next few tokens; the big model then checks all of them in a single parallel pass, keeping the ones it would have produced anyway and restarting from the last correct one. Nothing about the answer changes — the main model still decides every token — but you get several tokens per forward pass instead of one, commonly 2–4× the decode throughput. Today’s installment: DwarfStar’s maintainer says its speculative path, DSpark, “now works much better overall” alongside DeepSeek-V4-Flash. No new weights, no quality trade — just speed on hardware you already own.

Why it’s nearly free is the interesting part. Local decode is memory-bandwidth-bound: generating a single token means hauling every weight past the GPU, which spends most of its time waiting on data. Speculative decoding turns that waiting into one pass that verifies many candidates — exactly what a GPU or Apple Silicon batch is good at. The reason it’s only nearly free is the fine print: you carry a draft model (or an MTP head) alongside the real one; the win depends on how often the draft guesses right; and if your bottleneck is raw compute instead of bandwidth, the headline number never shows up. Measure your acceptance rate before you trust a speed claim — and confirm the verification step only ever keeps tokens the main model itself would have produced, or you’ve traded speed for a quality drift nobody notices.

The day’s one genuinely new model, briefly: OpenBMB’s MiniCPM5-2B, a 2B text generator under Apache-2.0 aimed at edge and on-device use. The numbers say people are running it, not reading about it: 2,879 downloads and 601 likes on the original since Sep 6, and its official GGUF build at 7,760 pulls — nearly three times the safetensors file. The local crowd votes in the format it can load straight into a tool, and for a niche edge model that’s more engagement than most releases ever see. It won’t replace a 27B on your desk; it fills the small-and-cheap slot, and it’s filling it.

The day’s one honest measurement also deserves a stop: Qwopus3.8-27B-Flash, a Qwen3.8-27B fine-tune trained to reason more efficiently, measured on an RTX 5090 — GPQA-diamond 70.7% at 4,229 tokens per correct answer (greedy, 198 questions, ~2.7 h). The tester reads that as roughly 2× the base model’s efficiency; the base’s own figure isn’t in the writeup, so the 2× is their reading, not mine. I like the unit regardless: tokens per correct answer is what a reasoning model actually costs you, and almost nobody publishes it.

From the harness layer, in one breath: DwarfStar has a Metal PR waiting to merge (+4% prefill, +6% decode), an in-progress experiment that compresses MoE experts with Q2_K instead of mxfp4 to free roughly 5 GB on 64 GB machines, and the merged agent /hints feature; Hermes can now carry a Claude Code or Codex CLI conversation into itself — Desktop browses and imports those sessions, and Bots Mode group chats landed. None of it is news; all of it is what actually compounds.

One last piece of color, flagged as the rumor it is: a $3B frontier model reportedly scored 0% on Terminal-Bench while a 27B open model beat it. I won’t build anything on an unverified frontier claim — but as a caption for a day about free speedups and small models, it lands.

Download and like counts read directly from HuggingFace; the DwarfStar, Hermes and Qwopus items are maintainers’ and testers’ own reports, labeled as such.