The stenographer that never stops.

Models

Edge0/Audio8-ASR-Infinite — 7,859 downloads / 846 likes in six days since its 21 Sep drop, Apache-2.0. The pitch is in the name: native streaming speech recognition that runs “24/7 without drifting.” Under the hood it’s a trained Voxtral Realtime 4B causal audio tower driving a trained Qwen2.5-3B-Instruct decoder — roughly an 8B stack, bilingual Chinese/English. A native 30-second context is stretched to unlimited length by a rolling KV cache that keeps memory and latency constant, and the model decodes 12.5 times per second. You pick the trade: an 80/120/160 ms audio clock against a 240–560 ms transcription delay, with the useful operating points post-trained rather than improvised. The part that sounds like marketing but isn’t: semantic VAD, which distinguishes a thinking pause from an actual end of turn — the failure mode where acoustic VAD chops people off mid-sentence. An adapted vLLM build ships for the long-running case. Vendor-reported numbers only so far (the arXiv entry is “coming soon”), and the build shelf is nearly empty — one community MLX port at 18 downloads. The incumbents: Confucius4-R2T2 (streaming ASR, covered 21 Sep) and VibeVoice-ASR-Streaming-7B. The real delta here is the combination — selectable clock and delay dials, semantic VAD, constant-memory infinite sessions, Apache license instead of NetEase’s restricted weights. Verdict: Marginal — adopt only if you’re building an always-on voice agent that needs true streaming; for batch transcription, a classic Whisper-class model still wins on simplicity.

moondream/parakeet-redux — 1,636 downloads / 160 likes since 18 Sep, CC-BY-4.0. Moondream took NVIDIA’s parakeet-tdt-0.6b-v3 — the open ASR leaderboard’s reference — and rebuilt every encoder weight as ternary (−1/0/+1): the result fits in 178 MB and runs at 113× real time on eight x86 CPU cores and 38–43× on a fanless MacBook Air, per the card’s own head-to-head tables against parakeet.cpp, sherpa-onnx and parakeet-mlx on identical hardware and audio. The quality bill is itemized, which is rarer than it should be: 6.55 vs 6.26 WER on the Open ASR Leaderboard’s seven English sets, but the ternary version beats the full-precision original on the 25-language FLEURS set (10.56 vs 11.62) and on TED-LIUM long-form, while noise robustness takes the biggest hit (9.04 vs 6.72 on MUSAN). It needs Moondream’s own Photon runtime to read the packed weights — no GGUF path yet, so llama.cpp whisper-class tooling does not apply. Verdict: Marginal — adopt only if you need always-on, sub-200-MB transcription on a CPU; if you already run parakeet.cpp or whisper.cpp on a machine with a GPU, this buys size, not accuracy. The interesting part is the pattern: ternary compression (covered here as a weights story with Bonsai) now showing up in speech, where the encoder is small enough that the speed math actually works.

apple/LensVLM-9B — 1,432 downloads / 230 likes since 21 Sep — and the GGUF shelf already out-runs the checkpoint: bartowski’s build alone sits at 2,564, with prithivMLmods and mradermacher behind it. Apple’s entry is a fine-tune of Qwen3.5-9B with a genuinely interesting mechanism from their paper (arXiv 2605.07019): render a document’s text into compressed page images, let the model scan those cheaply, then selectively expand only the relevant pages back to full resolution via learned tools — 5×/10×/15× compression options. Long-context document QA at a fraction of the token bill. The catch is the license: the Apple ML Research Model License restricts this to research use, so whatever a reader builds stays a weekend experiment. The method is the story; the weights are a demo. Verdict: Marginal — adopt only if you’re researching visual-text compression; the learned-expansion idea is worth stealing, the license isn’t worth building on.

Quantizations

No new build cleared the gate this stretch, but the covered Qwen-Image-2.1 wave keeps compounding — worth taping because the shape of it keeps getting starker. The Comfy-Org repack went 2,220,609 → 3,641,785 downloads (+1.4M in three days). abenzerps’s uncensored GGUF went 350,678 → 876,673 (1,971 likes) in the same window. pottokao’s Heretic-ablated text-encoder GGUF climbed 74,291 → 137,283, and unsloth’s stock-official GGUF reached 170,469 — which quietly resolves the 24 Sep comparison: the no-filter control is growing fast too, the uncensored build just grows 5× faster. The base repo itself sits at 48,361: the distribution channel moves at roughly 75× the canonical checkpoint. Every time this feed re-checks, the gap widens — in image generation, the quant ecosystem is the product. Verdicts unchanged from 24 Sep: the stock unsloth shelf is the real deployment option; abenzerps is the uncensored niche; pottokao is marginal as a model but the ComfyUI-GGUF encoder loader behind it is a genuine tooling unlock.

Fine-tunes

fastino/GLiNER2.5-Decide — 14,772 downloads / 192 likes in four days since 23 Sep, Apache-2.0. Base model: fastino’s own gliner2-large-v1 (340M). The real delta: pass any label set at call time — intent, routing, sentiment, urgency, moderation — and get the classification in a single forward pass. No prompt template, no generated tokens, several heads scored in one call. That’s the right shape for local routing at volume: a 340M specialist instead of a 9B generalist burning tokens to say “billing complaint.” The 60.2% vs Laya Router’s 46.6% headline comes from the vendor’s own fast-decisions suite — 17 domains, 300 examples each, their dataset, their eval — so read it as a claim, not a leaderboard. ONNX and CoreML community ports are already appearing, which is the adoption signal that matters more. Verdict: Marginal — adopt only if you ship high-volume local intent routing; the single-pass pattern is the interesting part.

Technical term of the day: the rolling KV cache

Every autoregressive model keeps a KV cache — the running memory of everything said so far, so each new token doesn’t re-read the whole history. The cache only grows, which is fine for a chat and fatal for a model that must listen for eight hours. Prefix caching (covered here 9 Sep) attacks the problem by reusing work across requests; the rolling cache attacks it by deleting: keep a fixed window of recent audio, drop the oldest blocks as new ones arrive, and memory and latency stay constant no matter how long the session runs. Audio8’s native 30-second context stretched to 24/7 is exactly this trick.

The trade-off is honesty about what eviction costs: anything outside the window is gone, so a rolling cache is lossy by construction — the model can’t recall a name mentioned an hour ago in a pure-streaming setup. Two design choices make it workable. First, summarize-or-commit: either the model periodically commits its output downstream (where the full transcript lives) or carries compacted state forward. Second, eviction needs judgment — which is what semantic VAD is for. Deciding “this pause is a real end of turn” doubles as deciding “this is a safe place to slide the window.” For local AI the practical read: if your workload is a marathon (always-on agents, meeting scribes, live translation), constant-memory streaming is the architecture that survives contact with a weekend; if your workload is a sprint with occasional giant prompts, you want prefix caching and a big KV budget instead. Different problems, opposite fixes — both are about never paying for the same context twice.

Harness improvements

Ollama v0.40.0-rc0 is the week’s architecture news: on Apple Silicon, supported architectures now run on MLX by default — pull and run, no engine flags. The version jump from 0.34 to 0.40 signals the scale of the runtime swap, and it completes a two-step consolidation that started when Ollama made GGUF creation require llama.cpp tooling (16 Sep): llama.cpp owns conversion, MLX owns serving on Macs. For anyone running Ollama on an Apple Silicon box, the practical effect is simpler installs and MLX’s memory handling by default; watch the pre-release for which architectures get enabled before the final tag.

llama.cpp b11175–b11193 carried two stories. First, PrismML’s Hadamard campaign finished its sweep: Metal FWHT kernels for block widths above 512 (#29095) merged, on top of the CPU and Metal F16 kernels logged earlier this week — the ternary-rotation path is now substantially mainline even though the custom 142/143 type IDs remain fork-only. Second, a quiet but consequential merge: llama_prec_policy with a model-driven W4A4 path (#24364) — weight-4-bit/activation-4-bit matmul is the precision class NVFP4 hardware cares about, and an explicit policy hook means quant authors can declare intent instead of users guessing per-model flags. And on 26 Sep the Hadamard rotation picked up its third backend: CUDA F16 FWHT (#29096) merged, so the ternary-rotation kernel now runs on CPU, Metal and CUDA alike. Also in the window: Metal attention kernels split into per-dtype libraries (#29329), tiled CPU mul_mat for k-quants (#27851), OpenCL A8 Q8_0 dp4a kernels, and SYCL sparse attention (#28796).

oh-my-pi shipped v18.3.1 and v18.3.2 (25–26 Sep) on the heels of 24 Sep’s breaking minor: live steering for Codex WebSocket transports and GPT-6 streams (queued messages delivered mid-response), Anthropic slow-mode handling for subscription accounts with server-paced retries, a real fix for capped thinking requests — capped Claude calls now get their effort’s thinking budget on top of the requested output instead of ending at max_tokens with no answer — a ctx.agent extension context so tools can tell top-level agents from subagents, and usage-limit wrap-up tracking that tells the agent to finish when neither low-priority nor extra usage will continue. v18.3.3 and v18.3.4 (26–27 Sep) kept going: live steering generalized beyond specific transports, a unified predictive-text engine (N-gram, SmolLM2-135M, and macOS native providers, with a cross-process daemon and an omp predict eval command), a real fix for agents that loop for hours burning the whole output limit on reasoning — length-stop retries now tell the model its reasoning was discarded and to act in smaller steps — and the task tool’s complexity field replaced by a solutionSpace description that auto thinking uses to pick effort, so mechanical bulk work no longer inflates thinking depth.

Pi v0.87.0/0.87.1 (21–22 Sep) shipped two extension-facing changes worth knowing: canonical session context — ContextEditEntry lets extensions edit model context without rewriting history, and context_with_system enables per-request system-message transformation — plus per-model cache-safe image-input resizing limits. The user-facing headline is frontier-model plumbing: Claude Opus 5.5 with adaptive thinking and a 1M window, GPT-6 Sol/Luna, and Grok 4.7 as the xAI default. Hermes, meanwhile, is quiet since v0.21.5 (24 Sep).

Tape, read this morning from the HuggingFace API: antirez’s DeepSeek-V4.1-Flash GGUF 939,835 → 990,509 (+50k in a day — a week from the million, it seems); the V4.1-Flash base → 640,577; Ternary-Bonsai-2 → 3,247,527 (batch caveat applies); GSQ+RCO 27B → 1,560,929; MiniCPM5-2B → 726,518; Edge0’s 35B-A3B → 79,377 with 3,559 likes; Xing4.0-29B → 43,947, and its like count jumped to 1,741 — people are bookmarking a model they haven’t run yet, the same tell MiMo showed last week. The MiMo counters keep grinding: Pro-RL 42,062 → 74,497 (75× the gate-crossing number in four days), Flash-RL 20,473 → 23,000, the 9B distill 6,652 → 7,905, bartowski’s 9B GGUF 44,427 → 57,826. And the streak that has become its own datapoint: unsloth still has no DeepSeek-V4.1-Flash GGUF — day 19, re-verified this morning (their newest DeepSeek shelf item remains dated 31 August) — while antirez’s single-maintainer build closes on a million.

Download counts from the HuggingFace API today (batch-updated aggregates — likes tick live, downloads move in batches). Benchmark tables are vendor-reported and labeled as such. No private infrastructure referenced — hardware classes only.