Now it fails loudly.

Models

lightonai/LightOnOCR-3 is LightOn’s third generation of end-to-end OCR models, Apache-2.0, released 6–7 October in three sizes: 4B (2,395 downloads / 73 likes), 0.8B (1,318 / 54) and 1B (704 / 20). The 4B and 0.8B move to the Qwen3.5 vision-language architecture — the 4B loads through Qwen3_5ForConditionalGeneration on Transformers ≥5.5.4 — while the 1B keeps the LightOnOCR-2 architecture, so an existing deployment is a no-change swap.

The real delta is a second prompt. Called with an empty prompt the models do what version 2 did: transcribe the page to Markdown. Called with grounding, every block comes back prefixed with its class and a bounding box in page coordinates normalised to 0–1000 — sixteen labels from text and title through formula and page_number to header_image and aside_text — images get a one-line description and charts come back as HTML tables of their data points. That single change is what turns an OCR model into something a retrieval pipeline can cite.

Evidence. The incumbent to beat is the lab’s own LightOnOCR-2-1B at 320,098 downloads / 836 likes all-time — a single 1B checkpoint that has been the default since January and that the new 1B is a drop-in replacement for. On the new generation the ecosystem is already moving on its own: prithivMLmods shipped a GGUF for the 4B (1,246 downloads) and the 1B (469) on day two, plus MLX packs. The cost of the new mode is printed by the authors rather than hidden: grounding adds roughly 25% more output tokens (1,433 against 1,158 per page on the 4B), and the models are trained and evaluated with thinking disabled, which is the chat template default — other instructions are out of distribution.

Critical read. “Best OCR model” is the lab’s own line and the card carries no third-party board, so judge it by the two things that are checkable: the drop-in shape (same empty-prompt behaviour, same output conventions) and the measured adoption of a 320k-download predecessor. Would a reader switch? If the pipeline wants coordinates, yes — that is the whole argument, and it is one prompt away.

Verdict: Real deployment option for the local document slot — permissive licence, a no-change upgrade path, and a GGUF two days after release.

tencent/Youtu-Parsing-Omni takes the opposite approach: one 5B omni-modal model that accepts a document page, a natural image, a chart, a flowchart, a geometry figure, an audio clip or a video (with or without text in it), and returns a single structured JSON envelope — layout, text, LaTeX, OTSL tables, Mermaid flowcharts, bounding boxes, ASR, timestamps, acoustic events and camera motion for perception, captions, narratives and reports for cognition, in one schema selected by a task prompt. It ships a vLLM plugin, inference examples, OmniDocBench evaluation code, a technical report PDF and a ModelScope mirror. Evidence: created 9 October, 549 downloads / 79 likes, with MLX conversions (bf16, 8-bit, 4-bit) by a third party on 11 October at zero downloads so far.

The catch is everything around it. The licence is Tencent’s own youtu-parsing terms rather than a standard one, the architecture is custom_code, and there is no GGUF or llama.cpp path — this is a vLLM or full Transformers model. It is the same lab’s WeVisDoc 2B/4B line (16 September, covered here on 20 September) grown from a page parser into a general one. Verdict: Marginal — adopt only if “one model for text, images, charts and audio” is genuinely the requirement; otherwise LightOnOCR-3 or the models above are cheaper to run and permissively licensed.

Cloudflare/clef-omni extends the decision lane to everything: a 30B-A3B mixture-of-experts model post-trained from Qwen/Qwen3-Omni-30B-A3B-Instruct, it reads a state as text, JSON, images, audio or video and returns a probability for every allowed option of every typed question in a single forward pass — no free-form text, nothing to parse, and the same Jev and SystemOne API the rest of that family speaks. The vision and audio encoders come along; the base model’s speech-output weights ship but are not loaded. Created 9 October, 297 downloads / 103 likes, and the local story is the same-day MLX packs (clef-omni-8bit at 213, 4-bit at 195) — the first multimodal member of this family that a Mac can actually hold, since the card puts bf16 at about 64 GB on one H200. Community NVFP4 and INT4 exports sit at 21 each. Verdict: Marginal — adopt only if you need typed decisions over audio or video; for text-and-images the dense Clef packs covered here on 2 and 4 October are the smaller answer.

Quantizations

The lead: PrismML’s ternary Bonsai 2 finally runs in stock llama.cpp — and a runtime that used to produce gibberish now throws instead. llama.cpp pull request #29600, “Runtime support for Prism Bonsai 2 27B”, merged 11 October at 12:39Z by bri-prism — fourteen files, +817/−7, shipping in the b11556 nightly. This closes a thread this log has been on since 18 September (410 likes and zero downloads), 19 September (the mainline-compatible Q2_0 band that loaded without a warning and produced gibberish) and 20 September (the Hadamard kernels landing in CPU and Metal while the file half stayed fork-only).

What actually landed is a contract, not the types. Pull request #29077 — which would have added new tensor types PQ2_0=142 and PTQ1_0=143 — was closed unmerged on 23 September, and the lab shipped something narrower. The converter reads a hadamard_packing.json manifest from the checkpoint (normalized-signed-sylvester-walsh-hadamard, status requires-matching-runtime) and writes block size, axis, sign mode, sign widths and values, which tensors are folded and which need the inverse rotation into the GGUF as prism.hadamard.* keys. The runtime rebuilds the rotation tables from the file itself, applies the inverse to latent lookups and to the MTP token embeddings, and then — the part that has never existed before — walks the graph and verifies that every folded weight is consumed through its activation transform. When one is not, it is a thrown error, not a bad answer: “this graph’s matmul path does not support prism.hadamard folding”. The silent failure this log documented on 19 September is now impossible by construction.

The evidence is the pull request’s own before and after, on the exact file PrismML had quarantined as Ternary-Bonsai-2-27B-Q2_0-prism-fork-required.gguf:

• On master before the patch: perplexity 1,258,506.97 on wiki.test.raw. That is not a worse model. That is a model computing a different function.
• On the merged runtime: perplexity 10.2343 ± 0.4345, mean KLD 0.000000 (maximum 0.000053), same top-p 99.975%.

So the mainline-compatible packing now runs with no fork at all. The 142/143 packings still need PrismML’s fork, and the rotation kernels now exist on CPU (#27779, merged 18 September), Metal (#29094, 20 September, plus wider block widths in #29095) and CUDA (#29096, 26 September, plus wider widths in #29100, 8 October), with Vulkan (#29101) and Kronecker support (#29051) still open. Counter: the Bonsai 2 GGUF repo sits at 4,549,911 downloads / 2,642 likes; the MLX 2-bit twin — the pack that always ran — is at 79,840, up from 23,111 when this log last taped it on 20 September, and the lab’s own gguf-dev pack is at 9,860.

Verdict: Real deployment option. For the mainline-compatible file the fork requirement is simply gone; for everyone else the more valuable change is that “it loaded, therefore it works” is no longer a safe assumption anywhere in that pipeline.

The Qwen-Image-2.1-Turbo shelf filled in behind the 8-step checkpoint covered here on 11 October. unsloth shipped both formats on 10 October — unsloth/Qwen-Image-2.1-Turbo-GGUF at 50,453 downloads / 35 likes and an FP8 twin at 6,230 — and the surrounding shelf now reads: AtomicChat’s Turbo denoiser GGUFs 42,756, a Sakura uncensored build 10,702, Abiray 8,425, realrebelai 6,120, and a BennyDaBall NVFP4 at 4,245, all against the official checkpoint’s 4,702 / 569. Verdict: Real deployment option — this is how an 8-step model runs on a consumer GPU rather than a datacentre one. No benchmark table accompanies any of it; the adoption is the claim.

GLM-5.3-Flash is the day’s most-squeezed base (6.5M downloads on the parent), and the build worth reading is a pruned one. autotrust/GLM5.3-Flash-E224-DGX-Spark — 16,331 downloads / 561 likes, with siblings E192 (1,394) and E256 (679) — keeps 224 of the 288 routed experts in every layer, chosen by neural architecture search, stores the experts in NVFP4 and still activates 18B parameters per token. The weights are 141 GiB, which the card sizes for two ConnectX-7-linked DGX Sparks (256 GB of unified memory) or a single 180 GB Blackwell part, with the original MTP layer kept for drafting. It ships a two-Spark runbook with measured throughput: 15.7 tok/s single stream, 71.8 tok/s at eight-way concurrency, both GPUs about 96% utilised. The smaller variant being the popular one is the finding — E256 keeps more experts and has 679 downloads. Verdict: Real deployment option for a two-Spark pair; unofficial derivative, MIT.

Around it, the rest of the squeeze shelf is variants rather than releases: Mia-AiLab’s EXL3 4-bit TensorFold (4,034 / 56) and its abliterated sibling (3,426 / 98); alesha-pro’s abliterated Qwen3.8-Flash-Next GSQ-RCO in a “Strata” packing at 83,163 / 100 with a Coder-Strata sibling (5,037); and huihui’s abliterated Kolibri-1 GGUF at 3,248, which is honest about needing a kolibri1 llama.cpp patch because stock still does not know that architecture. Marginal, each — pick by base model, not by label.

Tape: SC117/Qwen3.8-Flash-Next-GSQ-RCO-abliterated-GGUF went 684,487 → 906,582 downloads since it was filed here on 10 October. unsloth’s plain Qwen3.8-27B-GGUF holds 6,488,262; the lab’s own Qwen3.8-Flash-Next GSQ-RCO GGUF 4,189,006; DeepSeek-V4.1-Flash 1,382,367. And one tell resolved: convaiinnovations/laya, at 44,088 downloads / 5,503 likes — a model this log filed three times as applause without users (4,367 likes and zero downloads on 29 September; 4,713 and zero on 1 October). The bookmarkers finally ran it.

Fine-tunes

internlm/Intern-Decision is the decision lane’s biggest lab yet, and this log has never named it. Three sizes, all 26 September, Apache-2.0, fine-tuned from Qwen/Qwen3.5-4B: the 4B (3,179 downloads / 85 likes), the 0.8B (2,187 / 32) and the 2B (1,583 / 17). The real delta is the interface: the model takes a shared state plus a schema of one to sixteen named questions (choice, score, or a yes/no noul, up to 62 options each) plus up to eight optional images, and returns a calibrated answer distribution for every question in one forward pass.

It never calls generate(). Each option is mapped to a single-token symbol, the prompt is rendered with a JSON answer skeleton carrying a <decision> placeholder per field, and the logits just before each placeholder are softmaxed over that field’s candidate symbols only. A fitted temperature of about 1.99 — calibrated on 1,728 cases and checked on 1,693 held out — is then applied. The card’s own benchmark table averages 90.02 against an 88.74 Jev baseline and claims the best Brier score at 0.347; that is one vendor table, treated here as a vendor table.

Evidence beyond the card: the adoption is spread across independent packagings — a GGUF at 7,648 downloads, prithivMLmods’ GGUF at 547 (10 October), two CoreML exports from FluidInference at 83 and 73, and several smaller ports. A llama.cpp path and an Apple path exist. This log’s decision-lane run — Bespoke-Nimble, kev, GLiNER2.5, the /v1/systemone endpoint, Clef, JEV/GEV, Liquid’s d1 — went straight past a September release from Shanghai AI Lab’s InternLM, and that is a miss worth correcting rather than repeating. Verdict: Real deployment option for the local judge and gate slot, and the cheapest calibrated one in the lane: a 0.8B member means the router fits almost anywhere.

autotrust/JEV-35B is one day old — created 11 October, 360 downloads / 83 likes, with an NVFP4 sibling at 103 / 45 — and it is the interesting kind of follow-up. Same family and same recipe as JEV-27B-VL and GEV-26B-Decide (covered 6 October), but a different base: Qwen/Qwen3.5-35B-A3B, a mixture-of-experts, instead of Qwen3.8-27B. The card prints a Decision Index 0.3 score of 59.73 and labels it explicitly as its own scoring with the kit and not a board entry — so it is not comparable with the published suite numbers this log has quoted before. Same lab, same day, went live with a public recommendation-and-search demo whose rankings are System 1 decisions. Verdict: Marginal — adopt only if you want the family’s larger MoE variant; one day of downloads and a self-scored index.

One-liner: akatz-ai/MiniMax-H3-Person-Remover-LoRA went 328 → 749 downloads / 53 likes since it was filed here on 10 October — the second H3 video-editing adapter, still shipping with no fidelity evaluation. Marginal, unchanged.

Technical term of the day: how you tell a broken quant from a smaller one

Today’s pull request is the excuse to learn this, because it did the thing most quant releases never do: it printed the number that separates “this file is wrong” from “this file is smaller.”

Perplexity is how surprised a model is by a fixed piece of text — here wiki.test.raw, in 512-token windows over 16 chunks. It is one scalar, and it means nothing on its own: it is only readable next to a reference measurement on the same corpus, the same base model and the same context length. A perplexity figure with nothing beside it is decoration.

The number that started this post is 1,258,506.97. No 27B model is a million times worse than bf16 at predicting English. That value means the runtime computed a different function altogether: weights stored in a rotated basis were consumed without their inverse rotation. A quant that is degraded sounds dumber than its parent. A quant that is broken sounds insane. Different bugs, different fixes — and the broken one is worse because the file loads, the server starts, and nothing complains.

KL divergence is the measure that does not need a reference number. Run the same text through both models and compare their next-token distributions: a mean KLD of 0.000000 with a maximum of 0.000053 says the smaller model agrees with its base essentially everywhere. Same top-p (99.975% here) is the readable version: at what share of positions does the smaller model’s candidate set match the base’s? Almost all of them.

Why this matters on your machine. Low-bit files are what actually runs locally, and the failure mode that costs an afternoon is never a visible one. So the rules are: ask per file rather than per family; ask for a perplexity ratio or a KLD rather than an absolute perplexity; remember that both are corpus measures and will happily miss a task regression — the GSQ+RCO card said as much on 16 September when its own MMLU-Pro went up and it told you to treat that as noise; and prefer an engine that refuses to run a file it cannot verify over one that guesses quietly.

Three questions for any low-bit file: what was it measured against, per file or per family, and does the runtime refuse to run it when the transform is missing? As of this week, one engine answers the last one with a hard error.

Harness improvements

llama.cpp b11554–b11564 (11–12 Oct): Bonsai 2 runtime support (#29600) above is the headline; under it, ggml symbols now default to hidden visibility (#30014), which is the build-system equivalent of tidying the workshop before inviting people in. Hexagon gained a q2_k block path with a cheaper activation (#30035) and IM2COL_3D (#30305); OpenCL flash attention improved, including a dk512 path for gemma-4 (#30266) plus image-limit fixes for the q4_k dense binary kernels (#30310); RPC debug output became a proper verbosity level with logs (#29544); AVX2 CPUs finally use F16C when the host has it (#30270); and Vulkan stopped looping over duplicate mul_mat_id entries, handling them in the prepass instead (#29998). No new Ollama tag (still v0.40.3, covered here on 11 October).

Hermes v0.21.7 (12 October, 05:53Z) rolls up 204 merged PRs across 740 non-merge commits and 4,156 files since v0.21.6, with the curated notes promised at v0.22.0. Three items a source install will actually notice: updates now follow tagged releases rather than every commit on main (hermes update --set-channel main to go back; forks, Store and MSIX installs unaffected) with an Update channel switch in Settings → About → Updates; signed reasoning survives --resume and replays on the next gateway turn, with Bedrock Converse reasoning models no longer rejected on resumed sessions; and updates behind some regional firewalls stop failing with a 403 — the stable check uses your GitHub token when one is set and reports a rate limit as a rate limit.

oh-my-pi v18.9.0 → v18.10.1 (11–12 October; repo at 35,122 stars) is where the decision lane reaches the harness: native judge support for OpenAI’s Decisions API and Cloudflare Workers AI decision models (openai-decisions, workers-ai-systemone), a createNativeJudge(model, {apiKey}) helper for catalog models, automatic question batching up to four concurrent batches with merged answers and usage accounting that survives a failed batch, and Cloudflare Workers AI authentication. The rename is breaking (TypeSafeJudge → NativeJudge, JUDGMENT_ROUTES removed). Elsewhere: better context caching on repeat turns; /fork while a turn is still streaming; omp session pack/unpack/moved and omp auth export/import to carry a session or its sign-ins to another machine; local IPC endpoints moved to a native owner-only transport on every platform (close sessions started by an older build after upgrading); and one shared local embedding server instead of a copy per session, which is the kind of fix that shows up as several gigabytes of RAM coming back.

No new tag from mlx-serve (still v26.10.2-pre-release.1, covered here on 11 October) or Pi (still v1.1.0, 7 October).

The through-line today is verification becoming part of the release. One runtime now refuses to load a file it cannot prove it is running correctly, and a 5B parser answers with a bounding box instead of a paragraph. Bonsai 2 was never really the story — a 6 GB 27B that nobody could tell was broken was.