LMB Editorial - Local Model Bench

Local Model Bench

Sign in Subscribe

LMB Editorial

Flat editorial illustration: a vintage polygraph shows a perfectly flat line beside a silent mini computer; through the window a second chart runs wild

Is your local LLM actually local? We watched the wire

We packet-captured a Mac mini while Ollama, LM Studio and llama.cpp sat idle, loaded models and answered prompts. No packet we could attribute to a runtime carried anything out. What did cross the wire: update checks, a resolve call — and an OS that never stops talking.

Serverbench S06 All Green score card: seven models on the incident case, two resolved, three local models each stopping at a different step.

The dashboard said all green — the local models could fix it, but didn't finish

Serverbench S06 puts agents on a Docker-broken web server where monitoring says green and the page still answers 200 — from cache. Two API models resolved; the three local models each stopped at a different point short of done.

A roll of punched paper tape splitting into two nearly identical strips whose hole patterns diverge in a few positions

When LM Studio and Ollama send different prompts for the same model

Same weight file, same messages, temperature 0: Ollama reported two more input tokens than LM Studio on phi-4, and the two runtimes returned different text. On Granite the counts matched. The cheapest check is one number you probably do not log.

Score card: paperwork-v3 board via Devin CLI — Kimi K3 Max 94.4%, SWE-2 Max 88.9%, Luna High + Opus 77.8%, Grok 4.7 + Astra Max 72.2%, Luna Max 61.1%

Kimi K3 Max tops paperwork-v3 via Devin CLI: SWE-2 beats GPT-6 and Opus

Seven frontier model configs ran all nine paperwork-v3 cases through Devin CLI's agent loop. Kimi K3 Max takes first place on the board (94.4%), SWE-2 Max second (88.9%), ahead of GPT-6 Astra Max, Claude Opus 5.5 Max and Grok 4.7 XHigh.

A hand-held magnifying glass over a printed leaderboard under a desk lamp, revealing glowing hidden text inside one row

The no-think leaderboard that wasn't: 7 rows reasoned anyway, re-measured Gemma-4 falls 61.1% to 11.1%

Seven paperwork-v3 leaderboard rows sent every no-think flag and produced reasoning tokens anyway, up to 28,821 per model. Re-measured under verified suppression, Gemma-4 26B A4B drops from 61.1% to 11.1% practical.

Bonsai 2 27B paperwork-v3 score card: 0/9 strict without thinking, 6/9 strict and 7/9 core with thinking, 72.2% practical score.

Bonsai 2 27B on paperwork-v3: 0/9 resolved without thinking, 6/9 with (72.2% practical)

Bonsai 2 27B is PrismML's ternary successor on a Qwen3.8-27B backbone. On paperwork-v3 it scored 0/9 strict without thinking and 6/9 strict, 7/9 core (72.2% practical) at medium thinking effort — a 44.4-point mode split.

Qwen3.8 27B on paperwork-v3: 3/9 closed, down from Qwen3.6 27B's 5/9 on the same nine cases

Qwen3.8 27B is the successor to Qwen3.6 27B, which currently leads the Local Model Bench local row on the nine-case paperwork-v3 suite. On this generation Qwen3.8 closed 3/9 strictly and 7/9 core-oracle for a practical score of 55.6%. Qwen3.6 27B closed 5/9 strictly and 8/9 core-oracle for 72.2% on

Local Model Bench editorial hero for Refining paperwork-v3. Title right: REFINING PAPERWORK-V3, three failure categories. Left: JSON snippets with sticky notes on the keys.

Refining paperwork-v3 based on the Bonsai 27B run — Local Model Bench

What we found going back through the Bonsai 27B verify-output files case by case, and the bench-side changes that follow from each finding.

Bonsai 27B paperwork-v3 score card: 2/9 hidden oracle, 4/9 core oracle, 4/9 strict resolved, 0/5 visible checks, 22.2% practical score.

Bonsai 27B on paperwork-v3: 2/9 resolved, 4/9 core (33.3% practical)

prism-ml/bonsai-27b scored 2/9 strict and 4/9 core on paperwork-v3 (33.3% practical), ranking #10 on the leaderboard. Like Muse-Glimmer, it required an 8,000 max_tokens ceiling, but its llama.cpp template ignored reasoning flags. Failures were shape and evidence errors, not token stalls.

Thermal-printer receipt: INVOICE-AUDIT-RUN / content: "" / finish_reason: length, rest blank. Documentaristic still-life in the LMB paper-line.

Muse Glimmer: 6/9 on paperwork-v3. The first run returned empty content.

Muse-Glimmer scored 6/9 strict and 8/9 core on paperwork-v3 (77.8% practical), ranking as the top local model. Its first run returned empty output after 2,397 reasoning tokens exhausted the default 2,400-token limit. Raising max_tokens to 8,000 resolved the bottleneck.

Methodology

Local Model Bench tests whether local models can complete private, boring, failure-prone work: synthetic paperwork, messy folders, multi-step workflows, and one constrained visual output check.

Gemini 3.8 Flash through Antigravity — invoice, email, post-its on the left; score card on the right with Hidden Oracle 9/9, Core 9/9, Strict 4/9, Visible 0/5, Practical 72.2%.

Gemini 3.8 Flash through Antigravity: 9 of 9 hidden-oracle passes, but visible_checks fail on every image case

27/27 hidden-oracle passes — the headline. Visible_checks fail on every P-case at every effort: strict 4/9 (was 9/9), practical 72.2% (was 100%), rank #6 tied with qwen3.6-27b on score.

Apple Foundation Model on Mac mini M4 Pro: what one round of probing taught us

Apple Foundation Model on Mac mini M4 Pro: what one round of probing taught us

Apple's on-device foundation model (AFM 3 Core, 3B dense) is closed-weight, framework-bound, sealed in an Apple Cryptex. We probed it on Mac mini M4 Pro 64GB anyway — speed, Paperwork V3, the audit pattern. Editorial-layer note, not a leaderboard row.