Local Model Bench

privacy

Is your local LLM actually local? We watched the wire

We packet-captured a Mac mini while Ollama, LM Studio and llama.cpp sat idle, loaded models and answered prompts. No packet we could attribute to a runtime carried anything out. What did cross the wire: update checks, a resolve call — and an OS that never stops talking.

serverbench

The dashboard said all green — the local models could fix it, but didn't finish

Serverbench S06 puts agents on a Docker-broken web server where monitoring says green and the page still answers 200 — from cache. Two API models resolved; the three local models each stopped at a different point short of done.

runtime

When LM Studio and Ollama send different prompts for the same model

Same weight file, same messages, temperature 0: Ollama reported two more input tokens than LM Studio on phi-4, and the two runtimes returned different text. On Granite the counts matched. The cheapest check is one number you probably do not log.

paperwork-v3

Kimi K3 Max tops paperwork-v3 via Devin CLI: SWE-2 beats GPT-6 and Opus

Seven frontier model configs ran all nine paperwork-v3 cases through Devin CLI's agent loop. Kimi K3 Max takes first place on the board (94.4%), SWE-2 Max second (88.9%), ahead of GPT-6 Astra Max, Claude Opus 5.5 Max and Grok 4.7 XHigh.

paperwork-v3

The no-think leaderboard that wasn't: 7 rows reasoned anyway, re-measured Gemma-4 falls 61.1% to 11.1%

Seven paperwork-v3 leaderboard rows sent every no-think flag and produced reasoning tokens anyway, up to 28,821 per model. Re-measured under verified suppression, Gemma-4 26B A4B drops from 61.1% to 11.1% practical.

paperwork-v3

Bonsai 2 27B on paperwork-v3: 0/9 resolved without thinking, 6/9 with (72.2% practical)

Bonsai 2 27B is PrismML's ternary successor on a Qwen3.8-27B backbone. On paperwork-v3 it scored 0/9 strict without thinking and 6/9 strict, 7/9 core (72.2% practical) at medium thinking effort — a 44.4-point mode split.

paperwork-v3

Qwen3.8 27B on paperwork-v3: 3/9 closed, down from Qwen3.6 27B's 5/9 on the same nine cases

Qwen3.8 27B is the successor to Qwen3.6 27B, which currently leads the Local Model Bench local row on the nine-case paperwork-v3 suite. On this generation Qwen3.8 closed 3/9 strictly and 7/9 core-oracle for a practical score of 55.6%. Qwen3.6 27B closed 5/9 strictly and 8/9 core-oracle for 72.2% on

Refining paperwork-v3 based on the Bonsai 27B run — Local Model Bench

What we found going back through the Bonsai 27B verify-output files case by case, and the bench-side changes that follow from each finding.

Bonsai 27B on paperwork-v3: 2/9 resolved, 4/9 core (33.3% practical)

prism-ml/bonsai-27b scored 2/9 strict and 4/9 core on paperwork-v3 (33.3% practical), ranking #10 on the leaderboard. Like Muse-Glimmer, it required an 8,000 max_tokens ceiling, but its llama.cpp template ignored reasoning flags. Failures were shape and evidence errors, not token stalls.

Muse Glimmer: 6/9 on paperwork-v3. The first run returned empty content.

Muse-Glimmer scored 6/9 strict and 8/9 core on paperwork-v3 (77.8% practical), ranking as the top local model. Its first run returned empty output after 2,397 reasoning tokens exhausted the default 2,400-token limit. Raising max_tokens to 8,000 resolved the bottleneck.

Methodology

Local Model Bench tests whether local models can complete private, boring, failure-prone work: synthetic paperwork, messy folders, multi-step workflows, and one constrained visual output check.

Gemini 3.8 Flash through Antigravity: 9 of 9 hidden-oracle passes, but visible_checks fail on every image case

27/27 hidden-oracle passes — the headline. Visible_checks fail on every P-case at every effort: strict 4/9 (was 9/9), practical 72.2% (was 100%), rank #6 tied with qwen3.6-27b on score.

field-note

Reasoning budget beats raw size on local math

A 27B dense model with reasoning on solves all 36 math tasks. Reasoning budget beats raw size on local math. Tested 8 model configs across 9 stress dimensions.

editorial

Apple Foundation Model on Mac mini M4 Pro: what one round of probing taught us

Apple's on-device foundation model (AFM 3 Core, 3B dense) is closed-weight, framework-bound, sealed in an Apple Cryptex. We probed it on Mac mini M4 Pro 64GB anyway — speed, Paperwork V3, the audit pattern. Editorial-layer note, not a leaderboard row.

run log

Gemma 4 12B Unified read pieces, but did not close the work

Gemma 4 12B Unified was added as a local LM Studio run on the Mac mini M4. It found parts of the paperwork, but the final result was harsh: 10% Practical Score, 0/5 resolved generated-image cases, workflow loops, and no parseable City Plan SVG.

field note

MiniMax M3 looks broken in long agent loops

MiniMax M3 can score well in bounded benchmark work and still break in long autonomous agent loops. This field note is about that second failure mode: many files, generated assets, validation, and a final report that never cleanly arrives.

run log

MiniMax M3 Free leads the paperwork benchmark, with one ugly SVG caveat

MiniMax M3 Free is the top Practical Score on Local Model Bench: 8/9 resolved and 88.9% across the current paperwork suite. That does not make it a universal agent winner. It is a provider-routed benchmark result, and it still produced no parseable City Plan SVG after a 600 second timeout.

runtime note

Qwen3.6 MTP worked. The artifact still matters.

Qwen3.6 MTP is the local model people are arguing about right now. In our Mac mini checks, LM Studio showed a small speed gain and llama.cpp showed a tiny gain with the right flags. In a five-case text-only paperwork run, MTP solved 5/5 strict while non-MTP solved 4/5 strict. The speed difference wa

methodology

Best local LLMs for private document work

The useful question is not which local LLM sounds smartest in chat. It is which one can survive private desktop work: scans, CSVs, messy folders, revised invoices, evidence paths, proof codes, and exact final files.

methodology

OCR is the wrong word for this benchmark

Local Model Bench uses generated invoice images, but it is not mainly an OCR test. The hard part starts after the page has been read: selecting the right files, rejecting stale documents, reconciling CSV rows, citing evidence, writing artifacts, and passing a hidden oracle.

methodology

Benchmark-smart is not the same as work-ready

Public benchmarks are useful. They are also clean. Real work is full of stale files, wrong attachments, hidden assumptions, and final artifacts that actually have to exist. A model can be good at the leaderboard shape and still be bad at finishing the job.

run log

New model candidates, same paperwork problem

Three newer candidates were added to Local Model Bench. The useful signal was not that every model failed. It was where they failed: proof codes, evidence paths, workflow closure, and provider constraints that should not be confused with model capability.

benchmark note

Bigger was not always better in the paperwork benchmark

In the current nine-case paperwork suite, two larger-sounding local model choices produced fewer finished cases than leaner alternatives. That does not prove small models are better. It shows that parameter count and model labels a

runtime note

Ollama vs. LM Studio: speed is not the whole benchmark

We ran the same Mistral Small family through LM Studio and Ollama on a Mac mini M4. Ollama was slightly faster in the tiny runtime check, but both runtimes exposed the same practical problem: the model still failed exact paperwork closure.

benchmark note

Gemini 3.5 Flash can draw SVG. The runner was the problem.

A first Gemini 3.5 Flash SVG run looked like a model failure. It was not. The OpenAI-compatible API path let hidden thinking consume the output budget, so the model returned planning fragments instead of a complete SVG. The native Gemini API with thinkingBudget: 0 produced a valid City Plan SVG and