Local Model Bench
Local Model Bench

Can local models survive private desktop work?

The practical benchmark for local LLMs doing synthetic paperwork, messy local folders, hidden oracles, visible outputs, and one constrained City Plan SVG sanity check.

Mistral Small 4 City Plan SVG previewMistral Small 4gemini-3.5-flash City Plan SVG previewgemini-3.5-flash
88.9%Top practical
72.2%Top local model
9Scored cases
31City SVG runs
Infographic explaining the Local Model Bench workflow from synthetic documents to model artifacts and resolved, core pass, or fail outcomes
scoredThe Paperwork Trial

Synthetic invoice PNG scans plus bank exports, vendor records, purchase orders, and exact audit-result oracles.

23 runs · 23 modelsscoredPaperwork Workflow

Synthetic messy intake and email-attachment workflows with generated scans, protected sources, normalized artifacts, payment remapping, and hidden oracles.

23 runs · 23 modelsdiagnosticPaperwork Text-Only

The same generated invoice cases, but with normalized text extracts instead of image input. This separates bookkeeping logic from document vision.

27 runs · shown as leaderboard modevisual sampleCity Plan SVG

A city-plan SVG prompt with roads, blocks, and 3D or isometric buildings. Valid vector output, no Markdown excuses.

31 runs · not part of overall score

City Plan SVG Outputs

A small visual sanity check: standalone SVG only, city blocks, roads, and 3D or isometric buildings.

open gallery
Mistral Small 4 city plan SVG output
Mistral Small 4review · 2/3 checks
gemini-3.5-flash city plan SVG output
gemini-3.5-flashpass · 3/3 checks
Chrome Gemini Nano city plan SVG output
Chrome Gemini Nanoreview · 2/3 checks
gpt-oss-20b city plan SVG output
gpt-oss-20bpass · 3/3 checks

Latest Model Notes

Test reports that compare model positioning with observed benchmark behavior.

all notes
Gemma 4 12B Unified via LM Studio note infographic
run logGemma 4 12B did not close

10% practical · 0/5 resolved · 1/5 core

MiniMax M3, MiniMax M2.7 note infographic
field noteMiniMax M3: long-loop failure

MiniMax M3 agent-loop failure · M2.7 builder / M3 reviewer workaround

MiniMax M3 Free via OpenCode Zen note infographic
run logMiniMax M3 Free leads paperwork

88.9% practical · 8/9 resolved · 8/9 core

Qwen3.6 27B MTP note infographic
runtime noteQwen3.6 MTP: speed versus artifacts

Methodology guide and benchmark context

Overall Leaderboard

Top public comparison rows. Practical Score = 50% resolved cases + 50% core passes across the current v1 paperwork suite. Local LM Studio runs were executed on a Mac mini M4 with 64 GB unified memory.

full benchmark tablesmethodology

Generated scans and messy workflow folders. This is the main public comparison score.

10 of 23 shown
OKnear miss / core passfail

Swipe sideways to see all columns.

RankModelTypePracticalResolvedCoreTriedCase Matrix
1
opencode/minimax-m3-freedetails
api cheap88.9%8/98/99/9
P01OKP02OKP03FAILP04OKP05OKW04OKW05OKW06OKW07OK
2
OpenAI GPT-5.5 (Codex CLI)details
reference83.3%7/98/99/9
P01OKP02COREP03OKP04OKP05FAILW04OKW05OKW06OKW07OK
3
OpenAI GPT-5.4 Mini (Codex CLI)details
reference77.8%7/97/99/9
P01OKP02OKP03OKP04OKP05OKW04FAILW05FAILW06OKW07OK
4
qwen3.6-27bdetails
local72.2%5/98/99/9
P01OKP02COREP03FAILP04OKP05OKW04OKW05COREW06OKW07CORE
5
gemma-4-26b-a4bdetails
local61.1%4/97/99/9
P01OKP02OKP03FAILP04OKP05OKW04FAILW05COREW06COREW07CORE
6
qwen3.6-35b-a3bdetails
local38.9%1/96/99/9
P01COREP02FAILP03FAILP04COREP05OKW04COREW05COREW06COREW07FAIL
7
qwen3.6-flashdetails
api cheap33.3%0/96/99/9
P01COREP02COREP03FAILP04COREP05COREW04COREW05COREW06FAILW07FAIL
8
gemma-4-e4bdetails
local27.8%2/93/99/9
P01OKP02OKP03FAILP04COREP05FAILW04FAILW05FAILW06FAILW07FAIL
9
gemma-4-31b-itdetails
local27.8%0/95/99/9
P01COREP02COREP03FAILP04FAILP05FAILW04FAILW05COREW06COREW07CORE
10
gemini-3.1-flash-litedetails
api cheap27.8%0/95/99/9
P01FAILP02COREP03FAILP04FAILP05COREW04FAILW05COREW06COREW07CORE
11
gemini-2.5-flashdetails
api cheap27.8%0/95/99/9
P01COREP02COREP03FAILP04FAILP05FAILW04FAILW05COREW06COREW07CORE
12
Qwen3 VL 30B A3Bdetails
api cheap27.8%0/95/99/9
P01COREP02COREP03FAILP04FAILP05FAILW04FAILW05COREW06COREW07CORE
13
qwen3-vl-32b-instructdetails
api cheap22.2%0/94/99/9
P01COREP02COREP03FAILP04FAILP05FAILW04FAILW05COREW06COREW07FAIL
14
Seed 2.0 Minidetails
api cheap22.2%0/94/99/9
P01COREP02COREP03FAILP04FAILP05FAILW04FAILW05COREW06FAILW07CORE
15
Mistral Small 4details
api cheap22.2%0/94/99/9
P01COREP02COREP03FAILP04FAILP05COREW04FAILW05COREW06FAILW07FAIL
16
mistral-small-3.2details
local16.7%0/93/99/9
P01COREP02COREP03FAILP04FAILP05COREW04FAILW05FAILW06FAILW07FAIL
17
ministral-3-14bdetails
local16.7%0/93/99/9
P01COREP02COREP03FAILP04FAILP05FAILW04FAILW05FAILW06FAILW07CORE
18
gemma-4-12bdetails
local5.6%0/91/99/9
P01FAILP02COREP03FAILP04FAILP05FAILW04-W05-W06-W07-
19
gemma-4-e2bdetails
local0.0%0/90/99/9
P01FAILP02FAILP03FAILP04FAILP05FAILW04FAILW05FAILW06FAILW07FAIL
20
qwen3-vl-8b-instructdetails
local0.0%0/90/99/9
P01FAILP02FAILP03FAILP04FAILP05FAILW04FAILW05FAILW06FAILW07FAIL
21
qwen3-14bdetails
local0.0%0/90/99/9
P01FAILP02FAILP03FAILP04FAILP05FAILW04FAILW05FAILW06FAILW07FAIL
22
nemotron-3-nano-omni-30b-a3b-reasoning:freedetails
api cheap0.0%0/90/99/9
P01FAILP02FAILP03FAILP04FAILP05FAILW04FAILW05FAILW06FAILW07FAIL
23
ministral-3-3bdetails
local0.0%0/90/99/9
P01FAILP02FAILP03FAILP04FAILP05FAILW04FAILW05FAILW06FAILW07FAIL