LFM2 24B A2B: fast smoke test, zero workflow wins
LFM2 24B A2B passed a trivial JSON smoke test, then failed every current Local Model Bench paperwork case. The issue was not speed. It was task completion: wrong audit logic in scan cases, and no valid artifacts in agentic workflow cases.
LFM2 24B A2B is an interesting local candidate on paper: a 24B total-parameter mixture-of-experts model with roughly 2B active parameters, pitched for low-latency local use.
That makes it a good fit for the question Local Model Bench actually asks: not whether a model can answer a toy prompt, but whether it can survive private paperwork, messy folders, and exact final artifacts. On that version of the test, LFM2 did not survive.
Where it worked
- Fast local responses in LM Studio.
- Passed a trivial JSON smoke test.
- Produced syntactically parseable JSON in the invoice-image cases.
Where it failed
- Zero resolved cases across the current practical suite.
- Zero core passes across both scanned-paperwork and workflow cases.
- Approved invoices that should have gone to review.
- Used filenames instead of visible document IDs.
- Failed the agentic file-artifact workflow completely.