qwen2.5-vl-7b

2026-05-15T19-07-45Z_qwen_qwen2.5-vl-7b_paperwork_v3_generated_images

0% Practical score

0/1 Resolved tasks

0/1 Core tasks

Verdict

practical run result

failed

Core 0/1 · Visible 0/1 · Hidden 0/1

no failure labels

Prompt set v3_paperwork_generated_images.json

Run type local

Temperature 0

Top P 1

Missed tasks 1/1

Task Results

Compact view; open a row for prompt and raw answer

Task	Category	Score	Failed Checks	Failure Types	Prompt / Answer
generated_invoice_case_01	paperwork	0/4	audit_result_exists, visible_checks_pass, core_oracle_pass, hidden_oracle_pass	none	case files Inspect Prompt You are auditing a synthetic scanned paperwork folder. Read `README_TASK.md`, inspect the files in `scans/`, and use `bank_export.csv`, `vendor_master.csv`, and `purchase_orders.csv`. Write `audit_result.json` only. This is benchmark bookkeeping, not financial, tax, or legal advice. Important: - The scanned images are the source documents. - `Northwind Office Supply` is the customer, not the vendor. - Ignore documents that clearly say they are not invoices. - Do not invent fields that are not supported by the files. - Use visible document IDs for `ignored_document_ids`, not filenames. - Use relative paths with folder prefixes in `evidence`, for example `scans/example.png`. - Use only allowed lowercase warning codes from `README_TASK.md`. - Warning arrays must be flat arrays of strings. Workspace files available to the model README_TASK.md # Paperwork Trial v3 Calibration Case 01 This is synthetic benchmark bookkeeping. It is not financial, tax, or legal advice. Audit the scanned document folder and write `audit_result.json` only. Case ID: `P3-GEN-01` Output keys exactly: `case_id, approved_invoice_ids, review_invoice_ids, reject_invoice_ids, ignored_document_ids, total_approved_gross_cents, warnings_by_invoice, evidence, proof_code` Rules: - Treat the files in `scans/` as the source documents. - Ignore documents that are explicitly not invoices. - Match vendors by tax ID first, then by exact vendor name. - `payment_match` requires paid bank rows for the invoice whose summed amount equals the invoice gross. - `payment_short` applies when the paid bank amount is lower than invoice gross. - `under_review_stamp` applies when the scanned invoice visibly has an under-review stamp. - Approved invoices have no warnings and paid amount equals gross. - Review invoices have warnings but are not reject-level. `payment_short` and `under_review_stamp` are review-level warnings, not reject-level warnings. - Reject only if the invoice is impossible to process, uses an inactive vendor, or lacks a required PO. - Sort all invoice-id arrays ascending. - Allowed warning codes are exactly: `payment_short`, `under_review_stamp`. - `warnings_by_invoice` must include every real invoice ID and sorted warning arrays. - Warning arrays must be flat arrays of lowercase strings, never nested arrays and never prose labels. - `ignored_document_ids` must include visible document IDs from ignored non-invoice documents, not filenames. Example: use `QT-6400`, not `quote_6400_not_invoice.png`. - `total_approved_gross_cents` is the sum of approved invoice gross totals only. - `evidence` must list the relative source file paths used in stable alphabetical order, including folder prefixes such as `scans/`. - Include the relevant CSV files and every scanned document inspected in `evidence`, including ignored non-invoice scans. - `proof_code = total_approved_gross_cents + sum(numeric parts of all real invoice IDs) + 97 * total_warning_count`. Important: `Northwind Office Supply` is the customer, not the vendor. bank_export.csv date,description,invoice_id,amount_cents,status 2026-04-23,BluePeak Distributors,INV-7801,18737,paid 2026-04-24,StationHub LLC,INV-7802,5000,paid 2026-04-27,Orion Field Services,INV-8422,42245,paid vendor_master.csv vendor_id,name,tax_id,status V-BP4471,BluePeak Distributors,BP-4471,active V-SH9982,StationHub LLC,SH-9982,active V-OR1109,Orion Field Services,OR-1109,active V-NW001,Northwind Office Supply,NW-CUSTOMER,customer purchase_orders.csv po_id,vendor_id,limit_cents,status PO-4510,V-BP4471,20000,open PO-4510-B,V-SH9982,6000,open PO-8422,V-OR1109,50000,open Input images inv_7801_paid.png inv_7802_short_paid.png inv_8422_under_review.png quote_6400_not_invoice.png open case files and generated images Agent actions Model answer No final JSON captured. Expecting ',' delimiter: line 17 column 23 (char 517)

Task

Category

Score

Failed Checks

Failure Types

Prompt / Answer

generated_invoice_case_01

paperwork

0/4

audit_result_exists, visible_checks_pass, core_oracle_pass, hidden_oracle_pass

none

case files

Inspect

Prompt

You are auditing a synthetic scanned paperwork folder.

Read `README_TASK.md`, inspect the files in `scans/`, and use `bank_export.csv`, `vendor_master.csv`, and `purchase_orders.csv`.

Write `audit_result.json` only.

This is benchmark bookkeeping, not financial, tax, or legal advice.

Important:
- The scanned images are the source documents.
- `Northwind Office Supply` is the customer, not the vendor.
- Ignore documents that clearly say they are not invoices.
- Do not invent fields that are not supported by the files.
- Use visible document IDs for `ignored_document_ids`, not filenames.
- Use relative paths with folder prefixes in `evidence`, for example `scans/example.png`.
- Use only allowed lowercase warning codes from `README_TASK.md`.
- Warning arrays must be flat arrays of strings.

Workspace files available to the model

README_TASK.md

# Paperwork Trial v3 Calibration Case 01

This is synthetic benchmark bookkeeping. It is not financial, tax, or legal advice.

Audit the scanned document folder and write `audit_result.json` only.

Case ID: `P3-GEN-01`

Output keys exactly:

`case_id, approved_invoice_ids, review_invoice_ids, reject_invoice_ids, ignored_document_ids, total_approved_gross_cents, warnings_by_invoice, evidence, proof_code`

Rules:

- Treat the files in `scans/` as the source documents.
- Ignore documents that are explicitly not invoices.
- Match vendors by tax ID first, then by exact vendor name.
- `payment_match` requires paid bank rows for the invoice whose summed amount equals the invoice gross.
- `payment_short` applies when the paid bank amount is lower than invoice gross.
- `under_review_stamp` applies when the scanned invoice visibly has an under-review stamp.
- Approved invoices have no warnings and paid amount equals gross.
- Review invoices have warnings but are not reject-level. `payment_short` and `under_review_stamp` are review-level warnings, not reject-level warnings.
- Reject only if the invoice is impossible to process, uses an inactive vendor, or lacks a required PO.
- Sort all invoice-id arrays ascending.
- Allowed warning codes are exactly: `payment_short`, `under_review_stamp`.
- `warnings_by_invoice` must include every real invoice ID and sorted warning arrays.
- Warning arrays must be flat arrays of lowercase strings, never nested arrays and never prose labels.
- `ignored_document_ids` must include visible document IDs from ignored non-invoice documents, not filenames. Example: use `QT-6400`, not `quote_6400_not_invoice.png`.
- `total_approved_gross_cents` is the sum of approved invoice gross totals only.
- `evidence` must list the relative source file paths used in stable alphabetical order, including folder prefixes such as `scans/`.
- Include the relevant CSV files and every scanned document inspected in `evidence`, including ignored non-invoice scans.
- `proof_code = total_approved_gross_cents + sum(numeric parts of all real invoice IDs) + 97 * total_warning_count`.

Important: `Northwind Office Supply` is the customer, not the vendor.

bank_export.csv

date,description,invoice_id,amount_cents,status
2026-04-23,BluePeak Distributors,INV-7801,18737,paid
2026-04-24,StationHub LLC,INV-7802,5000,paid
2026-04-27,Orion Field Services,INV-8422,42245,paid

vendor_master.csv

vendor_id,name,tax_id,status
V-BP4471,BluePeak Distributors,BP-4471,active
V-SH9982,StationHub LLC,SH-9982,active
V-OR1109,Orion Field Services,OR-1109,active
V-NW001,Northwind Office Supply,NW-CUSTOMER,customer

purchase_orders.csv

po_id,vendor_id,limit_cents,status
PO-4510,V-BP4471,20000,open
PO-4510-B,V-SH9982,6000,open
PO-8422,V-OR1109,50000,open

Input images

inv_7801_paid.png
inv_7802_short_paid.png
inv_8422_under_review.png
quote_6400_not_invoice.png

open case files and generated images

Agent actions

Model answer

No final JSON captured.

Expecting ',' delimiter: line 17 column 23 (char 517)