Human-Centred, Risk-Constrained Evaluation of Financial Document Recognition: Evidence, Critical Errors, and Selective Automation (arXiv (Cornell University))

Summary

• Character and word error rates are insufficient for financial documents because changing '1,250.00' to '12,500.00' is not equivalent to confusing a letter in a narrative paragraph.p.1
• The paper proposes typed financial facts linked to visible source regions, separating transcription, normalization and business interpretation.p.2
• An illustrative zero-error requirement for a 95% upper bound below 10−3 is approximately 2,995 critical instances, not a recommended universal sample size.p.9
• In a synthetic trace, a payroll debit read as positive 480,000.00 is flagged by a running-balance validator and routed to human review.p.10
• Operational evaluation should report latency at p50, p90, p95 and p99, throughput, and cost per 1,000 pages.p.12

Key insights

1. Non-compensatory acceptance gates stop a high weighted average from hiding failures on critical fields such as amounts, signs and currency.p.7
2. A composite score cannot by itself certify a safety-relevant constraint; it may rank systems only after feasibility is established.p.8
3. A financial fact is correct only if value, unit, period, entity and polarity all agree.p.6
4. Every extracted field should link to the image region supporting it, and a value inferred from context is not a successful transcription.p.7
5. Selective automation is framed as a constrained routing decision, and a universal confidence threshold is not generally optimal.p.9
6. A minimum reproducibility card should disclose population, sampling, ground truth, schema, partitions, systems, metrics, operations and governance.p.11
7. Cost per successfully processed document should include inference, storage, review and retry costs.p.12