NumTrace: Auditing the Number Provenance of Financial Analyst Agents (arXiv (Cornell University))

Summary

• NumTrace introduces a benchmark with 540 questions over 2,163 SEC EDGAR documents to evaluate numerical claim provenance for financial analyst agents.p.1
• The benchmark snapshot contains 2,132 filings and 31 plan documents cut into 41,726 chunks spanning reporting periods from 2022Q3 to 2026Q2.p.3
• Right number, wrong account (RNWA) errors account for 67.5–81.8% of each agent's failing claims.p.1
• On Form 13F co-filing group questions, binding pass rates fall to 41.9–48.5% across all tested agents.p.8

Key insights

1. Value-only checks fail to detect financial hallucinations because LLM agents frequently copy exact figures while assigning them to incorrect financial accounts or scope.p.1
2. Scorer debugging and reference answer adjustments increased Claude's binding pass rate from 70.8% to 93.8% without changing agent outputs.p.7
3. Form 13F co-filing group questions generate 44% of all observed RNWA claims while comprising only 11% of the question set.p.8
4. A human expert review of 360 sampled claims found a weighted binding pass rate of 88.0% compared to 90.6% produced by the automated scorer.p.7