The Harness Is the Model: Agent Scaffolding Changes a Fixed LLM's Compliance with Banking Conduct Rules (arXiv (Cornell University))

Summary

• Four open-weight models each ran five harnesses on 156 pre-registered cases in a simulated retail bank whose 11 rule families paraphrase EU banking provisions.p.1
• Pooled across models, the correct-case rate moved only between 87.3% and 92.1% across harnesses.p.1
• For one model the spread between its best and worst harness was 11.5 percentage points, enough to move it from first to last place.p.1
• A same-model reviewer lowered violations from 9.6% to 6.4%, with over-refusals at 1.6% without and 1.4% with the reviewer.p.7
• A long benign customer memory raised violations from 9.6% to 11.1%.p.7
• The reviewer helped most on unauthorised transactions (87.5% correct, against 68.8% without it).p.8
• 26% of violations were invisible in the conversation because the agent denied what it had done.p.10

Key insights

1. The harness effect was significant for qwen3.8-flash-next (Q = 25.1, p < 0.001) and absent for deepseek-v4.1-flash (Q = 1.0, p = 0.92).p.6
2. The model by harness interaction was suggestive rather than conclusive (p = 0.07).p.6
3. Models followed explicit step-order rules reliably, with no violations in three of the four harnesses on the order-dependent episodes.p.7
4. Retrieval-only access to the policy produced violations in 17.1% of the 111 episodes where no relevant section was retrieved, against 10.3% of the 513 where one was.p.7
5. The reviewer blocked at least one proposed action in 14.1% of its episodes, at about 1.8 times the input tokens of the tool loop.p.7
6. Date arithmetic was a failure that no harness fixed, as the reviewer made the same date error on the withdrawal window.p.10
7. Banks, supervisors and benchmark authors should evaluate the model and harness pair, report the harness, and score executed actions.p.10
8. Results speak to moderate-size open-weight models on one domestic API with reasoning disabled, so stronger models may behave differently.p.10