Legacy benchmark · reproducible no-tools condition
We measured what happens when an LLM does your financial math.
We asked 6 production models from OpenAI and Anthropicto compute IRR, XIRR, NPV, a DCF valuation, and a private-equity distribution waterfall — with no calculator, just like an agent that “does the math itself.” The reference answers come from the same deterministic engine behind our live finance endpoints. These results describe this historical no-tools condition only.
How we measured it
- Ground truth = our shipped engine. Every “correct” value is computed by the deterministic code behind our live CashflowLens and WaterfallLens endpoints — and matches the textbook / Excel result.
- No tools, no calculator. Each model gets the problem as plain text and must answer with the number — the same situation as an agent computing inline instead of calling an API.
- Pinned model snapshots, temperature 0 where supported. GPT-4o mini, GPT-4o, GPT-4.1, GPT-5.1, Claude Haiku 4.5, Claude Sonnet 5. A miss counts only when it exceeds a generous per-problem tolerance — we are catching material errors, not the last basis point.
- Fully reproducible. The exact prompts are below; the harness lives at
scripts/benchmark/run-benchmark.mjs. Generated July 3, 2026.
Example prompt (verbatim)
Compute the XIRR (annualised internal rate of return on irregularly dated cash flows) for:
2008-01-01: -10000
2008-03-01: +2750
2008-10-30: +4250
2009-02-15: +3250
2009-04-01: +2750
Use actual day counts (a 365-day year). Answer as the annual rate as a percentage (e.g. 37.34 means 37.34%).
Do not use a calculator, code interpreter, or any tool — compute it yourself. You may show brief working, but the LAST line of your reply must be exactly:
ANSWER: <number>
Give only the bare number on that line (no units, no commas, no % sign, no $).The results
The GPT-4o-class models most agents call by default were wrong on 44% of attempts. Each cell shows the model’s answer against the exact value.
| Problem | Exact (our engine) | GPT-4o mini | GPT-4o | GPT-4.1 | GPT-5.1 | Claude Haiku 4.5 | Claude Sonnet 5 |
|---|---|---|---|---|---|---|---|
Periodic IRR, 3 equal inflows IRR · easy Compute exactly: CashflowLens · $0.20/call | 23.38% | 50.00% wrong | 328.90% wrong | 23.40% correct | 23.40% correct | 23.45% correct | 23.37% correct |
XIRR on irregular dates (Excel canonical) XIRR · hard Compute exactly: CashflowLens · $0.20/call | 37.34% | 10.00% wrong | 2450.00% wrong | 0.16% wrong | 37.30% correct | 37.48% correct | — wrong |
NPV at 10% NPV · easy Compute exactly: CashflowLens · $0.20/call | $243.43 | $244.66 wrong | $243.43 correct | $243.33 correct | $243.24 correct | $243.42 correct | $243.43 correct |
DCF enterprise value, Gordon-growth terminal DCF · medium Compute exactly: CashflowLens · $0.20/call | $1,207 | $1,207 correct | $1,207 correct | $1,208 correct | $1,207 correct | $1,206 correct | $1,207 correct |
DCF implied share price (exit multiple + net-debt bridge) DCF · hard Compute exactly: CashflowLens · $0.20/call | $8.00 | $8.00 correct | $8.00 correct | $8.00 correct | $8.00 correct | $8.00 correct | $8.00 correct |
LP/GP distribution waterfall — GP carry dollars Waterfall · hard Compute exactly: WaterfallLens · $0.25/call | $1,600,000 | $1,600,000 correct | $1,412,268 wrong | $1,600,000 correct | $1,600,000 correct | $0 wrong | $1,600,000 correct |
Black-Scholes European call price Options · hard Compute exactly: OptionLens · $0.25/call | $9.12 | $9.08 correct | $9.17 correct | $9.16 correct | $9.14 correct | $9.11 correct | $9.12 correct |
Yield to maturity of a coupon bond Bond · hard Compute exactly: BondLens · $0.25/call | 5.66% | 5.00% wrong | 5.70% correct | 5.84% wrong | 5.67% correct | 5.62% correct | 5.66% correct |
| Correct / 8 | 4/8 | 5/8 | 6/8 | 8/8 | 7/8 | 7/8 |
Stability and contamination checks
Two honesty mechanisms most benchmarks skip. Each problem is asked 3× and scored by majority — stability is how often a model returned the same number every time (the engine: always). And because these prompts are public, every run also executes a fresh-variant set with new numbers the models cannot have memorized — a large canonical-vs-fresh gap would be a contamination signal, and we publish it either way.
| Model | Canonical set | Fresh variants | Same answer every attempt |
|---|---|---|---|
| GPT-4o mini | 4/8 | 3/8 | 4/8 |
| GPT-4o | 5/8 | 5/8 | 2/8 |
| GPT-4.1 | 6/8 | 7/8 | 1/8 |
| GPT-5.1 | 8/8 | 5/8 | 1/8 |
| Claude Haiku 4.5 | 7/8 | 6/8 | 3/8 |
| Claude Sonnet 5 | 7/8 | 8/8 | 4/8 |
| Deterministic engine (reference) | 8/8 | 8/8 | always |
Fresh variants are generated from seeded parameter ranges (seed 202607, recorded in the dataset) and graded by the same engines — reproduce both sets with the published harness.
Run history
This archive preserves completed no-tools runs with pinned model snapshots. Future runs are added only when executed and verified; every completed run is archived with pinned snapshots, so improvement (or regression) is visible over time — scores are correct answers out of 8 problems.
| Run | GPT-4o mini | GPT-4o | GPT-4.1 | GPT-5.1 | Claude Haiku 4.5 | Claude Sonnet 5 |
|---|---|---|---|---|---|---|
| Jul 3, 2026current | 4/8 | 5/8 | 6/8 | 8/8 | 7/8 | 7/8 |
| Jul 3, 2026 | 2/6 | 3/6 | 5/6 | 6/6 | 5/6 | 6/6 |
| Jun 22, 2026 | 2/6 | 3/6 | 5/6 | 6/6 | — | — |
Anatomy of a confident wrong answer
The failures are not “I’m not sure.” The models show clean, plausible working and hand back a precise, wrong number. Verbatim tails:
GPT-4o · XIRR on irregular dates (Excel canonical)
approx 3100 \)
- \( \frac{3250}{(1.30)^{\frac{410}{365}}} \approx 2500 \)
- \( \frac{2750}{(1.30)^{\frac{456}{365}}} \approx 2000 \)
Summing these gives approximately: \( 2500 + 3100 + 2500 + 2000 = 10100 \)
The NPV is approximately \( 10100 - 10000 = 100 \), still positive, so \( r = 0.30 \) is also too low.
Try \( r = 0.35 \) (35%):
- \( \frac{2750}{(1.35)^{\frac{60}{365}}} \approx 2450 \)
-GPT-4o · Periodic IRR, 3 equal inflows
\[
NPV = -1000 + \frac{500}{1.2336} + \frac{500}{1.2336^2} + \frac{500}{1.2336^3}
\]
\[
NPV = -1000 + 405.46 + 328.91 + 266.76 = 0.03
\]
Positive, so the IRR is slightly less than 23.36%.
12. **Trial with \( r = 0.23365 \):**
\[
NPV = -1000 + \frac{500}{1.23365} + \frac{500}{1.23365^2} + \frac{500}{1.23365^3}
\]
\[
NPV = -1000 + 405.45 + 328.90 +Claude Haiku 4.5 · LP/GP distribution waterfall — GP carry dollars
(Return of capital): $10,000,000 to LPs
- Remaining: $8,000,000
This $8,000,000 goes to Tier 2 (Preferred return). Since the full preferred return is $14,693,280.13, and we only have $8,000,000, all remaining cash goes to LPs.
Therefore:
- LPs total: $18,000,000
- GP total: $0
The GP receives nothing because there is insufficient cash to cover return of capital and preferred return.
ANSWER: 0“But the best model got them all right.”
True — GPT-5.1 passed 8 of 8within tolerance. That is the honest result, and it is the right way to read this: the answer is not “LLMs are dumb.” It is that you cannot tell, at call time, whether you got the GPT-5.1 answer or the 2450% one. And even the model that passed:
- Wasn’t exact. On Periodic IRR, 3 equal inflows it returned 23.40% versus an exact 23.38%. Close enough to pass, not close enough to book.
- Was slow. It took up to 21.8s on a single problem and burned 19,059 tokens across the six — every call, every time.
- Gave no determinism and no audit trail. Re-run it and the number can drift. There is no per-rule evidence, no guarantee the same inputs return the same result.
Give the agent a deterministic tool instead
The same problems, computed by a unit-tested function that always returns the same audited result — with the evidence behind every number — for a few cents a call over x402. No data feed, no API key, no guessing. And when the model itself computes correctly, the remaining gap is still real: answers that drift across model versions, and no artifact anyone else can verify. Pin the math to a deterministic engine and seal each action with a signed attestation.
POST https://hermesplant.com/api/agent-services/cashflowlens/analyze
{ "cashflows": [
{ "amount": -10000, "date": "2008-01-01" },
{ "amount": 2750, "date": "2008-03-01" },
{ "amount": 4250, "date": "2008-10-30" },
{ "amount": 3250, "date": "2009-02-15" },
{ "amount": 2750, "date": "2009-04-01" } ] }
→ 200 OK
{ "summary": { "xirr": 0.373363, ... },
"findings": [ { "rule": "mode", "evidence": "mode=dated ..." } ] }The exact XIRR the models missed — 37.34% — returned deterministically, with evidence, every time.
Frequently asked questions
- Do large language models get IRR and XIRR wrong?
- Often, yes. In this benchmark the GPT-4o-class models that agents most commonly call in a loop were wrong on 44% of attempts when asked to compute IRR, XIRR, NPV, DCF and a private-equity waterfall without a calculator. Errors were not rounding noise — this run's worst misses included a 2450% answer for XIRR on irregular dates (Excel canonical) (correct: 37.34%), a 329% answer for Periodic IRR, 3 equal inflows (correct: 23.38%), a $1,600,000 miss on LP/GP distribution waterfall — GP carry dollars. The model states each wrong number with full confidence and no error flag.
- How do Claude and GPT models compare on finance math?
- This run covers both vendors: GPT-5.1 scored 8/8, Claude Haiku 4.5 scored 7/8, Claude Sonnet 5 scored 7/8, GPT-4.1 scored 6/8, GPT-4o scored 5/8, GPT-4o mini scored 4/8. The pattern is consistent across vendors — frontier reasoning models do far better than the default-tier models agents actually wire into loops, and even a perfect score costs reasoning latency and tokens on every call that a deterministic engine does not.
- How is the 'correct' answer determined?
- The reference column is computed by the same deterministic engine that powers Hermes Plant's live, paid finance endpoints (CashflowLens and WaterfallLens). Each reference value matches the standard textbook / Excel result — for example the XIRR case reproduces Excel's canonical 37.34%.
- Isn't this just because the model couldn't use a calculator?
- That is exactly the point. An AI agent that 'does the math itself' is computing without a calculator on every call. The fix is to give the agent a deterministic tool to call. The frontier model that did pass every problem here was still slower, burned reasoning tokens per call, and was not bit-for-bit exact — none of which a $0.20 deterministic API call has to trade off.
- Can I reproduce these results?
- Yes. The exact prompts are published below and in the dataset, the model snapshots are pinned (e.g. gpt-4o-2024-08-06), and the harness (scripts/benchmark/run-benchmark.mjs) transpiles the live engine for ground truth and replays the calls. Results were generated July 3, 2026; LLM outputs can vary slightly between runs even at temperature 0.
Ground truth computed by Hermes Plant's shipped finance engines (src/lib/cashflowlens-service.ts, waterfall-service.ts). LLM columns are real OpenAI chat-completions / Anthropic messages calls under a no-tools condition (no code interpreter), temperature 0 where supported. Reproduce with scripts/benchmark/run-benchmark.mjs. Model snapshots: gpt-4o-mini-2024-07-18, gpt-4o-2024-08-06, gpt-4.1-2025-04-14, gpt-5.1-2025-11-13, claude-haiku-4-5-20251001, claude-sonnet-5.