Skip to main content

​
Judgment, made production-grade

Every accounting system that puts a language model near the books has to answer one question. How do you know it works? This tab is the answer, and it is deliberately not a list of adjectives. It is 68 cases, 443 graded assertions and 104 score metrics, run against the real compiled graphs with the real model, on a scoreboard that publishes the failures next to the passes.

​
The argument, in three moves

The order matters. Made in any other order it sounds like marketing.
1

The engine does not need an eval, because it is correct by construction

Balance, immutability, idempotency, the hash chain and the period lock are enforced by code and by database constraints, inside a single writer.There is no probability distribution over whether an unbalanced entry posts. An unbalanced entry is refused by the 18 GL invariants, before the write, every time.Naming this first is what earns the right to the second move.
2

What cannot be made correct by construction is judgment

Is this scanned Vietnamese fax a supplier VAT invoice or a canteen invoice? Which of 146 document types? Is the total 57,240,000, or 57,240,000 with a VAT line the model invented?Judgment is where a language model is genuinely better than rules, and it is exactly where a language model is unaccountable. No database constraint makes a misclassification impossible.
3

So the judgment layer is measured, versioned and regression-gated

68 cases across the five deployed agents, graded on 443 reference assertions, driven against the real compiled graphs with the real model and a real migrated Postgres.A stubbed extractor cannot fail a classification regression, which is precisely why the previous harness was thrown away.

​
The two halves, and the one-way door between them

The arrow points one way, and it passes through a person. A proposal is a claim, not a write. Document intake uses two gates and three people; an Accountant workpaper uses a named non-agent human who did not propose it. The posting mandate is what makes either agent-staged release stand up under attack.

​
The current composition, with dated historical runs

Clerk · 87%

15 cases, 67 assertions. 🟡 Two model-variance failures: one goods-receipt classification, one judge verdict.

Bookkeeper · 11 cases

55 assertions. Its recorded 100% run covered 10 cases; the current composition has one additional case.

Auditor · 26 cases

195 assertions across integrity, reconciliation, Close phases, completeness, and register lifecycle. Its recorded 100% run covered 10 cases.

Accountant · 100%

5 cases, 22 assertions. 🟢 The close checklist and the adjusting entry, through the real posting gate.

Typewriter · 55-64%

11 cases, 104 assertions. 🟡 36% at its first baseline. The eval found four real defects on day one, and measured the fix for the largest.

Board · 68 cases

443 graded reference keys, 104 distinct score metrics. Composition and dated runs are kept separate.
Two of those pass rates are not 100%, and they are on the front page on purpose.A perfect board is either a very small board or a dishonest one. The Typewriter’s 36% first baseline is the most useful number in this tab, because it is the one that moved when a defect was fixed. See exactly what moved.

​
What this tab does not claim

An eval is not a proof of correctness for anything the deterministic layer already guarantees, and it is not a benchmark against another product.It drives a real model, so it is stochastic. A green run means the judgment layer behaved on the cases that named run actually exercised on that day. It does not mean the model cannot be wrong tomorrow, which is why the model never gets the last word.
The long form: what an eval is, and is not.

​
The honest closer

Raw Claude genuinely wins at ad-hoc, low-volume, exploratory work. GreatBook keeps that - Claude is the interface, and the Clerk is Claude reading the document - and adds the guarantees. The difference is not better AI. It is the same Claude, wrapped in guarantees.

What an eval is, and is not

The long form of the thesis, and the four things this tab refuses to claim.

Never stub what you test

What is real per agent, what is substituted, and the guards that stop the harness lying to itself.

The scoreboard

Every number, per agent and per dimension, with its run date and its refresh command.

The security boundary

Why nothing a browser can drive reaches the ledger, and the executed attacks that made it true.
  • Guarantees - the deterministic half, stated as commitments
  • The agent org - the five graphs these cases grade
  • Status - what is live, what is staged, and what is not built