Skip to main content

​
What an eval is, and is not

An eval is a dataset of recorded judgment tasks, with the right answer written down, run against the real system, and scored key by key. That is a narrow thing. Most of its value comes from being clear about how narrow.

​
Why the engine is deliberately not evaluated

An eval samples. A constraint holds. If a property can be expressed as a constraint, sampling it is a downgrade - and publishing the sample as evidence is worse, because it invites the reader to believe a 100% pass rate on forty postings is what makes the ledger balance. It is not. The ledger balances because the single writer refuses an unbalanced entry before the transaction opens, and because a database constraint refuses it again underneath. The right test for a constraint is a test that executes the refusal path, and those live in the engine’s own suites. See the 18 GL invariants for what is refused and which test drives each refusal.
This is the part most eval write-ups get backwards. Grading what has already been proven inflates the board, teaches the reader nothing, and quietly moves a guarantee into the category of things that usually work.

​
Why judgment must be

Four things in GreatBook are irreducibly judgment, and each one is a place where a wrong answer is expensive and plausible at the same time.

classify

Which of the 146 registry types is this? A canteen invoice and a supplier VAT invoice are one wording apart and carry different schemas, different approval chains and different posting rules.

extract

Which fields, and which numbers? A confidently-filled total on an illegible fax is a fabrication that reads exactly like a success.

propose

Which accounts? Dr 1200 / Cr 2000 and Dr 1200 + Dr 1300 / Cr 2000 both balance, so balance cannot tell them apart.

scope

Whose lane is this request? A refusal that fires on the wrong turn is annoying; one that fails to fire on the right turn is a control that does not exist.
Each has a right answer, each is checkable without a human, and none can be made impossible to get wrong. That combination is what an eval is for.

​
What an eval is not

A unit test is deterministic by construction: same input, same output, forever.An eval drives the real model, so it is stochastic. The Clerk’s SOP-citation case failed one run in ten while every other run of the same case scored 1.000, and that rate is itself the finding. A harness that hid the variance would have hidden the defect.The consequence is a discipline rather than a disclaimer: a failure is investigated for its rate before it is called a regression, and a known, understood, recorded defect is declared rather than tolerated. See recording a known defect.
There is no leaderboard here and no competitor column.The cases are GreatBook’s own document types, its own chart of accounts, its own separation-of-duties rules and its own Vietnamese source documents. A score on this board says something about this system on these cases, and nothing about anyone else’s.The one comparison this tab does make is against raw Claude, and it compares guarantees, not scores - because the model on both sides is the same model.
When the Bookkeeper’s posting case passes, what it establishes is that the agent proposed the right accounts and then stopped at the gate.That the resulting entry balances, hashes into the chain and posts exactly once is established by the engine, and would be equally true of an entry a human typed. The eval asserts those properties in its expectations so that a case fails loudly if the engine ever stops holding them - but it is checking a guarantee, not creating one.
Nothing on this board would change if the pass rate were 100% on every agent forever.The Bookkeeper still stops at post. A second person still approves.The eval measures how good a proposal is. It never earns an agent the right to skip the gate, because the gate is not a hedge against a bad model - it is an accounting control that exists for human error and human fraud too.

​
What makes a case worth having

A case earns its place by being falsifiable, expensive if wrong, and not already guaranteed. Three shapes recur across this dataset: And the counter-shape, deliberately included: a clean book that must come back clean. The Auditor’s known-good case is what stops the fault cases from being satisfiable by an agent that reports a problem every time.

​
Two rules the dataset itself has to follow

Both were learned by getting them wrong.
Write an expectation against the doc type’s real schema, never against what seems reasonable. canteen_invoice carries vat_rate but no vat_amount and no total_amount, and the graph’s allowlist correctly drops anything outside the schema. The first version of that expectation was wrong, and the agent was right.
Avoid a time-dependent fixture. The Clerk correctly flags a future doc_date, so a fixture dated at the current month-end starts failing as the calendar moves - a case that decays into a false alarm.

​
The sibling argument

The same case has been made elsewhere in the house as a commitment: this is what we will build, and this is what “dependable” would mean. This tab is the receipt. The claim was made, the cases were written, the runs happened, and the numbers - including the ones that are not 100% - are published with their dates.