> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vinmake.com/llms.txt
> Use this file to discover all available pages before exploring further.
# What an eval is, and is not
> The long form of the thesis. Why the deterministic layer is deliberately not evaluated, why the judgment layer must be, and the four claims this tab refuses to make.
# What an eval is, and is not
An eval is a **dataset of recorded judgment tasks, with the right answer written down, run against the real system, and scored key by key.**
That is a narrow thing.
Most of its value comes from being clear about how narrow.
## Why the engine is deliberately not evaluated
An eval samples.
A constraint holds.
If a property can be expressed as a constraint, sampling it is a downgrade - and publishing the sample as evidence is worse, because it invites the reader to believe a 100% pass rate on forty postings is what makes the ledger balance.
It is not.
The ledger balances because [the single writer](/greatbook/capabilities/general-ledger) refuses an unbalanced entry before the transaction opens, and because a database constraint refuses it again underneath.
| Property | How it is established | Why an eval is the wrong tool |
| --------------------------------- | ----------------------------------------------------- | ------------------------------------------------ |
| every entry balances | GL invariant, checked in the writer, before the write | a sample cannot establish a universal |
| a posted line never changes | immutability trigger in Postgres | the trigger has no failure rate to sample |
| a re-run posts once | GL idempotency key on a unique index | idempotency is a property of a key, not of a run |
| the hash chain is intact | recompute over every entry | already exhaustive, not sampled |
| a closed period rejects a posting | GL invariant plus the period lock | same |
The right test for a constraint is a test that **executes the refusal path**, and those live in the engine's own suites.
See [the 18 GL invariants](/greatbook/capabilities/gl-invariants) for what is refused and which test drives each refusal.
This is the part most eval write-ups get backwards.
Grading what has already been proven inflates the board, teaches the reader nothing, and quietly moves a **guarantee** into the category of **things that usually work**.
## Why judgment must be
Four things in GreatBook are irreducibly judgment, and each one is a place where a wrong answer is expensive and plausible at the same time.
Which of the 146 registry types is this?
A canteen invoice and a supplier VAT invoice are one wording apart and carry different schemas, different approval chains and different posting rules.
Which fields, and which numbers?
A confidently-filled total on an illegible fax is a fabrication that reads exactly like a success.
Which accounts?
`Dr 1200 / Cr 2000` and `Dr 1200 + Dr 1300 / Cr 2000` both balance, so balance cannot tell them apart.
Whose lane is this request?
A refusal that fires on the wrong turn is annoying; one that fails to fire on the right turn is a control that does not exist.
Each has a right answer, each is checkable without a human, and none can be made impossible to get wrong.
That combination is what an eval is for.
## What an eval is not
A unit test is deterministic by construction: same input, same output, forever.
An eval drives the real model, so it is **stochastic**.
The Clerk's SOP-citation case failed one run in ten while every other run of the same case scored 1.000, and that **rate** is itself the finding.
A harness that hid the variance would have hidden the defect.
The consequence is a discipline rather than a disclaimer: a failure is investigated for its rate before it is called a regression, and a known, understood, recorded defect is **declared** rather than tolerated.
See [recording a known defect](/greatbook/proof/harness).
There is no leaderboard here and no competitor column.
The cases are GreatBook's own document types, its own chart of accounts, its own separation-of-duties rules and its own Vietnamese source documents.
A score on this board says something about this system on these cases, and nothing about anyone else's.
The one comparison this tab does make is [against raw Claude](/greatbook/proof/vs-raw-claude), and it compares **guarantees**, not scores - because the model on both sides is the same model.
When the Bookkeeper's posting case passes, what it establishes is that the agent **proposed the right accounts and then stopped at the gate**.
That the resulting entry balances, hashes into the chain and posts exactly once is established by the engine, and would be equally true of an entry a human typed.
The eval asserts those properties in its expectations so that a case fails loudly if the engine ever stops holding them - but it is checking a guarantee, not creating one.
Nothing on this board would change if the pass rate were 100% on every agent forever.
The Bookkeeper still stops at `post`.
A second person still approves.
The eval measures how good a proposal is.
It never earns an agent the right to skip the gate, because the gate is not a hedge against a bad model - it is [an accounting control](/greatbook/capabilities/maker-checker) that exists for human error and human fraud too.
## What makes a case worth having
A case earns its place by being **falsifiable, expensive if wrong, and not already guaranteed**.
Three shapes recur across this dataset:
| Shape | Example in the dataset | Why it is worth a case |
| -------------------- | ---------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| a **near neighbour** | a canteen invoice against a supplier VAT invoice; a service invoice one wording away from a supplier VAT invoice | the failure is a plausible answer, so nothing downstream looks wrong |
| a **degraded input** | the illegible fax scan | the only correct behaviour is empty money fields and a flag, so a confident number is unambiguously a fabrication |
| a **seeded fault** | a wrong `prev_checksum`, a tampered line, an entry in a closed period | the right answer is known exactly, so a missed detection cannot be argued about |
And the counter-shape, deliberately included: **a clean book that must come back clean**.
The Auditor's known-good case is what stops the fault cases from being satisfiable by an agent that reports a problem every time.
## Two rules the dataset itself has to follow
Both were learned by getting them wrong.
**Write an expectation against the doc type's real schema, never against what seems reasonable.**
`canteen_invoice` carries `vat_rate` but no `vat_amount` and no `total_amount`, and the graph's allowlist correctly drops anything outside the schema.
The first version of that expectation was wrong, and the agent was right.
**Avoid a time-dependent fixture.**
The Clerk correctly flags a future `doc_date`, so a fixture dated at the current month-end starts failing as the calendar moves - a case that decays into a false alarm.
## The sibling argument
The same case has been made elsewhere in the house as a **commitment**: this is what we will build, and this is what "dependable" would mean.
This tab is the **receipt**.
The claim was made, the cases were written, the runs happened, and the numbers - including the ones that are not 100% - are published with their dates.
## Related
* [Never stub what you test](/greatbook/proof/harness) - how the harness is kept from lying to itself
* [The scoreboard](/greatbook/proof/scoreboard) - the numbers, per agent and per dimension
* [How this gates a release](/greatbook/proof/regression-gate) - what runs on every PR, and what does not
* [Guarantees](/greatbook/guarantees) - the deterministic half, which this page argues should never be sampled