> ## Documentation Index > Fetch the complete documentation index at: https://docs.vinmake.com/llms.txt > Use this file to discover all available pages before exploring further. # Never stub what you test > What is real per agent, what is substituted and why, and the guards that stop a harness from lying to itself - including the one that failed and how it was closed. # Never stub what you test The design rule of the whole package, in one sentence: > **An eval never stubs the thing it is testing.** It exists because the harness this one replaced broke it. The earlier data-entry evals injected a scripted extractor, so the model was never exercised - and a classification or extraction regression **could not fail the eval**. A green board meant the script still matched itself. ## What is real, per agent Every target runs the agent's **real compiled graph**. | Agent | What is real | What is substituted, and why | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Clerk** | the compiled Clerk graph plus the real keyless Claude Agent SDK extractor, built by the same factory `clerk/app.py` calls in production | the queue store and the file sink only. These are **storage side effects**: an eval must not write rows into the live intake queue or upload fixtures to production storage. They implement the same interfaces, so flag-vs-pass, FLAGGED-vs-PENDING and duplicate detection all still run the real code | | **Bookkeeper** | the compiled graph, the real posting rules, and the real ledger writer against a **real `alembic upgrade head` Postgres** - advisory lock, 18 GL invariants, hash chain and GL idempotency key all live | nothing under test | | **Auditor** | the compiled graph and all seven real check-family runners over a real migrated GL, reconciliation graph, Close oracle, and durable register, with faults injected out of band on a separate writer connection | nothing under test | | **Accountant** | the compiled graph, the real deterministic scope gate, the real Close-list oracle over a real migrated GL, and - for an adjusting entry - the **real Bookkeeper graph** it routes through, paused at that graph's human gate | nothing under test. `expect_no_direct_ledger_write` is checked by counting rows in `journal_entries`, never by reading the answer | | **Typewriter** | the compiled graph, the real 146-type catalog, the real scope gate, the real payload projection and number re-reading, and the real keyless Agent-SDK proposer from the same factory `typewriter/app.py` calls | **nothing at all** | **The Typewriter's row is the strongest one, and it is arithmetic rather than a promise.** That graph has exactly one injected boundary - the model - and no storage of any kind. There is nothing left to substitute, which is why its target is the shortest file in the package. ## The distinction that makes the rule usable "Never stub anything" is not a workable rule; every eval has to stop somewhere. The line this package draws: A **storage side effect**. The Clerk's queue and file sink become in-memory doubles behind the same interface, so no eval run can write into the live intake queue. Every governance decision still executes the real code. The **reasoning under test**. The extractor, the classifier, the posting rules, the check families, the scope gate. Substituting any of those produces a board that grades the substitute. ## Three design rules, each with the guard test that enforces it ### 1. A missing prerequisite reports SKIP, never a pass No Postgres, no model credentials, an agent branch that has not landed: the example reports **SKIP**. A skip is never counted as a pass, so a green board always means green code. And `--strict` makes skips fatal, so an environment that is *supposed* to have everything wired can demand it. This was verified rather than assumed: pointed at an unreachable database host, the ledger cases report `[SKIP] ... no Postgres reachable`, score nothing, and the process exits 0 normally but **1 under `--strict`**. ### 2. No agent is parked as un-gradeable `SKIPPED_AGENTS` is empty, and a test keeps it empty. **And that guard was not enough, which is the most useful thing on this page.** An empty skip list stays empty for an agent that was **never registered at all**. The Typewriter shipped deployed and ungraded for exactly that reason: it was absent from the eval registry entirely, four of five served graphs were scored, and every guard in the package agreed that was fine. The fix is not another rule about the registry. It is to **stop asking the registry about itself**: `test_every_served_graph_is_graded` reads the deploy manifest - the file that declares what production serves - and requires every served graph to be bound in the registry. Add a sixth graph without an eval and the build goes red. That hole was found, reproduced by negative control, closed, and written down. Publishing it is the point: a harness whose failure modes are secret is a harness nobody can audit. ### 3. An expectation nobody reads is worse than no expectation `test_every_reference_key_is_graded_by_some_evaluator` asserts that every key in every example's expected outputs is read by at least one of that agent's evaluators. An ungraded expectation still shows **PASS**, which is the most quietly dangerous state a case can be in. That test caught a dead expectation key during the build. ## The judge boundary **A model judges in exactly one place, and it is not the accounting.** The single judged score is the Clerk's free-form in-scope chat reply, graded against that example's written rubric. It **fails closed**: an unavailable judge scores 0, never a pass. Account codes, balance, entry counts, hash-chain findings, flag fields, severities, scope verdicts, schema types and row counts are all **exact comparisons**. The Typewriter deliberately has no judge at all, and that is a fact about the agent rather than a gap: its summary comes from a deterministic table and its refusals from the scope gate's own table, so "did it answer well" already has a closed form. The thing the model actually decides - the proposal - is graded field by field against the doc type's real schema. ## Recording a known defect A permanently red case hides a regression exactly as well as a permanently skipped one: the next real failure lands on an already-red board and nobody looks twice. So a case that fails for a **known, accepted** reason declares it - by evaluator key, and by the situation that produces it. ```json theme={null} "metadata": { "expected_baseline": [ {"note": "the row-schema list does not cover this type's rows, so they get no per-row schema", "failing_keys": ["collections_populated"]}, {"note": "the INTERMITTENT classify miss: 0 proposals, so every payload key fails at once", "failing_keys": ["doc_type_correct", "field_accuracy"], "only_when": {"n_proposals": 0}, "intermittent": true} ] } ``` | Outcome | Mark | Fails the run? | | ------------------------------------------------------------------------ | --------------- | -------------------------------------------------------------- | | every failing key is excused by a group whose condition matches this run | `XFAIL` | no | | a key fails that no matching group excuses | `FAIL` | **yes**, and the message names it | | every score passed, and a matching group is not intermittent | `XPASS` | **yes** - the defect appears fixed, so the marker is now stale | | no marker at all | `PASS` / `FAIL` | unchanged | Three things about this mechanism are load-bearing. **Naming the keys.** A bare "this case may fail" flag would swallow a new regression inside a case that was already red. **`only_when`, which is the same rule one level finer.** The Typewriter's intermittent classify miss fails every payload-derived key at once, but only by ending the turn with zero proposals. Excusing `doc_type_correct` unconditionally would also excuse a turn that **did** propose and named the wrong one of the 146 types - which is the mis-proposal the dataset exists to catch. Conditioned on `{"n_proposals": 0}`, the same key is an accepted defect on the turn that reproduces it and a hard failure on every other. **XPASS is a failure.** When a recorded defect is fixed, the run says so and goes red, so a stale excuse cannot outlive the bug. That is not theoretical: the first run against the number-coercion fix reported **three XPASS** - *"the defect this baseline recorded appears fixed"* - which is how the fix announced itself. A marker records a defect that was **observed**, never one that is merely possible. The Clerk's two failures are deliberately **not** marked. They are pre-existing model variance in a dataset that change did not investigate, and excusing an uninvestigated failure is precisely the wrong use of the mechanism. The structure tests check both directions of it: that a marked case still goes red on a wrong doc type, that the anti-fabrication bound is never excused, that a recorded intermittent miss never turns the board red, and that no marker names a key nothing scores - because a marker naming an impossible key would excuse a case forever for a reason that cannot occur. ## How a score is computed Every evaluator returns zero or more scores. Returning **nothing** means "this evaluator does not apply to this example", so a mixed dataset never manufactures a 1.0 for a case nothing graded. An example **passes** when every score it received is 1.0. One deliberate wrinkle: extraction and field accuracy are reported as a **raw ratio** alongside a pass/fail floor. The floor is the gate; the ratio is a trend line. A case carrying a floor of 0.85 therefore still fails at 0.90 on the raw key, which is a real and recorded cause of cases sitting below green. ## Layout ``` core.py Example / Dataset / Score / RunReport + the local runner registry.py binds each agent to (target, evaluators) - one source of truth gl_fixture.py provisions the migrated throwaway GL + every seeder and fault injector langsmith_sync.py dataset upsert + tracked experiments + the tracing health probe run.py the CLI datasets/*.json the datasets - source of truth, mirrored INTO LangSmith, never the reverse fixtures/*.txt real sample documents the Clerk actually has to read targets/ one per agent: runs the real agent, returns observable outputs evaluators/ one per agent, plus matching helpers and the single judge tests/ the CI gate: offline structure tests plus the marked real-agent suite ``` **The in-repo JSON is the source of truth.** The upload rebuilds the LangSmith copy rather than merging into it, so an edit made in the dashboard is overwritten on the next sync. A dataset is code, and it lives in the repo with the code it grades. ## Where it is enforced | Concern | Code | | --------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | the rule, stated where a future author will read it | `langgraph_chat/agents/evals/core.py` module docstring | | every served graph is graded | `agents/evals/tests/test_evals_ci.py::test_every_served_graph_is_graded` | | every target drives the real graph | `test_every_target_drives_the_real_agent_graph` | | no dead expectation | `test_every_reference_key_is_graded_by_some_evaluator` | | the baseline marker cannot over-excuse | `test_the_typewriter_mis_proposal_check_still_gates`, `test_the_anti_fabrication_bound_is_never_excused` | | skip is never a pass | `agents/evals/core.py`, `EvalSkipped` | ## Related * [The scoreboard](/greatbook/proof/scoreboard) - what this harness produced * [What an eval is, and is not](/greatbook/proof/why-evals) - why the engine is deliberately not on this board * [How this gates a release](/greatbook/proof/regression-gate) - which half of this runs on every PR * [Reading a run](/greatbook/proof/traces) - the trace a graded run leaves behind