> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vinmake.com/llms.txt
> Use this file to discover all available pages before exploring further.
# The scoreboard
> 68 cases, 443 graded reference keys, 104 score metrics. Per agent, per dimension, per score key - with the run that produced each number and the command that refreshes it.
# The scoreboard
Every number on this page is measured, dated and traceable to a command.
**Composition figures** are measured from the eval datasets in the GreatBook backend at `4972f62`, **2026-08-03**.
**Pass rates** come from named runs, each with its own date, because a pass rate is a property of a run and not of a repository.
## The board
| Agent | Dataset | Examples | Graded reference keys | Score metrics emitted |
| ---------- | ----------------------- | -------: | --------------------: | ------------------------------: |
| Clerk | `ws-clerk-eval-v1` | 15 | 67 | 18 (17 deterministic + 1 judge) |
| Bookkeeper | `ws-bookkeeper-eval-v1` | 11 | 55 | 16 |
| Auditor | `ws-auditor-eval-v1` | 26 | 195 | 40 |
| Accountant | `ws-accountant-eval-v1` | 5 | 22 | 15 |
| Typewriter | `ws-typewriter-eval-v1` | 11 | 104 | 20 |
| **Total** | | **68** | **443** | **104 distinct** |
104 is the count of **distinct** metric names.
The per-agent column sums to 109 because five names are shared across agents that grade the same behaviour - `decline_behaviour`, `handoff_named`, `no_fabricated_values`, `scope_gate_correct` and `terminal_status_correct`.
The figure worth carrying away is **443**.
Examples are cheap to add; graded assertions are what decides whether a case can silently pass.
## Pass rates, with their runs
| Agent | Pass rate | Examples in that run | Run | Date |
| ---------- | ------------------------------------------------------------------- | -------------------: | -------------------------------------------- | ------------------------ |
| Clerk | **87%** local, **93%** tracked | 15 | `make eval` / `ws-eval-clerk-b614cba4` | 2026-07-26 |
| Bookkeeper | **100%** | 10 | `make eval` / `ws-eval-bookkeeper-005ed0a4` | 2026-07-26 |
| Auditor | **100%** | 10 | `make eval` / `ws-eval-auditor-9bf1c5f8` | 2026-07-26 |
| Accountant | **100%** | 5 | `make eval` / `ws-eval-accountant-a030cbd6` | 2026-07-26 |
| Typewriter | **36%** at first baseline, **55-64%** after the number-coercion fix | 11 | `make eval-typewriter`, four real-model runs | 2026-07-26 to 2026-07-27 |
The tracked sync covered **51 examples, 51 scored, 0 errored, 0 skipped** - so every case really was exercised against a real agent, and none was silently skipped for a missing prerequisite.
**Seventeen cases in the current 68-case composition were not part of that 51-case tracked sync.**
The Bookkeeper row now has 11 cases, and the Auditor has grown from the 10 cases in that run to 26 across reconciliation, Close phases, document completeness, and register lifecycle behavior.
The next `make eval-langsmith` covers all 68.
Stating the gap is cheaper than presenting a historical rate as coverage of cases it never ran.
## What moved, and what has never moved
Two facts are worth more than the aggregate.
**The Typewriter's numbers moved when a defect was fixed, and the eval is how that is known.**
| Score key | First baseline | After the fix (three runs) |
| ----------------------------------------------- | -------------: | -------------------------- |
| `collection_fields_are_numbers` (the line rows) | 0.000 | **1.000, 1.000, 1.000** |
| `payload_matches_schema_types` | 0.250 | 0.875, 0.875, **1.000** |
| `numeric_fields_are_numbers` | 0.000 | 0.667, 0.833, 0.500 |
| pass rate | 36% | **64%, 64%, 55%** |
Read those rows in that order, because they say different things.
The first two **are** the defect: the line rows, where most of a document's money lives, are clean on every run, and whole-payload type conformance reached 1.000.
The third is noisier, and not because amounts are still strings - that key scores a named field 0 when it is **absent** as well as when it is a string, so it also counts turns that produced no draft at all.
**The Clerk's failures are model variance, and they are named.**
Two cases: one goods-receipt-note classification miss, and one judge verdict.
Neither is marked as an accepted baseline, deliberately - they are pre-existing variance that no change has investigated, and excusing an uninvestigated failure is the wrong use of that mechanism.
Every deterministic Clerk key scored 1.000 on every run for the first several weeks of this dataset's life, and the only score that had ever moved was the judge.
That is no longer true, and this page says so rather than repeating a sentence that was accurate when it was written.
## Coverage by dimension
Coverage is deliberate rather than incidental, and this is the table that shows it.
### Clerk · 15 cases
| Dimension | Cases | What is graded |
| -------------- | ----: | ------------------------------------------------------------------------------------------------------------------------------------------- |
| classification | 5 | exact registry doc type against the 146-type registry, including a near-neighbour probe |
| extraction | 4 | per-field accuracy against the doc type's **real** extraction schema, with money and date normalisation |
| uncertainty | 1 | the degraded fax: confidence stays low, money fields stay empty, the record flags |
| governance | 1 | a re-uploaded invoice flags as a possible duplicate, on the right field, with the right reason, and no row carries an approval |
| scope\_gate | 4 | tier-1 off-topic and tier-2 cross-domain both decline with a named handoff; in-scope answers substantively and the chat path writes nothing |
### Bookkeeper · 11 cases
| Dimension | Cases | What is graded |
| ---------------------- | ----: | ------------------------------------------------------------------------------------------------------------------------------------------- |
| proposal\_correctness | 5 | the right debit and credit accounts from a given active intake, kind-driven and doc-type fallback, balanced, gate held with an empty ledger |
| posting | 1 | the full loop produces exactly **one** balanced entry, with the intake row flipped to posted and stamped with its entry id |
| idempotency | 1 | a relaunch short-circuits, proposes nothing, and the entry count stays 1 |
| unmapped\_account | 1 | the error is actually **raised**, not merely "nothing posted" |
| separation\_of\_duties | 2 | maker-is-checker raises before the transaction opens; a rejected posting writes nothing and leaves the row active |
| eligibility | 1 | a still-pending row never reaches `propose` |
### Auditor · 26 cases
| Dimension | Cases | What is graded |
| ----------------- | ----: | ---------------------------------------------------------------------------------------------------------------------------------- |
| known\_good | 1 | a clean book passes the routine families, which also proves the fresh recompute reproduces every stored checksum |
| hash\_chain | 2 | a wrong `prev_checksum`; a posted line tampered with the immutability trigger disabled |
| invariant | 3 | an unbalanced entry, an entry in a closed period, and broken intake-to-GL links |
| orphan | 1 | a stale row flagged **alongside** a legitimately in-flight one, so false positives are graded too |
| routing | 1 | a three-part request runs and **reports** all three families, not only the one that raised an exception |
| scope\_gate | 2 | a write request and a document-intake request both decline before any database read |
| fail\_safe | 1 | an unreachable ledger comes back inconclusive and never renders as a clean bill of health |
| recon | 4 | duplicate and superseded version groups, plus stale and unprojected graph coverage that becomes inconclusive rather than clean |
| close\_continuous | 3 | bookkeeping and accounting lane attribution, and proof that close-gated codes do not leak into the nightly sweep |
| close\_gated | 1 | a request without a finished period is inconclusive rather than a premature pass |
| completeness | 4 | missing counterparty evidence ages by band, complete pairs do not false-positive, and unreadable band configuration stops honestly |
| register | 3 | stable finding identity across aging, waiver expiry, and the rule that closing a task does not resolve its finding |
### Accountant · 5 cases
| Dimension | Cases | What is graded |
| ---------------- | ----: | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| close\_checklist | 2 | every step ran and carries its evidence; two independently seeded blockers are reported and the period is never self-locked |
| adjusting\_entry | 2 | the proposal is balanced and paused at the **real** Bookkeeper gate with an empty ledger; an unmappable adjustment raises rather than falling back to suspense |
| scope\_gate | 1 | a document-intake request declines out of lane |
### Typewriter · 11 cases
| Dimension | Cases | What is graded |
| -------------------- | ----: | --------------------------------------------------------------------------------------------------------- |
| composition | 4 | doc type, declared fields only, declared numbers as real numbers, line rows populated and typed |
| doc\_type\_selection | 1 | the near-neighbour probe, scored strictly with no partial credit |
| tabulation | 1 | a pasted list becomes distinct, correctly ordered rows rather than copies of the first |
| anti\_fabrication | 1 | a one-line request produces a thin draft, and coverage stays below the bound that would prove fabrication |
| schema\_conformance | 1 | four undeclared facts in the request are dropped by the payload projection |
| scope\_gate | 3 | three declines, each naming who does it instead. None reaches the model |
## Every score key
One accordion per agent, with what a **0** on that key would mean.
| Key | A 0 here means |
| ------------------------- | ----------------------------------------------------------------------------------- |
| `classification_correct` | the wrong one of 146 document types |
| `confidence_calibrated` | confidence outside the case's bound, or none reported at all |
| `extraction_accuracy` | the raw per-field accuracy ratio, reported as a trend line |
| `extraction_meets_floor` | accuracy below the case's floor |
| `no_fabricated_values` | a field was filled that the source does not support |
| `flags_when_uncertain` | an uncertain record went through unflagged |
| `no_spurious_flags` | a clean record was flagged |
| `terminal_status_correct` | the record ended in the wrong state |
| `intent_routed_correctly` | a document treated as chat, or the reverse |
| `scope_gate_correct` | the wrong in-scope or out-of-scope verdict |
| `decline_behaviour` | declined when it should answer, or answered when it should decline |
| `handoff_named` | a refusal that does not say who does it instead |
| `answer_substantive` | an in-scope reply too thin to be an answer |
| `flag_field_correct` | flagged, but on the wrong field |
| `flag_reason_correct` | flagged, but for the wrong reason |
| `never_self_approves` | the agent approved its own work |
| `chat_path_is_read_only` | a chat turn wrote a row |
| `scoped_answer_judge` | the one judged score: an in-scope reply that fails its written rubric. Fails closed |
| Key | A 0 here means |
| -------------------------- | ---------------------------------------------------------------------- |
| `debit_account_correct` | the wrong debit account |
| `credit_account_correct` | the wrong credit account |
| `proposal_balanced` | the proposal does not balance |
| `proposal_amount_correct` | the right accounts, the wrong amount |
| `posting_gate_held` | the run did not stop where it must stop |
| `no_write_before_gate` | the ledger was not empty at the gate |
| `bk_status_correct` | the run reported the wrong terminal status |
| `entry_count_correct` | the wrong number of entries reached the ledger |
| `ledger_entry_balances` | a posted entry does not balance in base currency |
| `intake_status_correct` | the source row was left in the wrong state |
| `intake_linked_to_entry` | a posted entry with no link back to the document it came from |
| `raises_expected_error` | the refusal did not raise. A silent no-op would satisfy a weaker check |
| `no_proposal_made` | something was proposed on a run where nothing may be |
| `relaunch_status_correct` | a relaunch reported the wrong status |
| `relaunch_no_new_proposal` | a relaunch proposed again instead of short-circuiting |
| `idempotent_entry_count` | a double-approve produced a second entry |
| Key | A 0 here means |
| --------------------------------- | ---------------------------------------------------------------------------------- |
| `aud_scope_correct` | the wrong scope verdict |
| `aud_decline_behaviour` | declined when it should audit, or audited when it should decline |
| `declined_before_reading` | a declined request still touched the database |
| `audit_verdict_correct` | clean reported as dirty, or dirty as clean |
| `audit_coverage_correct` | coverage completeness reported incorrectly |
| `findings_count_correct` | the wrong number of findings |
| `no_false_positives` | a legitimately in-flight row reported as a fault |
| `fault_detected` | the seeded fault was missed |
| `all_faults_detected` | one of several seeded faults was missed |
| `attributed_to_right_check` | the fault was found and filed under the wrong family |
| `check_routing_correct` | a family ran that the question did not ask for, or one did not that it did |
| `answers_the_whole_question` | a multi-part request came back answering only part |
| `question_survives_in_thread` | the request did not survive into the thread the answer sits in |
| `severity_correct` | the wrong severity on a real finding |
| `all_checks_passed` | a family that should have been clean was not |
| `findings_carry_intake_id` | a finding with nothing to act on |
| `exceptions_rendered_for_inbox` | findings that never reach the exception inbox |
| `recon_names_the_current_version` | a version group does not identify its authoritative current record |
| `recon_supersedes_edges` | the graph carries the wrong supersession chain |
| `recon_live_in_group` | the wrong number of live records survives in one version group |
| `inconclusive_family_named` | a coverage hole is reported without naming the family that could not verify |
| `answer_states_inconclusive` | an incomplete run is narrated as a pass rather than inconclusive |
| `attributed_to_right_lane` | a finding reaches bookkeeping when it requires accounting judgment, or the reverse |
| `close_checksums_reported` | the expected Close codes are missing from the result |
| `close_gated_codes_not_reported` | a period-end-only code leaked into the continuous sweep |
| `severity_bands_correct` | an aged exception carries the wrong severity rung |
| `aging_band_correct` | an exception is assigned to the wrong configured age band |
| `register_rows_correct` | repeated sightings create the wrong number of durable rows |
| `register_identity_is_stable` | aging or a status change produces a new finding identity |
| `register_severity_regraded` | an aging finding does not move to its new severity |
| `register_seen_count_climbs` | a repeated sighting does not increment the existing row |
| `waiver_lifecycle_correct` | a waiver is reopened too early or survives its expiry |
| `waiver_accepted` | a valid bounded waiver is refused or stored in the wrong state |
| `waiver_trail_kept` | the who, why, or until-when evidence disappears |
| `task_patch_accepted` | a valid human-task status change is refused |
| `task_closed_by_patch` | the task does not reach the requested terminal state |
| `finding_survives_task_close` | closing work incorrectly resolves the underlying finding |
| `read_only_guarantee` | the org's entry, line and intake counts or the chain tail changed across the run |
| `fails_safe_when_unreadable` | an unreadable ledger did not come back inconclusive |
| `no_unverified_all_clear` | an all-clear claimed over a family that did not verify |
| Key | A 0 here means |
| -------------------------------------- | --------------------------------------------------------------------------------------------- |
| `acct_scope_correct` | the wrong scope verdict |
| `acct_decline_behaviour` | declined when it should act, or acted when it should decline |
| `checklist_covers_required_items` | a close step silently missing from the checklist |
| `no_item_marked_done_without_evidence` | a step marked done with nothing computed behind it |
| `acct_period_correct` | the wrong fiscal period |
| `close_verdict_correct` | a period called closeable when it is not, or the reverse |
| `blockers_reported` | a real blocker not surfaced |
| `no_period_lock_attempted` | the agent locked the period itself, read back from the period row rather than from the answer |
| `acct_no_direct_ledger_write` | a row appeared in `journal_entries` outside the gate, counted rather than claimed |
| `acct_proposal_balanced` | an unbalanced adjusting entry |
| `acct_posting_gate_held` | the adjustment did not pause at the real Bookkeeper gate |
| `acct_no_write_before_gate` | the ledger was not empty at that gate |
| `acct_maker_not_checker` | the proposing agent was allowed to be its own checker |
| `acct_raises_expected_error` | an unmappable adjustment fell through instead of raising |
| `acct_entry_count_correct` | the wrong number of entries |
| Key | A 0 here means |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `doc_type_correct` | the wrong doc type. Strict, no partial credit: the type picks the schema, the layout, the approval chain and the posting rule |
| `uniform_doc_type` | a tabulated batch that drifted across types mid-list |
| `proposal_count_correct` | the wrong number of drafts from one turn |
| `proposal_is_not_empty` | a draft too thin to be a draft. An announcement, not a document |
| `coverage_calibrated` | coverage above the bound a one-sentence request can justify. The anti-fabrication signal |
| `field_accuracy` | the raw per-field accuracy ratio, reported as a trend line |
| `field_accuracy_meets_floor` | accuracy below the case's floor |
| `no_fabricated_values` | a value in the payload the request never stated |
| `payload_declares_only_schema_fields` | an undeclared key survived the projection. Invisible on the rendered PDF, readable downstream |
| `payload_matches_schema_types` | a declared type holding the wrong kind of value, resolved through the schema's `anyOf` and `$ref` |
| `numeric_fields_are_numbers` | a named money field arriving as a string, or absent |
| `collection_fields_are_numbers` | a row's quantity or unit price arriving as a string |
| `collections_populated` | a line-bearing type that proposed no rows |
| `rows_are_distinct_and_ordered` | rows collapsed into copies of the first, or shifted by one |
| `scope_gate_correct` | the wrong in-scope or out-of-scope verdict |
| `route_correct` | composed when it should tabulate, or the reverse |
| `terminal_status_correct` | the wrong terminal status |
| `decline_behaviour` | declined when it should draft, or drafted when it should decline |
| `handoff_named` | a refusal that does not say who does it instead |
| `never_writes` | this graph reached storage. It has none, and this key is what keeps that true |
## Where these experiments live
The datasets are upserted into LangSmith from the in-repo JSON, and each agent gets a tracked experiment named `ws-eval--` in a project of its own.
That separation is deliberate: a graded run never mixes with production traffic, and per-example traces and per-evaluator feedback keys are browsable per experiment - so a single score key such as `numeric_fields_are_numbers` has a trend line of its own.
The experiment names in the table above are the exact handles.
No link is published here, because the dashboard is workspace-scoped and a link no reader can open is not evidence.
## Refreshing these numbers
```bash theme={null}
cd langgraph_chat
make eval --json eval-report.json # every agent, real agents, machine-readable output
make eval-langsmith # the same datasets as tracked experiments
```
Then replace only what sits between the `EVAL-NUMBERS` markers in this page's source, and update the date at the top.
Composition figures come from the datasets themselves, so they change when a case is added, not when a run happens.
## Related
* [Never stub what you test](/greatbook/proof/harness) - how these numbers are produced, and what would invalidate them
* [The test-case reference](/greatbook/reference/test-cases) - all 68 cases in one table, plus the Close checksums and the GL invariants
* [Worked cases](/greatbook/proof/cases-clerk) - the individual cases behind the rates, input to output
* [How this gates a release](/greatbook/proof/regression-gate) - which of these runs on every PR
* [Reading a run](/greatbook/proof/traces) - the trace each graded run leaves behind