The scoreboard

Every number on this page is measured, dated and traceable to a command.

Composition figures are measured from the eval datasets in the GreatBook backend at 4972f62, 2026-08-03.Pass rates come from named runs, each with its own date, because a pass rate is a property of a run and not of a repository.

The board

Agent	Dataset	Examples	Graded reference keys	Score metrics emitted
Clerk	`ws-clerk-eval-v1`	15	67	18 (17 deterministic + 1 judge)
Bookkeeper	`ws-bookkeeper-eval-v1`	11	55	16
Auditor	`ws-auditor-eval-v1`	26	195	40
Accountant	`ws-accountant-eval-v1`	5	22	15
Typewriter	`ws-typewriter-eval-v1`	11	104	20
Total		68	443	104 distinct

104 is the count of distinct metric names. The per-agent column sums to 109 because five names are shared across agents that grade the same behaviour - decline_behaviour, handoff_named, no_fabricated_values, scope_gate_correct and terminal_status_correct.

The figure worth carrying away is 443. Examples are cheap to add; graded assertions are what decides whether a case can silently pass.

Pass rates, with their runs

Agent	Pass rate	Examples in that run	Run	Date
Clerk	87% local, 93% tracked	15	`make eval` / `ws-eval-clerk-b614cba4`	2026-07-26
Bookkeeper	100%	10	`make eval` / `ws-eval-bookkeeper-005ed0a4`	2026-07-26
Auditor	100%	10	`make eval` / `ws-eval-auditor-9bf1c5f8`	2026-07-26
Accountant	100%	5	`make eval` / `ws-eval-accountant-a030cbd6`	2026-07-26
Typewriter	36% at first baseline, 55-64% after the number-coercion fix	11	`make eval-typewriter`, four real-model runs	2026-07-26 to 2026-07-27

The tracked sync covered 51 examples, 51 scored, 0 errored, 0 skipped - so every case really was exercised against a real agent, and none was silently skipped for a missing prerequisite.

Seventeen cases in the current 68-case composition were not part of that 51-case tracked sync.The Bookkeeper row now has 11 cases, and the Auditor has grown from the 10 cases in that run to 26 across reconciliation, Close phases, document completeness, and register lifecycle behavior.The next make eval-langsmith covers all 68. Stating the gap is cheaper than presenting a historical rate as coverage of cases it never ran.

What moved, and what has never moved

Two facts are worth more than the aggregate. The Typewriter’s numbers moved when a defect was fixed, and the eval is how that is known.

Score key	First baseline	After the fix (three runs)
`collection_fields_are_numbers` (the line rows)	0.000	1.000, 1.000, 1.000
`payload_matches_schema_types`	0.250	0.875, 0.875, 1.000
`numeric_fields_are_numbers`	0.000	0.667, 0.833, 0.500
pass rate	36%	64%, 64%, 55%

Read those rows in that order, because they say different things. The first two are the defect: the line rows, where most of a document’s money lives, are clean on every run, and whole-payload type conformance reached 1.000. The third is noisier, and not because amounts are still strings - that key scores a named field 0 when it is absent as well as when it is a string, so it also counts turns that produced no draft at all. The Clerk’s failures are model variance, and they are named. Two cases: one goods-receipt-note classification miss, and one judge verdict. Neither is marked as an accepted baseline, deliberately - they are pre-existing variance that no change has investigated, and excusing an uninvestigated failure is the wrong use of that mechanism. Every deterministic Clerk key scored 1.000 on every run for the first several weeks of this dataset’s life, and the only score that had ever moved was the judge. That is no longer true, and this page says so rather than repeating a sentence that was accurate when it was written.

Coverage by dimension

Coverage is deliberate rather than incidental, and this is the table that shows it.

Clerk · 15 cases

Dimension	Cases	What is graded
classification	5	exact registry doc type against the 146-type registry, including a near-neighbour probe
extraction	4	per-field accuracy against the doc type’s real extraction schema, with money and date normalisation
uncertainty	1	the degraded fax: confidence stays low, money fields stay empty, the record flags
governance	1	a re-uploaded invoice flags as a possible duplicate, on the right field, with the right reason, and no row carries an approval
scope_gate	4	tier-1 off-topic and tier-2 cross-domain both decline with a named handoff; in-scope answers substantively and the chat path writes nothing

Bookkeeper · 11 cases

Dimension	Cases	What is graded
proposal_correctness	5	the right debit and credit accounts from a given active intake, kind-driven and doc-type fallback, balanced, gate held with an empty ledger
posting	1	the full loop produces exactly one balanced entry, with the intake row flipped to posted and stamped with its entry id
idempotency	1	a relaunch short-circuits, proposes nothing, and the entry count stays 1
unmapped_account	1	the error is actually raised, not merely “nothing posted”
separation_of_duties	2	maker-is-checker raises before the transaction opens; a rejected posting writes nothing and leaves the row active
eligibility	1	a still-pending row never reaches `propose`

Auditor · 26 cases

Dimension	Cases	What is graded
known_good	1	a clean book passes the routine families, which also proves the fresh recompute reproduces every stored checksum
hash_chain	2	a wrong `prev_checksum`; a posted line tampered with the immutability trigger disabled
invariant	3	an unbalanced entry, an entry in a closed period, and broken intake-to-GL links
orphan	1	a stale row flagged alongside a legitimately in-flight one, so false positives are graded too
routing	1	a three-part request runs and reports all three families, not only the one that raised an exception
scope_gate	2	a write request and a document-intake request both decline before any database read
fail_safe	1	an unreachable ledger comes back inconclusive and never renders as a clean bill of health
recon	4	duplicate and superseded version groups, plus stale and unprojected graph coverage that becomes inconclusive rather than clean
close_continuous	3	bookkeeping and accounting lane attribution, and proof that close-gated codes do not leak into the nightly sweep
close_gated	1	a request without a finished period is inconclusive rather than a premature pass
completeness	4	missing counterparty evidence ages by band, complete pairs do not false-positive, and unreadable band configuration stops honestly
register	3	stable finding identity across aging, waiver expiry, and the rule that closing a task does not resolve its finding

Accountant · 5 cases

Dimension	Cases	What is graded
close_checklist	2	every step ran and carries its evidence; two independently seeded blockers are reported and the period is never self-locked
adjusting_entry	2	the proposal is balanced and paused at the real Bookkeeper gate with an empty ledger; an unmappable adjustment raises rather than falling back to suspense
scope_gate	1	a document-intake request declines out of lane

Typewriter · 11 cases

Dimension	Cases	What is graded
composition	4	doc type, declared fields only, declared numbers as real numbers, line rows populated and typed
doc_type_selection	1	the near-neighbour probe, scored strictly with no partial credit
tabulation	1	a pasted list becomes distinct, correctly ordered rows rather than copies of the first
anti_fabrication	1	a one-line request produces a thin draft, and coverage stays below the bound that would prove fabrication
schema_conformance	1	four undeclared facts in the request are dropped by the payload projection
scope_gate	3	three declines, each naming who does it instead. None reaches the model

Every score key

One accordion per agent, with what a 0 on that key would mean.

Clerk · 18 keys

Key	A 0 here means
`classification_correct`	the wrong one of 146 document types
`confidence_calibrated`	confidence outside the case’s bound, or none reported at all
`extraction_accuracy`	the raw per-field accuracy ratio, reported as a trend line
`extraction_meets_floor`	accuracy below the case’s floor
`no_fabricated_values`	a field was filled that the source does not support
`flags_when_uncertain`	an uncertain record went through unflagged
`no_spurious_flags`	a clean record was flagged
`terminal_status_correct`	the record ended in the wrong state
`intent_routed_correctly`	a document treated as chat, or the reverse
`scope_gate_correct`	the wrong in-scope or out-of-scope verdict
`decline_behaviour`	declined when it should answer, or answered when it should decline
`handoff_named`	a refusal that does not say who does it instead
`answer_substantive`	an in-scope reply too thin to be an answer
`flag_field_correct`	flagged, but on the wrong field
`flag_reason_correct`	flagged, but for the wrong reason
`never_self_approves`	the agent approved its own work
`chat_path_is_read_only`	a chat turn wrote a row
`scoped_answer_judge`	the one judged score: an in-scope reply that fails its written rubric. Fails closed

Bookkeeper · 16 keys

Key	A 0 here means
`debit_account_correct`	the wrong debit account
`credit_account_correct`	the wrong credit account
`proposal_balanced`	the proposal does not balance
`proposal_amount_correct`	the right accounts, the wrong amount
`posting_gate_held`	the run did not stop where it must stop
`no_write_before_gate`	the ledger was not empty at the gate
`bk_status_correct`	the run reported the wrong terminal status
`entry_count_correct`	the wrong number of entries reached the ledger
`ledger_entry_balances`	a posted entry does not balance in base currency
`intake_status_correct`	the source row was left in the wrong state
`intake_linked_to_entry`	a posted entry with no link back to the document it came from
`raises_expected_error`	the refusal did not raise. A silent no-op would satisfy a weaker check
`no_proposal_made`	something was proposed on a run where nothing may be
`relaunch_status_correct`	a relaunch reported the wrong status
`relaunch_no_new_proposal`	a relaunch proposed again instead of short-circuiting
`idempotent_entry_count`	a double-approve produced a second entry

Auditor · 40 keys

Key	A 0 here means
`aud_scope_correct`	the wrong scope verdict
`aud_decline_behaviour`	declined when it should audit, or audited when it should decline
`declined_before_reading`	a declined request still touched the database
`audit_verdict_correct`	clean reported as dirty, or dirty as clean
`audit_coverage_correct`	coverage completeness reported incorrectly
`findings_count_correct`	the wrong number of findings
`no_false_positives`	a legitimately in-flight row reported as a fault
`fault_detected`	the seeded fault was missed
`all_faults_detected`	one of several seeded faults was missed
`attributed_to_right_check`	the fault was found and filed under the wrong family
`check_routing_correct`	a family ran that the question did not ask for, or one did not that it did
`answers_the_whole_question`	a multi-part request came back answering only part
`question_survives_in_thread`	the request did not survive into the thread the answer sits in
`severity_correct`	the wrong severity on a real finding
`all_checks_passed`	a family that should have been clean was not
`findings_carry_intake_id`	a finding with nothing to act on
`exceptions_rendered_for_inbox`	findings that never reach the exception inbox
`recon_names_the_current_version`	a version group does not identify its authoritative current record
`recon_supersedes_edges`	the graph carries the wrong supersession chain
`recon_live_in_group`	the wrong number of live records survives in one version group
`inconclusive_family_named`	a coverage hole is reported without naming the family that could not verify
`answer_states_inconclusive`	an incomplete run is narrated as a pass rather than inconclusive
`attributed_to_right_lane`	a finding reaches bookkeeping when it requires accounting judgment, or the reverse
`close_checksums_reported`	the expected Close codes are missing from the result
`close_gated_codes_not_reported`	a period-end-only code leaked into the continuous sweep
`severity_bands_correct`	an aged exception carries the wrong severity rung
`aging_band_correct`	an exception is assigned to the wrong configured age band
`register_rows_correct`	repeated sightings create the wrong number of durable rows
`register_identity_is_stable`	aging or a status change produces a new finding identity
`register_severity_regraded`	an aging finding does not move to its new severity
`register_seen_count_climbs`	a repeated sighting does not increment the existing row
`waiver_lifecycle_correct`	a waiver is reopened too early or survives its expiry
`waiver_accepted`	a valid bounded waiver is refused or stored in the wrong state
`waiver_trail_kept`	the who, why, or until-when evidence disappears
`task_patch_accepted`	a valid human-task status change is refused
`task_closed_by_patch`	the task does not reach the requested terminal state
`finding_survives_task_close`	closing work incorrectly resolves the underlying finding
`read_only_guarantee`	the org’s entry, line and intake counts or the chain tail changed across the run
`fails_safe_when_unreadable`	an unreadable ledger did not come back inconclusive
`no_unverified_all_clear`	an all-clear claimed over a family that did not verify

Accountant · 15 keys

Key	A 0 here means
`acct_scope_correct`	the wrong scope verdict
`acct_decline_behaviour`	declined when it should act, or acted when it should decline
`checklist_covers_required_items`	a close step silently missing from the checklist
`no_item_marked_done_without_evidence`	a step marked done with nothing computed behind it
`acct_period_correct`	the wrong fiscal period
`close_verdict_correct`	a period called closeable when it is not, or the reverse
`blockers_reported`	a real blocker not surfaced
`no_period_lock_attempted`	the agent locked the period itself, read back from the period row rather than from the answer
`acct_no_direct_ledger_write`	a row appeared in `journal_entries` outside the gate, counted rather than claimed
`acct_proposal_balanced`	an unbalanced adjusting entry
`acct_posting_gate_held`	the adjustment did not pause at the real Bookkeeper gate
`acct_no_write_before_gate`	the ledger was not empty at that gate
`acct_maker_not_checker`	the proposing agent was allowed to be its own checker
`acct_raises_expected_error`	an unmappable adjustment fell through instead of raising
`acct_entry_count_correct`	the wrong number of entries

Typewriter · 20 keys

Key	A 0 here means
`doc_type_correct`	the wrong doc type. Strict, no partial credit: the type picks the schema, the layout, the approval chain and the posting rule
`uniform_doc_type`	a tabulated batch that drifted across types mid-list
`proposal_count_correct`	the wrong number of drafts from one turn
`proposal_is_not_empty`	a draft too thin to be a draft. An announcement, not a document
`coverage_calibrated`	coverage above the bound a one-sentence request can justify. The anti-fabrication signal
`field_accuracy`	the raw per-field accuracy ratio, reported as a trend line
`field_accuracy_meets_floor`	accuracy below the case’s floor
`no_fabricated_values`	a value in the payload the request never stated
`payload_declares_only_schema_fields`	an undeclared key survived the projection. Invisible on the rendered PDF, readable downstream
`payload_matches_schema_types`	a declared type holding the wrong kind of value, resolved through the schema’s `anyOf` and `$ref`
`numeric_fields_are_numbers`	a named money field arriving as a string, or absent
`collection_fields_are_numbers`	a row’s quantity or unit price arriving as a string
`collections_populated`	a line-bearing type that proposed no rows
`rows_are_distinct_and_ordered`	rows collapsed into copies of the first, or shifted by one
`scope_gate_correct`	the wrong in-scope or out-of-scope verdict
`route_correct`	composed when it should tabulate, or the reverse
`terminal_status_correct`	the wrong terminal status
`decline_behaviour`	declined when it should draft, or drafted when it should decline
`handoff_named`	a refusal that does not say who does it instead
`never_writes`	this graph reached storage. It has none, and this key is what keeps that true

Where these experiments live

The datasets are upserted into LangSmith from the in-repo JSON, and each agent gets a tracked experiment named ws-eval-<agent>-<id> in a project of its own. That separation is deliberate: a graded run never mixes with production traffic, and per-example traces and per-evaluator feedback keys are browsable per experiment - so a single score key such as numeric_fields_are_numbers has a trend line of its own. The experiment names in the table above are the exact handles. No link is published here, because the dashboard is workspace-scoped and a link no reader can open is not evidence.

Refreshing these numbers

cd langgraph_chat

make eval --json eval-report.json   # every agent, real agents, machine-readable output
make eval-langsmith                 # the same datasets as tracked experiments

Then replace only what sits between the EVAL-NUMBERS markers in this page’s source, and update the date at the top. Composition figures come from the datasets themselves, so they change when a case is added, not when a run happens.

Never stub what you test - how these numbers are produced, and what would invalidate them
The test-case reference - all 68 cases in one table, plus the Close checksums and the GL invariants
Worked cases - the individual cases behind the rates, input to output
How this gates a release - which of these runs on every PR
Reading a run - the trace each graded run leaves behind

The Claim

The Numbers

Worked Cases

Evidence

The scoreboard

The scoreboard

The board

Pass rates, with their runs

What moved, and what has never moved

Coverage by dimension

Clerk · 15 cases

Bookkeeper · 11 cases

Auditor · 26 cases

Accountant · 5 cases

Typewriter · 11 cases

Every score key

Where these experiments live

Refreshing these numbers

​The scoreboard

​The board

​Pass rates, with their runs

​What moved, and what has never moved

​Coverage by dimension

​Clerk · 15 cases

​Bookkeeper · 11 cases

​Auditor · 26 cases

​Accountant · 5 cases

​Typewriter · 11 cases

​Every score key

​Where these experiments live

​Refreshing these numbers

​Related

The scoreboard

The board

Pass rates, with their runs

What moved, and what has never moved

Coverage by dimension

Clerk · 15 cases

Bookkeeper · 11 cases

Auditor · 26 cases

Accountant · 5 cases

Typewriter · 11 cases

Every score key

Where these experiments live

Refreshing these numbers

Related