> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vinmake.com/llms.txt
> Use this file to discover all available pages before exploring further.
# How this gates a release
> What runs on every pull request, what runs before a release, and the honest reason the real-agent suite is not yet an unconditional gate.
# How this gates a release
A board nobody runs is a screenshot.
This page states exactly what runs, where, and what happens when it fails - including the part that is **not** yet automatic.
***
## The two halves
The eval package deliberately splits in two, and the split is what makes the offline half safe to run everywhere.
Dataset structure, the marker guards, and the no-dead-expectation gate.
Needs no model credentials, no database and no network.
Runs in CI on every PR, alongside the full test suite.
The 68 cases against the real compiled graphs, the real model and a real migrated Postgres.
Needs credentials and a database, so it runs from a Make target and before a release - not on every PR.
The marker that separates them is a pytest marker, deselected by default.
So `pytest` runs the offline suite and never accidentally spends money or requires a database, while `pytest -m eval` runs exactly the real-agent suite.
***
## What runs on every pull request
CI has three jobs - backend, the agent runtime, and the renderer - each against a real `postgres:16` service container and a real `alembic upgrade head`.
Inside the agent-runtime job, after the test suite:
```bash theme={null}
make eval-structure # offline: dataset structure + the no-op-expectation guard
```
That target is the CI contract, and it enforces six things.
| Guard | What it fails on |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `test_every_served_graph_is_graded` | a graph the deploy manifest serves that no eval is bound to. **Add a sixth agent without an eval and the build goes red** |
| `test_every_target_drives_the_real_agent_graph` | a target that stopped driving the real compiled graph |
| `test_every_reference_key_is_graded_by_some_evaluator` | an expectation no evaluator reads. An ungraded expectation still shows PASS |
| `test_expected_baseline_names_only_keys_an_evaluator_emits` | a baseline marker naming a key nothing scores, which would excuse a case forever for a reason that cannot occur |
| `test_the_typewriter_mis_proposal_check_still_gates` | a marker that grew broad enough to excuse a wrong document type |
| `test_the_anti_fabrication_bound_is_never_excused` | a marker that would excuse the coverage bound the anti-fabrication case exists to enforce |
**These are guards on the harness, not on the agents.**
That is the point of running them on every PR: the expensive suite runs rarely, so the cheap suite has to protect it from silently decaying between runs.
The most valuable of the six is the first, and it exists because the failure it prevents already happened once: [the Typewriter shipped deployed and ungraded](/greatbook/proof/cases-typewriter) while every guard in the package stayed green, because it was never registered at all.
The fix was to stop asking the registry about itself and read the deploy manifest instead.
***
## What runs on demand and before a release
```bash theme={null}
cd langgraph_chat
make eval # every agent against the REAL agents; non-zero exit on failure
make eval-clerk # one agent (also -bookkeeper, -auditor, -accountant, -typewriter)
make eval-langsmith # upload the datasets and run tracked experiments
make eval-health # does tracing authenticate, and where are runs landing
pytest -m eval # the same suite, as a pytest gate
```
Useful flags: `--only `, `--concurrency N`, `--json out.json`, and `--strict`.
### Prerequisites, and what happens without them
| Need | For | Without it |
| ------------------------ | --------------------------------------- | --------------------------- |
| a logged-in `claude` CLI | Clerk, Typewriter - the real model path | those cases **SKIP** |
| a reachable Postgres | Bookkeeper, Auditor, Accountant | those cases **SKIP** |
| a tracing key | the tracked experiments only | the local run is unaffected |
**A skip is never a pass, and `--strict` is how you say so.**
Normally a missing prerequisite reports SKIP and the process exits 0, because a developer without a database should still be able to run the Clerk's cases.
`--strict` makes skips **fatal**, which is what an environment that is supposed to have everything wired should use.
This was verified rather than assumed: pointed at an unreachable database host, the ledger cases report `[SKIP] ... no Postgres reachable`, score nothing, and the process exits 0 normally and **1 under `--strict`**.
***
## What a failure means
`run.py` exits non-zero on a real **FAIL**, on a target **error**, and on an **XPASS**.
It never exits non-zero on an XFAIL alone.
| Outcome | Meaning | Exit |
| ------- | --------------------------------------------------------------------- | ------------------------ |
| `PASS` | every score 1.0 | 0 |
| `FAIL` | a score below 1.0 that no matching marker excuses | **1** |
| `XFAIL` | a recorded, understood defect reproducing exactly as declared | 0 |
| `XPASS` | a recorded defect that no longer reproduces - **the marker is stale** | **1** |
| `SKIP` | a prerequisite was missing. Never counted as a pass | 0, or 1 under `--strict` |
The XPASS row is the one worth pausing on.
A fixed defect turning the board **red** sounds backwards, and it is exactly right: the excuse now outlives the bug, and an excuse nobody removes is how a case quietly stops being graded.
That is not theoretical - it is how [the number-coercion fix announced itself](/greatbook/proof/cases-typewriter), with three XPASS on the first run against it.
***
## The honest caveat
**`make eval` is not yet an unconditional pull-request gate, and there are two reasons rather than one.**
**The practical reason:** it needs model credentials and a Postgres in CI, which is a cost and a secrets decision, not a technical obstacle.
**The real reason:** the Clerk's known SOP-citation defect fails roughly one run in ten, so an unconditional gate would be intermittently red for a defect that is already recorded.
That is **a real defect being tracked, not a flaky test being tolerated**, and the difference matters enough to say which.
A gate that is red one time in ten trains everyone to re-run it, and a gate everyone re-runs is not a gate.
The correct fix is to close the defect, not to loosen the gate - and until then the expensive suite runs before a release rather than on every push.
***
## Why an experiment score and a CI score are comparable
The tracked experiments grade with the **same evaluator functions** as the local run.
There is no second scoring path, no dashboard-side rubric and no separate configuration - the in-repo datasets are the source of truth and the upload rebuilds the remote copy rather than merging into it, so an edit made in the dashboard is overwritten on the next sync.
A dataset is code.
It lives in the repo, next to the code it grades, and it moves through review like everything else.
***
## What would make this stronger
Stated plainly, because a page about gating should not pretend the gate is finished:
1. **close the Clerk's SOP-citation defect**, then make `make eval` an unconditional gate;
2. **wire model credentials and a Postgres** into a nightly job, so the board has a trend line rather than a set of dated runs;
3. **re-sync the tracked experiments**, because the current board has 68 cases and the last tracked sync covered 51.
## Related
* [Never stub what you test](/greatbook/proof/harness) - the guards this page runs, and why each exists
* [The scoreboard](/greatbook/proof/scoreboard) - what the expensive half produced, and how to refresh it
* [Worked cases: the Typewriter](/greatbook/proof/cases-typewriter) - the XPASS mechanism, in action
* [Status](/greatbook/status) - the wider honest board for the whole system