Skip to main content

​
How this gates a release

A board nobody runs is a screenshot. This page states exactly what runs, where, and what happens when it fails - including the part that is not yet automatic.

​
The two halves

The eval package deliberately splits in two, and the split is what makes the offline half safe to run everywhere.

The offline half · every pull request

Dataset structure, the marker guards, and the no-dead-expectation gate.Needs no model credentials, no database and no network. Runs in CI on every PR, alongside the full test suite.

The real-agent half · on demand

The 68 cases against the real compiled graphs, the real model and a real migrated Postgres.Needs credentials and a database, so it runs from a Make target and before a release - not on every PR.
The marker that separates them is a pytest marker, deselected by default. So pytest runs the offline suite and never accidentally spends money or requires a database, while pytest -m eval runs exactly the real-agent suite.

​
What runs on every pull request

CI has three jobs - backend, the agent runtime, and the renderer - each against a real postgres:16 service container and a real alembic upgrade head. Inside the agent-runtime job, after the test suite:
That target is the CI contract, and it enforces six things.
These are guards on the harness, not on the agents.That is the point of running them on every PR: the expensive suite runs rarely, so the cheap suite has to protect it from silently decaying between runs.The most valuable of the six is the first, and it exists because the failure it prevents already happened once: the Typewriter shipped deployed and ungraded while every guard in the package stayed green, because it was never registered at all.The fix was to stop asking the registry about itself and read the deploy manifest instead.

​
What runs on demand and before a release

Useful flags: --only <example-id>, --concurrency N, --json out.json, and --strict.

​
Prerequisites, and what happens without them

A skip is never a pass, and --strict is how you say so.Normally a missing prerequisite reports SKIP and the process exits 0, because a developer without a database should still be able to run the Clerk’s cases.--strict makes skips fatal, which is what an environment that is supposed to have everything wired should use.This was verified rather than assumed: pointed at an unreachable database host, the ledger cases report [SKIP] ... no Postgres reachable, score nothing, and the process exits 0 normally and 1 under --strict.

​
What a failure means

run.py exits non-zero on a real FAIL, on a target error, and on an XPASS. It never exits non-zero on an XFAIL alone. The XPASS row is the one worth pausing on. A fixed defect turning the board red sounds backwards, and it is exactly right: the excuse now outlives the bug, and an excuse nobody removes is how a case quietly stops being graded. That is not theoretical - it is how the number-coercion fix announced itself, with three XPASS on the first run against it.

​
The honest caveat

make eval is not yet an unconditional pull-request gate, and there are two reasons rather than one.The practical reason: it needs model credentials and a Postgres in CI, which is a cost and a secrets decision, not a technical obstacle.The real reason: the Clerk’s known SOP-citation defect fails roughly one run in ten, so an unconditional gate would be intermittently red for a defect that is already recorded.That is a real defect being tracked, not a flaky test being tolerated, and the difference matters enough to say which.A gate that is red one time in ten trains everyone to re-run it, and a gate everyone re-runs is not a gate. The correct fix is to close the defect, not to loosen the gate - and until then the expensive suite runs before a release rather than on every push.

​
Why an experiment score and a CI score are comparable

The tracked experiments grade with the same evaluator functions as the local run. There is no second scoring path, no dashboard-side rubric and no separate configuration - the in-repo datasets are the source of truth and the upload rebuilds the remote copy rather than merging into it, so an edit made in the dashboard is overwritten on the next sync. A dataset is code. It lives in the repo, next to the code it grades, and it moves through review like everything else.

​
What would make this stronger

Stated plainly, because a page about gating should not pretend the gate is finished:
  1. close the Clerk’s SOP-citation defect, then make make eval an unconditional gate;
  2. wire model credentials and a Postgres into a nightly job, so the board has a trend line rather than a set of dated runs;
  3. re-sync the tracked experiments, because the current board has 68 cases and the last tracked sync covered 51.