Skip to main content

​
Idempotency, outbox and durability

​
What it guarantees

Re-running anything is safe. A crash mid-flight loses nothing and strands nothing. And one invoice becomes one journal entry, however many times it is uploaded, approved or retried. Three separate mechanisms carry that, at three different layers.

​
1 · Ledger idempotency

Every entry carries a source key:
The writer looks it up before it does anything else, under the advisory lock. An exact replay returns the existing entry and writes nothing. A reuse of the key with different header, line, currency, dimension or prepared base facts raises a typed source-replay conflict instead of returning an unrelated old entry. New entries store a request fingerprint over those immutable facts. Historical entries without one are compared by reconstructing the same identity from their immutable header and lines; GreatBook does not backfill a guess or reinterpret their stored checksum. That is GL invariant 10, and it is what makes retries, replays and agent re-runs safe. There is also a database-level backstop. A concurrent exact double-post resolves to the one durable entry. A concurrent request carrying different facts conflicts and the caller rolls it back.
The key is namespaced per intake row, not per document reference. That is deliberate: two genuinely distinct documents that happen to quote the same bill number must not collapse onto one entry. The duplicate problem is solved one layer up, by the duplicate-group guard, which is a control a human participates in.

​
2 · The intake pipeline: dedup at two levels

​
A repeated document reference flags, it never crashes

(org_id, doc_type, source_ref) is a document’s natural key, and it is a partial unique index. The second upload of the same reference used to die at the terminal write and strand the row at received with a null document type - an orphan nobody could act on. Now both queue stores raise a typed conflict, never a bare unique violation, and the terminal node recovers into a flagged duplicate for a human. The Clerk never merges. Deciding that two documents are the same document is a judgment with money attached, so it goes to a person.

​
A duplicate group may enter the ledger once

The other half of that posture, and the one that protects the ledger. Because the ledger’s idempotency key is namespaced per intake row, approving two copies would write two immutable entries for one invoice - and an immutable entry can only be reversed. So intake_service.approve refuses with a 409 when any other row of the same group is already accepted for posting: active, posted, or carrying a result entry id. Three properties of that rule:
  • It is stated over the group - every row connected by duplicate_of edges in either direction, transitively - never over one edge. Forward, reverse, sibling and chain are one rule, and enumerating topologies is how the next one is missed.
  • Members that are merely pending, flagged or rejected do not block. That is the reconciliation the flag exists for, and over-refusing strands the whole group.
  • The group lock must stay the first lock approve takes: the whole group, ordered by id, in one statement. Otherwise two reviewers approving two members deadlock into a Postgres serialisation failure the API renders as a 500.

​
The version-group guard

The Typewriter adds a third case. A document approved as v1 and then revised to v2 leaves v2 legitimately approvable and submittable, with a different PDF hash, therefore a different idempotency key, and nothing linking its intake row to v1’s. So the submit path asks for any live intake row filed by any version of the group - stated over the group, never over the supersedes_id edge - and files the new row flagged with duplicate_of pointing at the incumbent. That is what makes the pair visible to the duplicate-group guard, which is the control that actually protects the ledger. The incumbent read and the queue insert are one critical section, under a namespaced advisory lock held across the insert. A check outside the lock is advisory, and two versions submitted concurrently would both see no incumbent.

​
3 · The approve handoff is an outbox, not a call

intake_service.approve() writes the status change, the audit event and an outbox row in one transaction, commits, and launches the Bookkeeper only after that commit. Moving the launch back inside the transaction is the bug this shape exists to prevent: the run reads intake_queue on a different connection and cannot see an uncommitted active row, so it dies with a not-postable error and the approved document is lost. Three consequences:
  • launch_bookkeeper raises on failure by design. The single owner of launch-failure handling records the outcome on the job row and in the audit trail, and returns it in the approve response, so a reviewer learns immediately.
  • Two recovery paths exist: an operator retry endpoint, and a sweep that also picks up jobs left pending by a crash between the commit and the launch.
  • abandoned is a dead letter and is deliberately never auto-retried. A job that has exhausted recovery needs a person, not another attempt.

​
4 · Durability: the checkpoint is the source of truth

Agent runs are checkpointed in Postgres, so a run - including a paused one - survives a process or container restart. The rule that makes that safe:
The checkpoint is the in-flight source of truth; the queue row is a terminal projection.
The Clerk graph writes to intake_queue at exactly two points: ingest (idempotent on the content-hash key) and the terminal queue node (one transaction). Intermediate nodes never touch the database. So a mid-run crash leaves at most a received row that the checkpoint resumes. There is no orphan sitting at classified or extracted for someone to find later and wonder about. Retries are bounded and typed: only transient faults - network, timeout, 5xx, rate limit - are retried, never a permanent shape error. And the validate to extract repair loop is bounded, so a document the model cannot get right flags for a human instead of looping.
Use the session-mode database DSN (port 5432) for the checkpointer, not the transaction pooler (6543). Both the saver and the GL’s advisory lock need a stable session. This is the kind of configuration detail that works in development and produces intermittent, unattributable failures in production.

​
Where it is enforced

​
The tests that would fail if it broke

The dedup suite is worth a note: the in-memory queue store enforces the same partial unique index as Postgres, so the offline test exercises the real recovery path rather than a simplified one.

​
What goes wrong without it

Without ledger idempotency, a retried webhook posts the invoice twice, and because entries are immutable the fix is a reversal that every downstream reader has to understand as noise. Without the outbox, an approved document silently never reaches the Bookkeeper, and nobody finds out until a close reports approved documents with no journal entry. Without the duplicate-group guard, the same invoice posts twice under two different reviewers, each of whom did their job correctly.