Skip to main content

​
Worked cases: the Typewriter

The Typewriter turns a sentence into a schema-conformant draft document. It is the fifth deployed graph, and until recently it was the ungraded one. This page is the most useful in the tab, and not because the numbers are good.
Its first baseline was 36% - four of eleven cases passing - and that is what an eval is for.Every one of those failures was verified against the raw payload rather than trusted from the score, and all four were real defects in the agent. None was a defect in the eval.The board then moved to 55-64% when one of them was fixed, which is the measurable improvement this eval exists to show. Before it existed there was no way to state the Typewriter’s quality at all, let alone watch it move.
Nothing here is doubled. This graph has exactly one injected boundary - the model - and no storage at all, so its target calls the same dependency factory the deployed app calls, verbatim.

​
Why this agent needed an eval rather than a test

Both of its failure modes look like success on a screen. The graph always answers “Drafted a <label> with N field(s) filled”. So a wrong document type, or an empty payload, reads as a win in the chat pane and only becomes visible three steps later, on a PDF somebody has already signed. doc_type_correct is therefore scored strictly, with no partial credit, and min_filled_fields is what separates a draft from an announcement.

​
tw-compose-payment-request · the case the money defect was found on

Given “Draft a payment request for Vinh Phat Logistics for 144,000,000 VND, requested by Thai Nguyen, to settle supplier invoice 0004217.” Graded on At the first baseline, amount came back as the string "144,000,000" on every single run.
That one defect made the whole chat lane commercially unusable, and nobody had noticed.numeric_fields_are_numbers scored 0.000 on every run, without exception - "53,000,000", "4,240,000", "30%", "8%", "500", "42,000".Which means no proposal carrying an amount was savable through the builder’s validation gate at all. The Typewriter’s chat lane could not produce a usable document with money in it, and the summary line said “Drafted a payment request with 6 fields filled” every time.
And the second assertion on that list is the quieter one. approved_by is a declared field on this schema, which is exactly why it is forbidden here: a Typewriter that helpfully pre-fills an approver has proposed its own approval, and maker-is-not-checker is the rule this entire lane rests on.

​
tw-compose-purchase-order-with-lines · where most of a document’s money lives

Given “Draft a purchase order to ACME Textiles for 500 metres of cotton twill at 42,000 VND per metre and 200 metres of lining fabric at 18,000 VND per metre, with a 30% deposit.” Graded on Why line rows are graded separately from header fields. Most of a document’s money is in the rows, not the header. A header total that is a real number over rows that are all strings is a document that renders half correctly, which is worse than one that renders wrongly throughout. And one deliberate restraint. deposit_pct is graded for type only, never value. “30% deposit” is faithfully either 30 or 0.3, and grading that free choice would make the case a coin flip - while a string there is unambiguously wrong. A case that can fail for a reason nobody agrees on is a case that trains people to ignore the board.

​
tw-compose-service-invoice-near-neighbour · one wording apart

Given “Draft a service invoice (hoá đơn dịch vụ) from Sài Gòn Wash House for washing services in July 2026: 18,000,000 VND before VAT plus 1,440,000 VND of VAT.” Graded on doc_type == service_invoice strictly, amount_pretax 18,000,000 and vat_amount 1,440,000 as numbers, supplier name contains “Sài Gòn”, field accuracy ≥ 0.9. Why it is worth a case. A VAT-bearing bill from a supplier is one wording away from supplier_vat_invoice. The two carry different schemas, different renderer layouts and different posting rules, so drifting to the near neighbour produces a document that looks entirely fine and books to the wrong account later. The request names the type explicitly in both languages, which makes exactly one answer correct while still testing that the model does not drift to the generic invoice.

​
tw-compose-declared-fields-only · the control, mirrored as a case

Given a request carrying four facts the doc type does not declare: a delivery address, a vehicle plate, payment terms and a contact name. Graded on payload_declares_only_schema_fields - the four undeclared facts are dropped by the payload projection, and putting the delivery detail into the declared note field is correct behaviour and passes.
This is the eval-level mirror of a real control, and the control exists because the attack was executed.The renderer draws only declared fields. So an undeclared key is invisible on the bytes an approver signs, while surviving into the staged payload where a downstream agent’s key-precedence lookup can read it.Approve what you see, and something else travels. The projection is what closes it; this case is what keeps it closed.

​
tw-compose-underspecified-invents-nothing · coverage as an honesty signal

Given one line and nothing else: “Draft a payment request for Vinh Phat Logistics.” Graded on payee contains “Vinh Phat”, must_not_invent_fields includes amount, and - the sharp one - max_confidence: 0.5.
Confidence here is not a model self-report. It is arithmetic.The reported confidence is coverage: a computed fraction of the doc type’s declared shape that actually got filled.So a well-covered draft produced from a one-sentence request is itself the evidence of fabrication. There is no way to talk your way past this bound: filling more fields raises the number that fails the case.An invented amount is invented money on a document that authorises a settlement, and the human who opens the proposal has no way to tell a filled field from a stated one.

​
The scope gate, three ways, and none of them reaches the model

The first is a compound turn, and it is the interesting one. The first half is squarely in the Typewriter’s lane. The gate still refuses the whole turn rather than doing the half it can, because a draft handed back alongside “and I posted it” is exactly the failure the gate exists for. The second has an answer that is not another agent. Approving is a human decision, made on the document’s own page, and never by whoever wrote it. A refusal that named a different agent here would be a wrong answer wearing the right shape. The three declines cost nothing to run: that path never reaches the model at all.

​
The four defects the eval found on day one

Each was reproduced against the raw payload, and none was fixed in the same change - each is a product decision rather than an eval defect.
The second one fails safe, and that is worth separating from the others.When classify misses, the answer is honest - “I could not tell which of the 146 document types you meant”. It just fails. An agent that fails loudly one turn in three is a quality problem; an agent that silently drafts the wrong type is a control problem.
tw-compose-stock-count-sheet exists to report a gap rather than to confirm a behaviour, and it carries a marker saying so - the case is correct and the agent is not. That is the point of the expected-baseline marker: a known defect is recorded by name so the next real failure still turns the board red.

​
And then the fix, measured

The money-as-string defect was closed in its own change, because whether a proposal coerces "53,000,000" or refuses it is a call about what a proposal asserts, not an eval bug. Here is what the board did. The eval announced the fix itself. The first run against it reported three XPASS - “the defect this baseline recorded appears fixed, remove the marker” - and an XPASS fails the run, so a stale excuse cannot outlive the bug it was written for. Two details of the fix are worth stating, because they are what stops a coercion from becoming its own fabrication:
  • a written magnitude suffix is refused rather than guessed. 500tr, 50m, 500k, 500 kg and 500 metres are left verbatim, while an explicit currency mark - VND, USD, đồng - still coerces. Guessing at “500k” is inventing a zero;
  • grouping is read by shape, so 1.500 reads as 1500 and 42,000 as 42000, while 0.500 and 0.080 read as 0.5 and 0.08. A leading zero is a decimal, not a thousands group.
And the preview path coerces and renders while only the save path refuses, so a human sees what the number was read as before anything is stored.

​
Why it is 55-64% and not the ~80% first predicted

Both remaining causes are pre-existing, both are recorded, and neither is about numbers.
1. Field accuracy is graded as a raw ratio as well as against a floor.A case carrying min_extraction_accuracy: 0.85 passes the floor at 0.90 and still fails the raw key, because an example passes only when every score is 1.0. The floor is the gate; the ratio is the trend line, and the trend line is graded too.2. An intermittent empty proposal, in two shapes, and the dataset records only one.The classify miss ends the turn with zero proposals, and that shape is excused conditionally. An extract that returns nothing yields one proposal with an empty payload - and that shape is excused by nothing, because the projection reports the payload empty before number coercion ever runs, which puts it upstream of the fix.It is deliberately not marked. It is model variance that change did not investigate, and excusing an uninvestigated failure is precisely the wrong use of the mechanism - the same call the Clerk’s two failures got.

​
What the whole page adds up to

An agent shipped to production, driving a chat surface, unable to produce a document containing money - and every guard in the eval package agreed that was fine, because the agent was never registered at all. The coverage hole that allowed it is closed by reading the deploy manifest rather than the registry. The defects are recorded by name. One of them is fixed, and the number moved. That sequence - ungraded, graded, measured, fixed, measured again - is the entire argument of this tab, and the Typewriter is the only agent that has been all the way through it.