Observability

Observability is not optional. Every workflow that ships does so with tracing, correction logging, and a changelog entry on day one. This is the bar from Principles #7. The reason is structural: Brain learns from corrections, and the correction loop is dead without instrumentation. Observability is the substrate that makes the iterative approach work over a year, not just a month.

The three layers

Brain’s observability stack has three layers, and each serves a different question.

Layer 1 — Tracing (primary tool: Langfuse, self-hosted)

Question it answers: what happened on this run, step by step? For every workflow execution, Langfuse captures:

Every LLM call (prompts, model, parameters, tokens, latency)
Every tool call (inputs, outputs, errors)
Every retrieval (query, results, sources)
Every approval gate (who, when, decision)
The full causal chain from trigger to final output

This is the layer engineers use to debug a misbehaving workflow. It’s also where prompt and tool changes are evaluated. Why Langfuse, self-hosted, and not LangSmith. Brain processes real client data — RFQ emails, client PIC names, tech packs, pricing, BOMs. That information should not leave VinMake’s infrastructure. Langfuse is open-source, self-hostable, and has the same DX as LangSmith for the workflows we run. LangSmith cloud is a non-starter for production internal ops data.

Layer 2 — Correction logging (application code)

Question it answers: what did the human change about the AI’s draft, and why? This is the most important data Brain produces. It’s not a vendor product — it’s a first-class table in Brain’s Supabase. For every AI-drafted record that a human reviews:

workflow_id
agent_id
input_source       # email message ID, file hash, etc.
ai_output          # the full draft
human_final_output # what the human approved
field_changed      # which field(s) differed
change_type        # correction | addition | deletion | recategorization
timestamp
user_id
reason_if_given    # optional, from the merchandiser

Common correction types we expect:

Missing BOM items (Brain’s draft missed a trim)
Wrong garment category (jacket vs. coat vs. blazer)
Incorrect button count (image analysis miscounted)
Wrong fabric interpretation (polyester vs. cotton blend)
Missing hidden trims (interlining, fusing)
Incorrect client PIC (wrong person inferred from email signature)
Wrong style reference (parsed from wrong field)
Duplicate style confusion (near-match handled poorly)

These corrections are the data that drives every subsequent improvement. When a correction type recurs, that’s a signal to fix the underlying skill, template, or prompt — not to scold the merchandiser.

Layer 3 — Eval datasets (added at Phase 2-3)

Question it answers: is the current version of Brain better or worse than the previous version on a measurable subset of fields? This is where Braintrust gets added, but not on day one. Braintrust is an eval platform — it runs structured tests against your model using labeled examples. On day one, we have zero labeled examples and zero eval datasets. Adding Braintrust now means buying a tool that sits empty. We add Braintrust at month 2-3 when:

The correction log has enough examples to build datasets from
We have measurable accuracy targets (e.g., “BOM line completeness ≥ 90%”)
We need CI gates that prevent prompt or model regressions

Until then, the correction log is the eval substrate.

Workflow debugging tools

LangGraph Studio — visual debugger for Tier 2 workflows. Free, local, already wired into the deer-flow runtime. Engineers use it during development.
Langfuse self-hosted UI — runtime trace viewer for production runs.

What we explicitly don’t use

LangSmith — SaaS, sees all data. Use during very-early prototype if needed, but flip to Langfuse before Mai sees the system.
Laminar — wired into deer-flow but ecosystem is thinner; skip unless specific need arises.
Generic LLM-only observability tools (Helicone, etc.) — proxy-level visibility doesn’t capture the workflow state we need.

Triple-instrumenting (LangSmith + Langfuse + Laminar) wastes engineering time and produces inconsistent traces. Pick one runtime tracer.

What ships on day one

For the first workflow (Tech Pack → BOM):

Langfuse self-hosted instance, configured and validating traces
LangGraph Studio for dev work
corrections table in Brain’s Supabase with the schema above
Application code that writes a correction row on every human edit of a BOM draft
Changelog entry in this documentation

What we measure

Workflow metrics

Metric	Meaning
Number of correction hops	How many edits Mai makes per AI draft
Draft approval rate	How often Mai approves with no edits
Field accuracy	Per-field accuracy of extracted values
Duplicate detection accuracy	Whether near-matches are flagged correctly
Missing field rate	How often required fields are missed
Time saved	Time from RFQ to quotation, vs. baseline
Human review time	Time Mai spends reviewing each draft
Task completion time	Time between workflow stages

AI quality metrics

Metric	Example
Style ref extraction accuracy	Correct style code
Client PIC extraction accuracy	Correct buyer contact
Category classification accuracy	Jacket vs. coat vs. blazer
BOM completeness	Did we miss any trims or fabric
BOM over-generation	Did we add unnecessary lines
Measurement extraction accuracy	Correct POM data placement
Costing assumption quality	Correct price source
Quote calculation accuracy	Correct markup math

Business metrics

Metric	Why It Matters
RFQ response time	Faster quotation turnaround
RFQ drop-off rate	Missed opportunity tracking
Quote conversion rate	Sales effectiveness
Margin by client	Pricing discipline
Staff workload	Team productivity
Client follow-up delay	Relationship management

Principles — why every shipped feature requires observability
Roadmap — when Braintrust gets added
Tech Pack → BOM — first workflow to feed the correction log

​Observability

​The three layers

​Layer 1 — Tracing (primary tool: Langfuse, self-hosted)

​Layer 2 — Correction logging (application code)

​Layer 3 — Eval datasets (added at Phase 2-3)

​Workflow debugging tools

​What we explicitly don’t use

​What ships on day one

​What we measure

​Workflow metrics

​AI quality metrics

​Business metrics

​Related