> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vinmake.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> Tracing, correction logging, eval datasets. The feedback loop in detail.

# Observability

Observability is not optional. Every workflow that ships does so with tracing,
correction logging, and a changelog entry on day one. This is the bar from
[Principles #7](/brain/staging/principles).

The reason is structural: Brain learns from corrections, and the correction loop is
dead without instrumentation. Observability is the substrate that makes the
iterative approach work over a year, not just a month.

## The three layers

Brain's observability stack has three layers, and each serves a different question.

### Layer 1 — Tracing (primary tool: Langfuse, self-hosted)

**Question it answers:** what happened on this run, step by step?

For every workflow execution, Langfuse captures:

* Every LLM call (prompts, model, parameters, tokens, latency)
* Every tool call (inputs, outputs, errors)
* Every retrieval (query, results, sources)
* Every approval gate (who, when, decision)
* The full causal chain from trigger to final output

This is the layer engineers use to debug a misbehaving workflow. It's also where
prompt and tool changes are evaluated.

**Why Langfuse, self-hosted, and not LangSmith.** Brain processes real client
data — RFQ emails, client PIC names, tech packs, pricing, BOMs. That information
should not leave VinMake's infrastructure. Langfuse is open-source, self-hostable,
and has the same DX as LangSmith for the workflows we run. LangSmith cloud is a
non-starter for production internal ops data.

### Layer 2 — Correction logging (application code)

**Question it answers:** what did the human change about the AI's draft, and why?

This is the most important data Brain produces. It's not a vendor product — it's
a first-class table in Brain's Supabase.

For every AI-drafted record that a human reviews:

```text theme={null}
workflow_id
agent_id
input_source       # email message ID, file hash, etc.
ai_output          # the full draft
human_final_output # what the human approved
field_changed      # which field(s) differed
change_type        # correction | addition | deletion | recategorization
timestamp
user_id
reason_if_given    # optional, from the merchandiser
```

Common correction types we expect:

* Missing BOM items (Brain's draft missed a trim)
* Wrong garment category (jacket vs. coat vs. blazer)
* Incorrect button count (image analysis miscounted)
* Wrong fabric interpretation (polyester vs. cotton blend)
* Missing hidden trims (interlining, fusing)
* Incorrect client PIC (wrong person inferred from email signature)
* Wrong style reference (parsed from wrong field)
* Duplicate style confusion (near-match handled poorly)

These corrections are the data that drives every subsequent improvement. When a
correction type recurs, that's a signal to fix the underlying skill, template, or
prompt — not to scold the merchandiser.

### Layer 3 — Eval datasets (added at Phase 2-3)

**Question it answers:** is the current version of Brain better or worse than the
previous version on a measurable subset of fields?

This is where **Braintrust** gets added, but **not on day one**. Braintrust is an
eval platform — it runs structured tests against your model using labeled examples.
On day one, we have zero labeled examples and zero eval datasets. Adding Braintrust
now means buying a tool that sits empty.

We add Braintrust at month 2-3 when:

* The correction log has enough examples to build datasets from
* We have measurable accuracy targets (e.g., "BOM line completeness ≥ 90%")
* We need CI gates that prevent prompt or model regressions

Until then, the correction log is the eval substrate.

## Workflow debugging tools

* **LangGraph Studio** — visual debugger for Tier 2 workflows. Free, local, already
  wired into the deer-flow runtime. Engineers use it during development.
* **Langfuse self-hosted UI** — runtime trace viewer for production runs.

## What we explicitly don't use

* **LangSmith** — SaaS, sees all data. Use during very-early prototype if needed,
  but flip to Langfuse before Mai sees the system.
* **Laminar** — wired into deer-flow but ecosystem is thinner; skip unless specific
  need arises.
* **Generic LLM-only observability tools (Helicone, etc.)** — proxy-level visibility
  doesn't capture the workflow state we need.

Triple-instrumenting (LangSmith + Langfuse + Laminar) wastes engineering time and
produces inconsistent traces. **Pick one runtime tracer.**

## What ships on day one

For the first workflow ([Tech Pack → BOM](/brain/staging/workflows/tech-pack-to-bom)):

* Langfuse self-hosted instance, configured and validating traces
* LangGraph Studio for dev work
* `corrections` table in Brain's Supabase with the schema above
* Application code that writes a correction row on every human edit of a BOM draft
* Changelog entry in this documentation

## What we measure

### Workflow metrics

| Metric                       | Meaning                                    |
| ---------------------------- | ------------------------------------------ |
| Number of correction hops    | How many edits Mai makes per AI draft      |
| Draft approval rate          | How often Mai approves with no edits       |
| Field accuracy               | Per-field accuracy of extracted values     |
| Duplicate detection accuracy | Whether near-matches are flagged correctly |
| Missing field rate           | How often required fields are missed       |
| Time saved                   | Time from RFQ to quotation, vs. baseline   |
| Human review time            | Time Mai spends reviewing each draft       |
| Task completion time         | Time between workflow stages               |

### AI quality metrics

| Metric                           | Example                         |
| -------------------------------- | ------------------------------- |
| Style ref extraction accuracy    | Correct style code              |
| Client PIC extraction accuracy   | Correct buyer contact           |
| Category classification accuracy | Jacket vs. coat vs. blazer      |
| BOM completeness                 | Did we miss any trims or fabric |
| BOM over-generation              | Did we add unnecessary lines    |
| Measurement extraction accuracy  | Correct POM data placement      |
| Costing assumption quality       | Correct price source            |
| Quote calculation accuracy       | Correct markup math             |

### Business metrics

| Metric                 | Why It Matters              |
| ---------------------- | --------------------------- |
| RFQ response time      | Faster quotation turnaround |
| RFQ drop-off rate      | Missed opportunity tracking |
| Quote conversion rate  | Sales effectiveness         |
| Margin by client       | Pricing discipline          |
| Staff workload         | Team productivity           |
| Client follow-up delay | Relationship management     |

## Related

* [Principles](/brain/staging/principles) — why every shipped feature requires
  observability
* [Roadmap](/brain/staging/roadmap) — when Braintrust gets added
* [Tech Pack → BOM](/brain/staging/workflows/tech-pack-to-bom) — first workflow to
  feed the correction log