Evals that run on every deploy.
Unit tests tell you the code still runs. Evals tell you the model still answers correctly. Tracewell runs them on every pull request, against a dataset grown from your own production traces, and blocks the merge when a number crosses the line you drew.
01 — Regression gates
A check on the PR, not a dashboard after.
Open a pull request that touches a prompt, a tool or a model version. Tracewell runs the eval suite against the branch, compares every metric with main, and posts a check. Below threshold, the merge button goes grey.
- 01Diff, not absolute — a gate can say “no worse than main by more than 1 pt”, so flaky judges don't block honest work.
- 02Three dimensions — quality, latency and cost per case, all gateable.
- 03Signed report — every run produces a PDF with cases, scores and model versions. Compliance likes this.
Six ways to say “correct.”
No single scorer is right for every output. Mix deterministic checks for structure with model judges for meaning, and keep a human queue for the cases that matter most.
Model-graded rubrics
Our calibrated judges or your own prompt. Scores come with a rationale and the evidence spans they cite. Judge drift is itself monitored.
Deterministic matching
Exact, normalised, regex, contains, numeric tolerance. Zero cost, zero flakiness — use them wherever the answer has one right shape.
JSON & type checks
Pydantic, Zod or JSON Schema. Validates structured output, tool arguments and function-calling payloads on every case.
Embedding distance
Cosine similarity to a reference answer, with a threshold per case. Cheap semantic check for free-text outputs that may vary in wording.
Review queue
Route low-confidence or high-stakes cases to people. Their labels calibrate the judges and become ground truth in the dataset.
return my_rules(output)
# runs in your CI runner
Your own scorer
A function that returns a number. Runs inside your pipeline, so it can hit your database, your APIs, your secrets.
The model changed on a Tuesday. You knew by 09:02.
Providers update models silently. Users change what they ask. Tracewell scores a sample of live traffic continuously and alerts when pass rate, score distribution, refusal rate or token growth leaves its band.
Static lines you set: pass rate ≥ 0.95, refusal rate ≤ 2%, tokens/request ≤ 4k.
Seasonal bands learned per feature and hour of day. Alerts on departure, not on noise.
Split by model, tenant, locale or release so a regression in one slice isn't averaged away.
Failing traces are promoted to a dataset with the alert id. The fix ships with its own regression test.
04 — Datasets from production
Your test set writes itself.
Synthetic test cases miss what users actually type. Every trace in Tracewell can be promoted to a dataset case with one click or one rule — redacted, versioned, with the expected output you approve or edit.
- 01Rules — “every trace where faithfulness < 0.9” or “every thumbs-down” lands in a dataset automatically.
- 02Versioned — datasets are immutable snapshots; gates pin a version so results are reproducible.
- 03Balanced — stratified sampling by tenant and intent keeps the suite representative.
expected output · pending review · reviewer N. Haddad · 14 approved today
05 — Configuration
One file in the repo. Reviewed like code.
tracewell.yaml declares datasets, judges and gates. It lives next to your prompts, goes through code review and is versioned with the release it protects. The CLI runs the same suite locally in your terminal before you push.
- 01Gates compare to main —
deltathresholds for noisy judges,minfor hard floors. - 02Budget per run — cap tokens or dollars an eval run may spend.
- 03Matrix — run the suite across models or prompt versions in one go.
version: 2
project: support-copilot
datasets:
core: { ref: triage-regressions@v7 }
refunds: { ref: refunds-golden@v3 }
judges:
faithfulness: { use: tw/faithfulness@2 }
relevance: { use: tw/relevance@1 }
tone_gb: { use: ./judges/tone_gb.py }
schema: { use: schema, model: OrderReply }
gates:
- { metric: faithfulness, min: 0.90 }
- { metric: relevance, delta: -0.01 } # vs main
- { metric: tone_gb, min: 0.85 }
- { metric: schema, min: 1.00 }
- { metric: p95_latency_ms, max: 2000 }
- { metric: cost_usd, max: 0.006 }
budget: { usd_per_run: 4.00 }
matrix: { model: [gpt-4.1, claude-sonnet-4.5] }Kestrel's numbers, one quarter apart.
A regulated fintech with 14 LLM features. Before gates: regressions found by customers, in support tickets, days later. After: found by CI, in minutes, before merge.
Q2 · dashboards only
- Regressions reaching production
- 11
- Median time to detect
- 3.4 days
- Found by
- customers
- Eval cases maintained
- 60 · manual
- Compliance evidence per release
- none
Q3 · Tracewell gates
- Regressions reaching production
- 0
- Median time to detect
- 4 min
- Found by
- CI check
- Eval cases maintained
- 412 · auto
- Compliance evidence per release
- signed PDF
“The gate failed on a Friday afternoon. The model had started adding a disclaimer we're not allowed to show. Nobody would have noticed until the auditor did.”
Block the regression. Keep the Friday.
Add tracewell.yaml and a GitHub Action. The first run uses the traces you already have.