lumeo · support AIseries B
−41%LLM spend in 60 days
“Attribution showed that one tenant's agent looped on a dead tool 2,000 times a day. That alone was 18% of our bill. We'd never have found it in provider dashboards.”
Observability + evals for LLM applications
See every token. Trust every release.
A trace for every LLM call — prompt, tools, retrieval, latency, cost. Evals that run on every deploy. A gate in CI that blocks the regression before a customer finds it.
Running in production at
lumeo
Fairbanks & Co.
kestrel_
Bramble
Tidewater
Halden Rowe
01 — Numbers, not adjectives
Measured on the hot path, reported live.
Every figure below is a real metric in a real console. The SDK adds 3 ms at p95 because it batches spans off-thread and ships them as columnar Arrow — not JSON per call.
spans ingested / day
0.0B
Across 318 projects. Peak 61k spans/s during a customer's launch week.
livelast 60 s
p95 SDK overhead
0ms
Measured in-process, wall-clock. Async flush, zero allocations on the request thread.
livep95 · 1 s buckets
cost reduction
0%
Median LLM spend cut by customers within 60 days of turning on per-feature attribution.
live$ / 1k requests
silent regressions
0/ 90d
Across gated customers. 214 regressions caught in CI; none reached production unnoticed.
liveeval pass rate
02 — Anatomy of a trace
One request, six spans, nothing hidden.
01 / 06"span": "request.received",
"trace_id": "tr_8f2a41c0",
"route": "POST /v1/chat",
"tenant": "lumeo",
"feature": "support-copilot",
"release": "v2.14.0+a81f",
"user_hash": "u_…3c9e",
"ts": "2026-10-15T09:14:02.118Z""span": "prompt.render",
"template": "support/answer@v9",
"tokens": 2118,
"variables": { "tone": "concise", "locale": "en-GB" },
"redactions": ["email", "card_number"],
"duration_ms": 14,
// full rendered prompt stored, 2 fields masked"span": "tool.search_orders",
"args": { "customer_id": "c_41b2", "status": "open" },
"result": { "count": 2, "bytes": 1844 },
"status": "ok",
"duration_ms": 212,
"retries": 0,
// 94th percentile for this tool today: 231 ms"span": "retrieval.pgvector",
"index": "help-center-v3",
"k": 12, "returned": 12,
"top_scores": [0.91, 0.88, 0.84, 0.71],
"used_in_answer": [0, 1, 3],
"duration_ms": 88"span": "llm.completion",
"model": "gpt-4.1", "temperature": 0.2,
"ttft_ms": 388, "duration_ms": 1204,
"usage": { "in": 2118, "out": 412, "cached": 1536 },
"cost_usd": 0.0041,
"finish": "stop""span": "eval.faithfulness",
"judge": "tw/faithfulness@2",
"score": 0.94, "threshold": 0.90,
"verdict": "pass",
"evidence": ["chunk 0", "chunk 1"],
"duration_ms": 31,
// async — never on the request path
03 — The console
Built like a terminal, reads like a ledger.
Filter two billion spans by tenant, model, feature or eval verdict in under 400 ms. Every row expands into the full trace. Every number links to the spans behind it.
support-copilot
TracesEvalsCostsAlertsDatasets
prod-eu-1 · v2.14.0
Views
All traces 24h Errors 38 Slow > 2 s 412 Eval failed 17 Loops > 3 4Tenants
lumeo 61% fairbanks 22% bramble 11% other 6%feature = support-copilotmodel = anylatency > 1,000 mslast 24h1,284 traces · 388 ms
| trace | route | model | tokens | cost | latency | eval | status |
|---|---|---|---|---|---|---|---|
| tr_8f2a41c0 | POST /v1/chat | gpt-4.1 | 2,118 → 412 | $0.0041 | 1,549 ms | 0.94 | ok |
| tr_8f2a3b77 | POST /v1/agent/run | claude-sonnet-4.5 | 6,402 → 1,188 | $0.0312 | 4,871 ms | 0.88 | slow |
| tr_8f2a2e19 | POST /v1/chat | gpt-4.1-mini | 1,204 → 230 | $0.0006 | 1,102 ms | 0.97 | ok |
| tr_8f2a1d05 | POST /v1/summarize | llama-3.3-70b | 9,811 → 640 | $0.0089 | 2,310 ms | 0.61 | eval fail |
| tr_8f2a0c42 | POST /v1/chat | gpt-4.1 | 2,006 → 388 | $0.0039 | 1,388 ms | 0.92 | ok |
| tr_8f29ff80 | POST /v1/agent/run | gpt-4.1 | 14,220 → 2,044 | $0.0561 | 9,204 ms | — | loop ×5 |
| tr_8f29ee11 | POST /v1/extract | claude-haiku-4.5 | 3,310 → 512 | $0.0012 | 1,044 ms | 0.99 | ok |
Cost by feature · 24h$1,284
Faithfulness · 7d0.93
Drift · triage · 09:02Pass rate fell 0.96 → 0.89 after release v2.14.0. 17 traces promoted to dataset triage-regressions.
04 — What ships in the box
Six primitives. One data model.
Traces, evals, gates, costs, alerts and datasets all share the same span. Promote a bad trace to a test. Gate a release on the test. Attribute the cost of the fix. No glue code.
01 / traces
A trace for every call, with the receipt attached.
Prompt, tool arguments, retrieved chunks, streamed output, usage, cost. OpenTelemetry-native: if you already emit spans, add one exporter.
02 / evals
Evals that run on every deploy.
LLM judges, exact match, schema, embeddings, your own code. Run on a dataset or sampled from live traffic.
03 / gates
Regression gates in CI.
A GitHub check that fails the PR when quality, latency or cost crosses a line you drew. Diff against main, not against a vibe.
04 / costs
Cost attribution per feature, per customer.
Know which tenant's agent loop cost $900 last night, before finance asks. Export to your billing system.
05 / alerts
Alerting on quality drift.
Pass rate, score distribution, refusal rate, token growth — per feature, per model. Pager, Slack, webhook. Threshold or anomaly.
06 / platform
Your region. Your cluster, if you want.
EU and US regions with zero-retention proxying for providers. Or run the whole thing in your Kubernetes with the same console.
05 — In production
Three teams, three numbers.
We publish customer numbers only after their finance or engineering lead signs off on the figure. These three did.
kestrel · fintechregulated
0silent regressions · 90 days
“Compliance asked how we know a model update didn't change our disclosures. Now the answer is a CI check with 412 cases and a signed report per release.”
tidewater · dev platformseries A
2.1→0.9seconds · p95 latency
“The waterfall made it obvious: 60% of p95 was one retrieval call waiting on a cold index. One cache later, half the latency was gone.”
06 — Trust
Prompts are your source code. We treat them that way.
Redaction runs in your process before a byte leaves. Payload storage is encrypted per tenant with keys you can revoke. Audit logs are append-only.
SOC 2 Type II
Audited annually. ISO 27001 certified in 2026. Report under NDA.
In-process redaction
PII, secrets and custom patterns masked in the SDK. The raw text never reaches us.
EU + US residency
eu-central-1 or us-west-2. Zero-retention proxy for model providers.
Self-host
Helm chart, single binary, your Postgres and object store. Same console, same SDKs.
First trace in under ten minutes
Instrument once. Ship with evidence.
One million spans a month free, no card. Then $249 a month for a team, flat — no per-seat pricing, ever.
$pip install tracewell