Quickstart
In ten minutes you will install the SDK, trace one real LLM call, open its span tree in the console, score it with a judge and add a gate that runs on every pull request. Python is shown first; TypeScript equivalents are in each block's second tab.
Settings → Keys. Keys starting with tw_live_ write to production projects; tw_test_ keys write to a sandbox with 24-hour retention.ConceptsThree nouns
A trace is one request through your system. A span is one timed operation inside it — prompt render, tool call, retrieval, completion, eval. A dataset is a versioned set of cases (input + expected output) that evals run against. Gates compare eval results between your branch and main.
| Span kind | Captured by default | Typical duration |
|---|---|---|
| prompt.render | template id, variables, rendered text, token count | 5–40 ms |
| tool.* | arguments, result, status, retries | 50–500 ms |
| retrieval.* | query, k, chunks with scores & sources | 30–200 ms |
| llm.completion | model, params, ttft, output, usage, cost | 300–3,000 ms |
| eval.* | judge, score, verdict, evidence | async |
Step 1Install the SDK
The package is pure Python / pure TypeScript with no native dependencies. It patches the OpenAI and Anthropic clients on import, and ships an OTLP exporter if you already run OpenTelemetry.
# Python 3.9+
pip install tracewell
# Node 18+
npm install @tracewell/sdkStep 2Configure the client
Set the key and project once. Redaction rules run inside your process before any span is serialised — the built-in rules cover emails, phone numbers, card numbers and common secret formats; add your own regexes per field.
from tracewell import Tracewell, Redact
tw = Tracewell(
api_key=os.environ["TW_KEY"],
project="support-copilot",
environment="prod-eu-1",
release=os.environ.get("GIT_SHA", "dev"),
redact=[Redact.EMAIL, Redact.CARD, Redact.regex(r"ORD-\d{6}", field="prompt")],
flush_interval_ms=200, # batches ship off-thread as Arrow
)TW_KEY, TW_PROJECT and TW_ENV are read automatically; constructor arguments win when both are set. Never commit a tw_live_ key — the CLI refuses to run with one in a tracked file.Step 3Trace your first call
Decorate the function that handles a request. Model calls inside it are patched automatically; tool and retrieval calls become child spans when you wrap them, or automatically if you use LangChain, LlamaIndex or the Vercel AI SDK.
from tracewell import trace, span
from openai import OpenAI
client = OpenAI() # patched: every completion becomes llm.completion
@trace(name="answer_ticket", capture=["prompt", "tools", "cost"])
def answer(ticket: Ticket) -> str:
with span("retrieval.pgvector", k=12) as s:
chunks = retrieve(ticket.text, k=12)
s.set_chunks(chunks) # scores + sources on the span
orders = span.tool(search_orders)(ticket.user_id)
r = client.chat.completions.create(
model="gpt-4.1",
messages=render(chunks, orders, ticket),
)
return r.choices[0].message.contentimport { trace, span } from "@tracewell/sdk";
import OpenAI from "openai";
const client = new OpenAI(); // patched
export const answer = trace("answer_ticket", async (ticket: Ticket) => {
const chunks = await span("retrieval.pgvector", () => retrieve(ticket.text, { k: 12 }));
const orders = await span.tool(searchOrders)(ticket.userId);
const r = await client.chat.completions.create({ model: "gpt-4.1", messages: render(chunks, orders, ticket) });
return r.choices[0].message.content!;
}, { capture: ["prompt", "tools", "cost"] });Step 4Open the trace
Run your service and make one request. Within about a second the trace appears in the console under Traces; the SDK also prints a link in development mode.
[tracewell] trace tr_8f2a41c0 · answer_ticket · 6 spans · 1,549 ms · $0.0041
[tracewell] https://app.tracewell.dev/support-copilot/traces/tr_8f2a41c0Click llm.completion to see the rendered prompt, the streamed output with per-token timestamps, usage and cost. Click retrieval.pgvector to see which chunks the answer actually used.
Step 5Score it with a judge
Attach an async judge to the function. Scores never run on the request path; they appear on the trace a few hundred milliseconds later and feed the drift monitor. Promote any trace to a dataset with tw datasets add or a rule in the console.
@trace(name="answer_ticket", evals=["tw/faithfulness@2", "tw/relevance@1"], sample=0.02)
def answer(ticket: Ticket) -> str: ...
# promote the trace you just made into a dataset
$ tw datasets add triage-regressions tr_8f2a41c0 --expect "Order #41882 shipped 12 Oct…"
added 1 case · triage-regressions@v8 · 413 casesStep 6Gate the pull request
Create tracewell.yaml next to your prompts and add the Action. On every PR, Tracewell runs the dataset against the branch, compares with main and posts a check. The report is attached to the check and signed.
version: 2
project: support-copilot
datasets: { core: { ref: triage-regressions@v8 } }
judges: { faithfulness: { use: tw/faithfulness@2 }, schema: { use: schema, model: OrderReply } }
gates:
- { metric: faithfulness, min: 0.90 }
- { metric: schema, min: 1.00 }
- { metric: p95_latency_ms, max: 2000 }name: eval
on: [pull_request]
jobs:
gate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: tracewell/gate@v2
with: { key: ${{ secrets.TW_KEY }}, compare: main }tw eval --compare main in your terminal to get the same table before you push. Runs are cached by dataset version and prompt hash, so a re-run with no changes is free.DoneWhere next
You have a trace, a score and a gate. From here, most teams turn on cost attribution per tenant, set a drift alert on pass rate and connect the human review queue.