Add Tracing

Problem

At 3:12 a.m., the incident agent restarted the catalog service, which was healthy. The broken service was payments. The next morning, the log had two lines: “alert received” and “done”. Nobody could say what the agent read, what it concluded, or which prompt version was live.

When something goes wrong, nobody can see which calls, tools, and context led to the answer. These are the No Tracing and Unversioned Prompt smells.

Solution

Wrap each run in a trace, and each model call, tool, and code decision in a child span, with the attributes that explain the decision: prompt and version, input, output, tokens, cost, latency, and tool results.

Before and after

 def handle_alert(alert):
-    log.info(f"alert received: {alert.name}")
-    result = agent.run(alert)
-    log.info("done")
-    return result
+    with tracer.start_as_current_span("incident.handle") as root:
+        root.set_attributes({"alert.name": alert.name, "agent.version": AGENT_VERSION,
+                             "prompt.version": PROMPT.version})
+        result = agent.run(alert)     # the agent creates child model and tool spans
+        root.set_attributes({"result.action": result.action,
+                             "cost.usd": result.cost_usd, "stop.reason": result.stop_reason})
+        return result
+
+# inside the agent loop
+with tracer.start_as_current_span(f"chat {MODEL}") as span:
+    reply = client.messages.create(...)
+    span.set_attributes({"gen_ai.operation.name": "chat",
+        "gen_ai.request.model": MODEL,
+        "gen_ai.usage.input_tokens": reply.usage.input_tokens,
+        "gen_ai.usage.output_tokens": reply.usage.output_tokens,
+        "gen_ai.response.finish_reasons": [reply.stop_reason]})

Before: the log says it started and it ended.

flowchart LR
  A(["1. Alert"]):::io --> M(["2. Agent"]):::llm
  M --> L[("3. Two log lines")]:::store

After: a tree of spans that tells the whole run.

flowchart TD
  R(["1. Trace: incident.handle"]):::io --> S1(["2. chat"]):::llm
  R --> T1["3. execute_tool read_logs"]:::tool
  R --> S2(["2. chat"]):::llm
  R --> T2["3. execute_tool restart_service"]:::tool
  S1 --> A1["4. prompt, version, tokens, cost"]:::tool
  T1 --> A2["4. arguments, result, error"]:::tool
  R --> D[("5. Trace storage")]:::store

1. Every task becomes a trace, with an identifier, user, and versions.
2. Each model call is a span with model, tokens, cost, stop reason, and (when allowed) input and output.
3. Each tool is a span with arguments, result or summary, error, and duration.
4. Attributes follow a convention, so dashboards and searches work across services.
5. Traces go to searchable storage, with defined retention and access.

Why it works

A model’s decision is a function of what went into its context. A trace stores exactly that, in order, linked to the tool that produced each piece. With it, “why did the agent restart the catalog?” becomes a reading exercise.

Reading the incident trace

This is the 3:12 a.m. trace after the refactoring (abridged):

incident.handle            4.8s   alert=payments-5xx  agent=v14  prompt=triage@7  cost=$0.052
├─ chat claude-...         1.9s   in=11,840 out=212  finish=tool_use  tools=[read_logs]
├─ execute_tool read_logs  0.3s   args={"service": "catalog"}  result="1,204 lines, 0 errors"
├─ chat claude-...         2.1s   in=13,102 out=148  finish=tool_use  tools=[restart_service]
└─ execute_tool restart_service  0.5s  args={"service": "catalog"}  result="ok"

Reading it takes a minute. The alert was about payments, but the first chat asked for the catalog’s logs. The log came back clean, and instead of growing suspicious, the model concluded the catalog needed a restart. The cause sits in the first span: prompt triage@7 carries an example where the service is always catalog.

Without the trace, the team would have argued about “the model hallucinated”. With it, the fix is concrete: replace the example in the prompt and add an eval case with this alert.

A trace is not a log with more fields

The tree structure is what sets a trace apart from a log. Every span knows its parent, so you can sum cost per task, see the critical latency path, and compare good and bad runs of the same task type. Two tasks running at the same time do not get mixed up, because every record carries the run’s identifier.

The prompt version on each span closes the loop: a bad answer points to the exact text that produced it, and the case goes into the eval with everything needed to reproduce it.

Safe mechanics

0. Measure what exists. Pick three recent incidents and try to explain each one with today’s records. Write down what was missing: the prompt? the tool result? the version? That list becomes the instrumentation requirement.
1. Pick the standard. OpenTelemetry with the gen_ai.* semantic conventions works with most observability tools (Jaeger, Grafana Tempo, Langfuse, Arize Phoenix, Datadog, among others). These conventions are still marked as in development and have already renamed attributes between versions, so pin the version you follow.
2. Define the data policy. What is stored in full, what is masked, retention, and who can read it. Mask before exporting, never after.
3. Create the root span at every entry point (request, job, event). If the run crosses services, propagate the context (the traceparent header from the W3C Trace Context standard) so everything lands in the same trace.
4. Instrument model calls, by hand or with an off-the-shelf library (OpenLLMetry and OpenInference instrument the most common SDKs).
5. Instrument tools with arguments, result (or summary), and error. The tool result is what the model read, and that is almost always where the explanation lives.
6. Version prompts and record the version on every call span.
7. Compute cost from tokens and record it per span.
8. Measure again. Go back to the three incidents (or new ones) and check that you can now explain every step.

A worked cost example

Suppose a model that charges US$ 3 per million input tokens and US$ 15 per million output tokens (illustrative prices; use your provider’s table). A span with 11,840 input tokens and 212 output tokens costs:

11,840 × 3 / 1,000,000  = US$ 0.0355
   212 × 15 / 1,000,000 = US$ 0.0032
span total               = US$ 0.0387

Recorded on every span, this number adds up on the root span by itself. Then “which alert type costs the most?” becomes a query, not a spreadsheet.

What to record and what to sample

Keep attributes and content apart. Attributes (model, tokens, cost, version, stop reason, duration, ids) are small and should exist on 100% of runs. Content (full prompt, answer, tool output) is heavy and carries personal data, so it can be sampled.

Sampling has to look at the end of the run, not the start. Sampling a random 5% up front misses exactly the bad case. Decide after the run finishes, and always keep runs with errors, high cost, a limit stop, or an action with side effects.

Risks

• Personal data in traces. Prompts and answers carry tax IDs, addresses, balances, and chat transcripts. Under privacy laws such as Brazil’s LGPD or the GDPR, storing them in an observability tool still counts as processing personal data. Mask before exporting, restrict access, and set retention.
• Storage cost. Keeping full prompts for all traffic adds up. Sample content and keep numeric attributes for everything.
• Latency. Exporting traces synchronously delays the answer. In the OpenTelemetry SDK, SimpleSpanProcessor sends each span right away; in production use BatchSpanProcessor, which sends batches in the background.
• Sampling that hides what matters. Sampling 1% can miss exactly the bad case. Always keep runs with errors, high cost, or a limit stop.
• A broken trace. A service that does not propagate context starts a new trace, and the run shows up in two unlinked pieces. In step 8, check that a task crossing services shows up whole.

Challenge

This span holds information, but it does not help explain the decision. What is missing?

with tracer.start_as_current_span("llm") as span:
    reply = client.messages.create(model=MODEL, system=SYSTEM, messages=msgs, tools=TOOLS)
    span.set_attribute("duration_ms", elapsed_ms)
    span.set_attribute("ok", True)

with tracer.start_as_current_span(f"chat {MODEL}") as span:
    reply = client.messages.create(model=MODEL, system=SYSTEM, messages=msgs, tools=TOOLS)
    span.set_attributes({
        "gen_ai.operation.name": "chat",
        "gen_ai.request.model": MODEL,
        "prompt.version": SYSTEM_VERSION,
        "gen_ai.usage.input_tokens": reply.usage.input_tokens,
        "gen_ai.usage.output_tokens": reply.usage.output_tokens,
        "gen_ai.response.finish_reasons": [reply.stop_reason],
        "tools.requested": [b.name for b in reply.content if b.type == "tool_use"],
    })

Check your understanding