Agent traces humans can actually read
The problem
Datadog already had the traces. Spans, tool calls, and token counts were all collected. The problem was who could read them. Following a trace meant knowing how the agent was wired together, which ruled out the product managers and the subject-matter experts, and those were the people who could tell you whether an answer was correct in the first place.
So every conversation about quality came back to whether something felt better after a prompt change. We had no way to check, and no way to tell whether a change had made things worse somewhere we weren't looking.
How it works
What I built
- It reads production traces out of Datadog and rebuilds the whole interaction: what the user asked, which agents ran, which tools they called, and what came back. Multi-agent and multi-turn, assembled into something that can be read top to bottom instead of a span waterfall.
- The review interface lives in Backstage, the developer portal internal users already have access to. Users can now filter down to the interactions worth looking at, read them, annotate them, and score the agent's behavior.
- I built it for product managers and subject-matter experts, not just engineers. The people who know what a good answer looks like were the ones who couldn't see the traces.
- Dashboards roll up quality, reliability, latency, usage, and cost, so agent quality sits next to the operational numbers instead of in a separate deck.
- Scored interactions get curated into Datadog experiment datasets. Today's human judgments roll into tomorrow's regression suites.