Debugging a Distributed Trace Through Three LLM Calls and a Human Handoff
A customer-facing error traced back through an intent classification call, a retrieval step, a generation call, and a handoff to a human agent — with the actual root cause sitting in the retrieval step, three hops away from where the error surfaced. What made the incident a twenty-minute fix instead of a multi-day investigation was a trace ID that survived every hop, span-level timing and input/output logging on every LLM call, and an explicit log of the handoff decision itself — instrumentation put in place before there was an incident to justify it.
We've written about why LLM observability is non-negotiable in production in the abstract. This is what it looks like concretely, walked through a real incident from the alert to the actual root cause — because the value of good instrumentation is hard to feel until you're the one trying to debug something without it, and easy to underinvest in until then.
The alert
A customer reported getting an unhelpful, generic response to a specific, answerable question. Not an error, not a crash — worse, in some ways, because nothing in the system flagged it as a failure. The response was fluent and confident, and it was wrong. The only reason we found out was the customer said so.
Hop one: intent classification looked fine
The trace ID from that conversation pulled up the full request chain. First stop: the intent classification call, which correctly routed the message to the right handling flow. Input, output, latency, model version — all logged, all unremarkable. Not the problem, and knowing that immediately, without re-running anything, is the entire point of logging every hop instead of only the ones that seem important in advance.
Hop two: retrieval returned confidently wrong context
The second hop, a retrieval call against the knowledge base, is where the actual root cause was sitting. The logged retrieval output showed the system had pulled back three chunks — all real, all from legitimate documents, none actually relevant to the customer's specific question. A recent content update had shifted how a particular topic was chunked, and the new chunk boundaries happened to score well on lexical overlap with the query while missing the specific detail the customer needed. Nothing crashed. Nothing looked obviously wrong in isolation. It just quietly returned the wrong context, confidently.
Hop three: generation did exactly what it was given
The generation call, hop three, wasn't at fault at all — it correctly synthesized a fluent answer from the retrieved context it was handed, which happened to be the wrong context. This is the part that's easy to misdiagnose without full-chain visibility: the failure surfaces at generation, where the user sees it, but the actual defect is upstream. Without span-level logging on the retrieval step specifically, the natural (and wrong) instinct is to start tuning the generation prompt for a problem that isn't there.
The handoff, logged as a decision
The conversation had, separately, triggered a handoff to a human agent later on for an unrelated reason — and because the handoff decision itself was logged, with the reasoning the routing logic used, it was immediately clear that handoff was correct and unrelated to the retrieval issue, rather than a second thing to investigate. Logging a routing or handoff decision as a first-class event, not just the fact that a message was sent to a human, removed an entire branch of investigation before it started.
Why this took twenty minutes
Every hop — intent classification, retrieval, generation, and the handoff decision — was independently inspectable, tied together by one trace ID, with full input and output logged at each step. Finding the root cause meant reading four log entries in sequence, not re-running the conversation blind and guessing which step to instrument after the fact. The instrumentation that made this possible was built before there was an incident that justified it — which is exactly the point. Waiting until a hard-to-debug incident happens to add tracing means the first hard incident is the expensive one.
Have a similar challenge?
Book a free 30-minute architecture call and we'll tell you honestly whether and how we can help.
Book Free Discovery Call →