Observability for Nondeterministic Systems

Debugging classical software is a search for the line where behaviour diverged from intent. Debugging a system with a model in it is different: the code is identical, the inputs differ slightly, and the output changed completely. Standard application logging is close to useless here, because it captures the control flow and discards the thing that actually determined the result.

Capture the full call, not a summary

For every model call, store the rendered prompt exactly as sent, the model identifier including version, the sampling parameters, the tools offered, the raw response, token counts, and latency. Teams routinely log a truncated prompt and then find themselves unable to reproduce an incident, because the truncated part was where the problem lived.

Storage is the objection and it is usually wrong. Compressed text is cheap, a thirty-day retention window is enough for almost all debugging, and the alternative is engineers reconstructing prompts by hand during an outage.

One trace per user intent

A single user action can produce a dozen model calls, several retrievals, and multiple tool invocations. These belong in one trace with a shared identifier, ordered, with parent-child relationships intact. The question you will actually ask during an incident is "show me everything that happened for this user's request at this time", and if that requires joining four log sources by timestamp, you will not ask it.

Track distributions, not averages

• Output length distribution catches a prompt regression faster than any quality metric; a sudden shift usually means an instruction stopped being followed.
• Tool call frequency reveals loops and avoidance, both of which are silent failures.
• Abstention rate tells you when retrieval quality moved without anyone touching retrieval.
• Retry and escalation rate is your leading indicator of a degraded provider.

Averages hide all four. Watch percentiles and histograms, and alert on shape changes rather than thresholds.

Give users a feedback channel that produces data

A thumbs-down with no context is nearly worthless. A thumbs-down that captures the trace identifier turns a complaint into a reproducible case you can add to your evaluation set the same day. This one link, from user feedback to stored trace to test case, is the highest-leverage piece of tooling most AI teams are missing.

If a user reports a bad answer and you cannot see the exact prompt that produced it, you do not have a quality problem. You have an instrumentation problem wearing a quality problem's clothes.

Sampling without lying

At volume you will sample. Sample by trace rather than by call so you never store half a conversation, always keep one hundred percent of errors, escalations, and negative feedback, and record the sampling rate alongside the data so nobody misreads a downsampled chart as reality.