Observability and evals: measure against the production goal, not the vibe
A dashboard full of green metrics means nothing if none of them predict the outcome you actually ship for. Tie every trace and eval back to the end goal — or you're measuring theatre.
Most teams instrument their LLM app to death — token counts, latency histograms, request volumes — and still can't answer the only question that matters: is it actually good? The problem isn't a lack of metrics. It's that the metrics aren't wired to the outcome the product is supposed to deliver.
Start from the outcome, work backwards
Before you build a single dashboard, name the production goal in business terms: qualified leads, resolved tickets, correct extractions, answers a human doesn't have to double-check. Every eval you build should be a proxy that predicts that number — not a metric that happens to be easy to log.
- Pick one north-star outcome tied to real value, and make it non-negotiable.
- Build offline evals on a golden dataset that correlate with that outcome.
- Instrument online traces to catch the drift and edge cases your golden set misses.
Observability without evals is just logging
Traces tell you what happened; evals tell you whether it was any good. You need both, and you need them connected. A trace of a confidently wrong answer looks identical to a trace of a great one until an eval scores it. Sampling production traffic into your golden set is how the two feed each other.
Close the loop in CI
The loop only works if it gates. Wire regression evals into CI so no prompt or model change merges without clearing the bar, and feed surprising production traces back into the golden set so the bar keeps getting harder.
const score = await runEvals(candidate, goldenSet);
if (score.outcomeProxy < BASELINE) {
throw new Error('Regression on the metric that maps to revenue');
}
// only now does it shipIf your metric can be green while the product is failing its users, it's not observability — it's decoration.