Field notes on agent observability and evals: what this lane covers

Agents went from demos to production faster than the tools for understanding them. A team can ship an agent in a week and then spend months unsure whether it is improving, which runs are failing, and why the bill went up.

This lane is about that gap across the industry, not just inside Morse. We'll write about how teams observe and evaluate agents in production, what practices hold up, and where we think the tooling is heading.

Where the field is

Two toolkits grew up separately. Observability came from tracing and monitoring: capture every run, store spans, chart latency and errors. Evals came from model development: build a test set, score outputs, compare versions.

Production agents need both at once, and the seams show:

• Cost and quality are read in different places. A team sees spend on one screen and quality scores on another, and has to line them up by hand.
• Offline evals don't see production. A test set reflects the cases someone thought of. Production is full of the ones nobody did.
• Model-graded checks are cheap to start and easy to over-trust. An LLM judge is useful, but it has its own biases and its own costs.
• Traces are abundant and answers are scarce. Having every span stored is not the same as knowing which agent needs attention this week.

The positions we start from

We'll argue for a few things in this lane, and we'll try to show our reasoning each time:

• Read cost and quality side by side, not fused into one score that hides which one moved.
• Measure success on labeled production outcomes where you can get them, and keep sampled quality checks clearly separate and clearly labeled as samples.
• Treat model judges as instruments to calibrate, not as ground truth.
• Look at trends over comparable windows, not single snapshots.

The first two posts make the case for the first and third: read agent cost and quality side by side and sample your LLM judges, don't trust them blindly.

What you'll read here

Practical analysis for founders and engineers running agents: what to measure, what tends to go wrong, and what we've seen work. Where a claim depends on a study or a standard, we'll say so. Where it's our opinion, we'll say that too.

Subscribe via RSS to follow along.