Read agent cost and quality side by side, not as one score

Ask a team running agents in production two questions: what did this agent cost last month, and is it any good? Most can answer the first from a billing dashboard. The second usually gets a pause, then an anecdote.

The obvious fix is to put both in one place. The tempting next step is to go further and combine them into a single number, a "value score" or "efficiency index" that drops the two into one column you can sort by. We think that second step is a mistake. Cost and quality belong on the same page, read side by side. They should not be fused.

Why one number is so tempting

A single score is easy to rank, easy to chart and easy to put in a weekly update. If agent A scores 82 and agent B scores 64, the conversation seems to be over.

The trouble is that any blend of cost and quality needs a weighting, and the weighting is a business judgment hidden inside arithmetic. Is a 10% drop in answer quality worth a 30% cut in spend? For a support triage agent, maybe. For an agent that drafts contract language, almost certainly not. A fused score makes that call once, silently, for every workflow at the same time.

It also loses direction. When the blended number falls, you can't tell from the number alone whether the agent got worse, got more expensive, or both. Someone has to unpack it, and at that point the score has added a step rather than removed one.

What a fused score hides

A few patterns we see often that a single number tends to mask:

• Cheaper and worse at once. A prompt change or a smaller model cuts cost per run, and quality slips a little. Blended, the two cancel out and the score barely moves. Read separately, you see a real tradeoff that someone should decide on deliberately.
• More expensive because it is retrying. Spend climbs because the agent loops on a failing tool call. The final answers may still look fine, so a quality-weighted blend stays flat. The cost line is the early warning.
• Quality measured on a different population. Quality signals usually come from a sample, while cost covers every run. Averaging a sampled number with a complete one produces something that is neither.
• A small workflow dragging a big one. Blended across workflows, a high-volume, healthy path can bury a low-volume one that is failing badly.

Start with cost per successful run, per workflow

Raw spend tells you how much you paid. It doesn't tell you what you got. The more useful cost number is cost per successful run: total spend for a workflow over a window, divided by the runs that actually achieved what the workflow is for.

A few practical points:

• Define success per workflow. "Ticket resolved without escalation", "report delivered and accepted", "order placed". This is the hard part, and it is worth doing explicitly rather than inferring it from the absence of errors.
• Keep failed runs in the numerator. Failed runs still cost money. Dividing all spend by successful runs is what makes this number honest: an agent that is cheap per call but fails half the time shows up as expensive.
• Keep the workflow grain. An agent often serves several workflows with very different economics. Averaging them together is the same mistake as fusing cost and quality, one level down.
• Say how many runs carry a label. If only some runs have a known outcome, show that coverage next to the number so nobody over-reads it.

Keep quality signals separate, and say they are sampled

Quality is usually measured differently from cost. You might run an automated grader on a sample of production runs, collect user feedback on a few, and have a person review a handful each week. Each of these is a signal with its own coverage, its own bias and its own cost.

Show them as what they are: separate lines, each labeled with its source and sample size. "Grader pass rate, 200 of 4,000 runs sampled" means something different from "thumbs-up rate from the 3% of users who clicked". Collapsing them into one quality number, and then into a cost-and-quality number, throws away exactly the context you need to trust it.

Hard outcomes and sampled judgments also deserve to stay apart. "The order was placed" is a fact. "The response was helpful" is an opinion, even when a model holds it. Both are useful. They are not interchangeable.

Watch trends, not snapshots

Cost per successful run of a few cents is neither good nor bad on its own. What matters is the direction, and whether it moved after something you changed.

Read the lines over comparable windows: this week against last, before and after a deploy, old model against new. Put cost per successful run, the production success rate and the sampled quality signals on the same time axis, and the useful questions mostly answer themselves:

• Did success rate hold while cost fell? That is a real win.
• Did cost fall while a quality signal dropped? That is a tradeoff to decide on.
• Did cost rise with no change in success or quality? Look for retries, longer contexts or a pricier model that isn't earning its keep.

A trend also tells you when to stop. If the sample is too small or the window too short to say anything, the right answer is "not enough evidence yet", not a confident number.

Ranking is fine, as long as it ranks attention

None of this means giving up on prioritization. Teams with more than a handful of agents need to know where to look first. The point is to rank which workflows need a human to look at them, not to rank agents by a blended grade. An ordered list that says "this one's cost per successful run doubled after Tuesday's deploy, and its sampled pass rate is flat" is more useful than a leaderboard of scores, because it tells you what to go and check.

Where Morse fits

This is the view we are building at Morse. Its performance brief reads spend, labeled outcomes and production-success trends next to separate sampled quality signals, ranks what needs attention, and keeps the traces one click away. It does not produce a fused score. If that is the view you want for your agents, you can join the waitlist.