TL;DR: Agent evaluations should measure compute budget because token use, retries, tool calls, and verification loops decide whether an AI agent is profitable in production. A model that looks accurate in a demo can become too expensive, slow, or unstable when real users trigger long agentic workflows.
Compute budget now belongs inside every serious agent evaluation, because agentic tasks can consume roughly 1,000x more tokens than code chat or code reasoning. That gap hurts. According to McKinsey, repeated runs of the same programming task can vary by up to 30x in total token use, based on 2026 analysis citing Bai et al. on arXiv.
I’ve seen this surprise teams. A prototype answers ten golden test questions well, the room feels good, and then production traffic exposes the real bill: retries, retrieval misses, tool loops, JSON repair, planner mistakes, and long context windows.
After 50+ AI projects across fintech, healthtech, e-commerce, legal, and marketing, we’ve learned that agent quality is not one number. Accuracy matters, of course. But speed, cost variance, trace quality, fallback behavior, and business value decide whether the system survives month two.
Why must agent evaluations measure compute budget?
Agent evaluations must measure compute budget because agents spend money while thinking, checking, searching, calling tools, and fixing their own mistakes. Chat metrics miss that. A single answer may look cheap, while the full workflow burns tokens through retrieval, reflection, retries, and validation steps that never appear in a screenshot.
According to McKinsey’s 2026 Global Survey, 20% of organizations said AI operating costs, including token costs, constrained their AI use. McKinsey also states: “Average-cost budgets will not be enough,” because agent costs vary by task, user, tool path, and retry pattern.
The catch is variance. According to McKinsey, 10% of users can account for about 65% of total token consumption in some company programs. We’ve seen a similar shape when power users ask broad questions, upload messy documents, or force agents into open-ended research. Average cost hides them. Percentiles expose them.
Our team of 10+ specialists has worked with LangChain, LangGraph, CrewAI, and Agno in production-like settings, and the lesson is blunt: if you don’t measure p95 and p99 cost per task, you don’t know the product.
What does compute budget reveal about agent quality?
Compute budget reveals whether an agent reaches the right answer directly or wanders into expensive repair loops. Two agents can score the same on final answer accuracy, yet one may need three tool calls and the other may need thirty. That difference changes latency, API spend, user trust, and incident risk.
According to Anthropic Engineering in June 2025, agents use about 4x more tokens than chat interactions, while multi-agent systems use about 15x more tokens than chats. Anthropic Engineering, research team at Anthropic, states: “For economic viability, multi-agent systems require tasks where the value of the task is high enough.”
That sentence should sit inside every agent review. Bigger architecture isn’t automatically better. It must earn its cost.
When we implemented a RAG chatbot for a fintech client, the result was a 40% reduction in support tickets in three months. But we only trusted the rollout after measuring retrieved chunks, failed retrievals, token spend per resolved ticket, and escalation rate. Good answers were not enough. The finance team needed cost per deflected ticket.
Short version: quality without compute is theater.
How should teams benchmark agents against cost?
Teams should benchmark agents with a scorecard that ties answer quality to compute budget, latency, and business outcome. A useful evaluation compares models, prompts, retrieval settings, and tool plans on the same task set, then reports cost distributions instead of one neat average.
According to Stanford HAI’s AI Index, the cost of GPT-3.5-level inference fell from $20 per million tokens in November 2022 to $0.07 by October 2024, more than 280x cheaper. Cheap tokens help, but agent loops can still erase the gain when workflows retry too often.
A practical benchmark should include easy, normal, adversarial, and messy tasks. Use real traces. Include failed runs. I recommend logging every step as structured data, because vague console text becomes useless when finance asks why one customer session cost $8.
| Evaluation dimension | What to measure | Why it matters |
|---|---|---|
| Answer quality | Human score, task pass rate, groundedness | Prevents cheap but wrong agents |
| Compute budget | Input tokens, output tokens, tool calls, model mix | Shows real cost per completed task |
| Cost variance | Median, p90, p95, p99 task cost | Finds expensive edge cases |
| Repair loops | Retry count, validation failures, JSON repair attempts | Reveals brittle agent design |
| Latency | Time to first useful action, total task time | Protects user experience |
| Business value | Deflected tickets, saved hours, revenue lift | Proves the agent is worth running |
Here’s a small Python pattern we use in early tests. It records budget per task, not just per model call.
from dataclasses import dataclass, field
from time import perf_counter
@dataclass
class AgentRunBudget:
task_id: str
input_tokens: int = 0
output_tokens: int = 0
tool_calls: int = 0
retries: int = 0
started_at: float = field(default_factory=perf_counter)
def add_model_call(self, usage: dict) -> None:
self.input_tokens += usage.get("input_tokens", 0)
self.output_tokens += usage.get("output_tokens", 0)
def add_tool_call(self) -> None:
self.tool_calls += 1
def add_retry(self) -> None:
self.retries += 1
def snapshot(self) -> dict:
return {
"task_id": self.task_id,
"input_tokens": self.input_tokens,
"output_tokens": self.output_tokens,
"tool_calls": self.tool_calls,
"retries": self.retries,
"elapsed_seconds": round(perf_counter() - self.started_at, 2),
}
This doesn’t solve governance by itself. It gives you the raw material.
Five signals every agent evaluation should track
Agent evaluations need a short list of signals that product, engineering, and finance can all read. If the scorecard is too academic, nobody uses it; if it only tracks cost, teams ship cheap failures. The better path is a mixed view: quality, budget, variance, repair, and value.
According to Gartner in June 2025, over 40% of agentic AI projects may be canceled by the end of 2027 because of escalating costs, unclear business value, or weak risk controls. Anushree Verma, Senior Director Analyst at Gartner, states: “Most agentic AI projects right now are early stage experiments or proof of concepts.”
1. Task-level cost
Measure cost per completed task, not cost per message. Agents solve workflows. A support agent might read account data, search a policy, draft an answer, verify it, and escalate when confidence is low. That is one task.
2. Token variance
Track median, p90, p95, and p99 token consumption. McKinsey states: “Cost behaves as a distribution, not a fixed unit price.” I agree. Averages make the budget look calmer than it is.
3. Repair and reverification loops
According to McKinsey, about 60% of agentic task costs can be tied to refining answers, including checking, repairing, and reverifying. This is where weak prompts, loose schemas, and poor tool descriptions become expensive.
4. Tool-call efficiency
Tool calls are not free, even when the API endpoint itself costs little. They add latency, failure points, context, and reasoning overhead. Count them, then inspect the traces that call tools repeatedly.
5. Business value per run
A costly agent can still be a good product if the task is valuable enough. When we implemented a legal document processing pipeline, it automated 80% of contract review and saved 120 hours per month. That budget made sense.
When does a higher compute budget make sense?
A higher compute budget makes sense when the agent handles high-value, high-risk, or high-complexity work that cheaper flows cannot complete reliably. Research, contract review, technical support, fraud investigation, and regulated customer service may justify stronger models, longer context, or multi-agent review. A product description generator usually won’t.
According to Anthropic Engineering’s 2025 multi-agent research system case study, a multi-agent architecture improved performance on complex research tasks but required about 15x the tokens of chat interactions. The right lesson is not “avoid multi-agent systems.” The right lesson is to report quality and compute budget together.
Sierra’s customer-service deployments show the upside. According to Sierra, a premium athletic apparel brand launched an AI agent in under two months across 19 languages, increasing automated resolution by 340% and CSAT by 27%. Vendor case studies need caution, but the pattern is useful: higher spend can work when the value metric is clear.
The honest limitation: many agent projects don’t need autonomy. A workflow with fixed steps, reliable forms, and predictable decisions may work better with rules plus one model call.
How can teams reduce compute budget without hurting quality?
Teams can reduce compute budget by making prompts shorter, limiting context, routing tasks by difficulty, caching stable outputs, and replacing agent loops with deterministic code where possible. The aim is not to starve the model. It’s to stop paying for avoidable confusion.
According to McKinsey in 2026, concise prompt and output guidelines can reduce token consumption by 30-40% in some workflows without materially affecting quality. According to the AgentDiet paper on arXiv in 2026, AgentDiet reduced input tokens by 39.9-59.7% and total computational cost by 21.1-35.9% while maintaining performance.
The best savings often look boring. Tight schemas. Smaller retrieval chunks. Better metadata. Clear stop conditions. A cheaper model for classification, a stronger model for final judgment. We tested this pattern in content operations, where an AI-powered content system helped produce 10x more blog output while keeping quality scores consistent.
But compression has a floor. Remove too much context and hallucinations rise. Force tiny answers and users ask follow-up questions, which moves cost instead of reducing it. Measure the full task.
If your team is planning an agent evaluation and wants a second set of eyes on cost, traces, RAG design, or production readiness, contact us. We can review the scorecard before the expensive lessons arrive.
Conclusion: compute budget is a product metric
Compute budget is not only an engineering metric. It is a product metric, a finance metric, and a risk metric. The teams that win with agents will ask a harder question than “did it answer correctly?” They’ll ask, “did it answer correctly at a cost, speed, and failure rate the business can defend?”
According to Goldman Sachs Research in May 2026, agentic AI could drive a 24-fold increase in token consumption by 2030, reaching 120 quadrillion tokens per month. That is a projection, not a fact, but it signals why agent evaluations need budget discipline before adoption scales.
After 50+ projects, we’ve learned that production agents fail quietly before they fail publicly. The dashboard says accuracy is fine. The invoice says otherwise. Then users discover slow edge cases, finance questions the unit economics, and engineering starts patching traces under pressure.
Measure compute early. Keep the table simple. Tie cost to value. That’s how agent evaluations become useful instead of decorative.
Sources
- arXiv — retrieved 2026-09-01
- McKinsey & Company — retrieved 2026-09-01
- Anthropic — retrieved 2026-09-01
- Stanford — retrieved 2026-09-01
