Measure cost per outcome, not per token
Token pricing is the input. What the business cares about is what a qualified lead or a resolved ticket costs — and those are not the same number.
Nodexa team6 min read
Every model provider prices per token, so that is how most teams track spend. It is the wrong denominator. A cheap model that needs three attempts is more expensive than a costly one that succeeds first time, and the token bill will tell you the opposite.
Attach cost to the run, not the call
If cost accrues to the run — including retries, tool calls and the second model you fell back to — then dividing by successful outcomes gives you a number a finance team recognises. It also makes the case for a better model, which usually pays for itself.
Watch the tail
Median cost per run is reassuring and mostly useless. The p95 is where the money goes: the run that looped eleven times before hitting the step cap. Cap steps, alert on the tail, and investigate the outliers rather than the average.