Ship an AI feature and the first invoice from the model provider tells you almost nothing about what it actually costs to serve. The per-token price is the visible tenth of the iceberg. Underneath it sits retrieval, orchestration, logging, human review for the cases the model gets wrong, and the support tickets the feature generates when it is confidently incorrect. Most teams price the feature, or decide whether to keep it, using only the part they can see.
What does it really cost to serve an AI feature?
The model API bill is only one line in the true cost. A complete figure adds retrieval and vector storage, orchestration and logging infrastructure, human review or moderation for a share of outputs, and the support cost of handling mistakes. For most AI features these surrounding costs are comparable to, and sometimes larger than, the inference bill itself - which is why treating the API invoice as the cost-to-serve badly understates it.
The bill you see, and the one you do not
The inference bill is easy to track because it arrives as a single, itemised invoice: tokens in, tokens out, a price per million. That visibility is exactly why it gets treated as the whole cost. Everything else that makes the feature work - the retrieval step that fetches context before the model ever runs, the orchestration layer that calls the model, checks the response, and sometimes retries it, the logging that lets you debug a bad answer six weeks later - is infrastructure spend that shows up on a cloud bill under a dozen unrelated line items, not on the model provider’s invoice.
Retrieval and orchestration are not free
A retrieval-augmented feature runs a search before it runs a model: an embedding call, a vector database lookup, sometimes a re-ranking step. Each of those has its own cost, scales with usage the same way the model call does, and is very often left out of the unit economics entirely because it lives in a different part of the stack, billed by a different vendor, and tracked by a different team. Orchestration adds retries, fallbacks to a cheaper model when the primary one is unavailable, and the compute for the checks that decide which model to call. None of this is exotic. All of it is a real cost per request that a cost-to-serve model has to include.
The cost nobody puts in the spreadsheet: being wrong
Every AI feature gets some share of its answers wrong, and wrong answers have a cost that is real even when it never appears on an invoice: a support ticket, a human reviewer’s time, a refund, a customer who leaves. A feature with a 3% error rate and a feature with a 15% error rate can have an identical inference bill and a completely different total cost, because the second one is generating far more downstream work. Treating the API bill as the cost of the feature makes the cheap, unreliable model look like the better choice, right up until the support queue reports otherwise.
Building a real cost-to-serve for an AI feature
The method does not change because the cost happens to involve a model: define the activity - answering a query, drafting a document, classifying a ticket - and trace every cost pool that a unit of that activity consumes. Inference is one driver. Retrieval calls are another. Minutes of human review, when a share of outputs needs it, are a third, costed the same way any labour-intensive activity is costed in a TDABC model. Add them per unit of the actual work performed, not per dollar of model spend, and the feature’s real margin - and whether it is worth keeping in its current form - becomes visible for the first time.
The invoice from the model provider is the cost of the tokens. It is not the cost of the feature.
What to check before you trust an AI feature’s unit economics
Before quoting a cost per request or a margin for an AI feature, confirm that retrieval and orchestration infrastructure has been allocated to it, not left inside a shared platform cost that nobody traces back. Confirm that a realistic error rate has been estimated and priced into the model, using actual review or support data rather than the accuracy claimed in a vendor’s benchmark. And confirm the comparison between a cheaper, less accurate model and a more expensive, more accurate one includes the downstream cost of being wrong, not just the two invoices side by side.
Book a live walkthrough of CostCtrl and see how inference, retrieval, review and support costs come together into one real number per feature.
Related reading: cost-to-serve for AI features and where LLMs get overhead allocation wrong.