Software taught a generation of founders that gross margin is a solved problem. You build the product once, the next customer costs almost nothing to serve, and the margin line settles somewhere north of 80 per cent on its own. AI products do not behave that way. Every request costs real money, that cost moves with how hard each customer uses the product rather than with how many customers you have, and gross margin stops being a consequence of scale. It becomes something you have to design.
Why do AI startups have a gross margin problem?
Because inference is a variable cost that grows with usage, not a fixed cost that dilutes with growth. A traditional software company adds customers and watches gross margin climb; an AI company adds customers and adds cost in proportion. The margin is decided by how much compute each unit of delivered value consumes, which requests customers actually send, and how much free and heavy usage the paying accounts are quietly subsidising. It has to be measured per feature and per customer segment, because the blended average hides the accounts that are underwater.
Why the margin does not improve on its own
The classic software margin curve comes from a cost base that is mostly fixed. Hosting, storage and bandwidth per customer are small enough that growth spreads the engineering cost over more revenue and the percentage climbs year after year. An AI product has a second cost base sitting underneath that one, and it is variable by construction: the model call, the retrieval, the re-run when the first answer was not good enough. Doubling revenue doubles that cost too, so the margin percentage stays roughly where it started unless something about the product changes. Founders who budget on the assumption that margin improves with scale are budgeting for a curve their product does not have.
Cost of revenue is wider than the model bill
The invoice from the model provider is the visible part and rarely the whole of it. A working AI feature also carries the embedding and vector storage that make retrieval possible, the evaluation and guardrail calls that check an answer before it reaches a user, the retries when a call fails or returns something unusable, and in most serious products some amount of human review on the outputs that matter. Each of those is a cost of delivering the service, which makes it cost of revenue, not research and development. Leaving them in an engineering or infrastructure line does not make them disappear; it only moves the margin problem somewhere nobody is looking at it.
The free tier is a cost centre, not a marketing line
In conventional software a free tier costs a rounding error and buys distribution. In an AI product the free tier consumes the same expensive compute as the paid one, and the users who consume the most of it are, by definition, the ones who found the product most useful. That is a genuine acquisition strategy, and it is also a real number that belongs in the margin calculation rather than in a marketing budget nobody reconciles against usage. The question is not whether to run a free tier. It is whether anyone can say, this month, what it cost and which paying cohort absorbed it.
Pricing cannot fix a cost you cannot see
The usual reaction to a thin AI margin is a pricing change: raise the seat price, add usage limits, move to a credit model. All three can work, and none of them can be aimed without knowing which features and which customers consume the compute. A per-seat increase spread across the whole base punishes the light users who were already profitable and barely touches the heavy accounts that caused the problem. Usage limits set from an average cut off customers the product was designed for. The pricing lever is real, but it is only useful once cost is traced to the activity that causes it, one feature and one segment at a time.
A blended gross margin is an average of the products you make money on and the products you do not. Averages do not survive diligence.
What to check before the next board pack
Before gross margin goes into a board deck or a data room, confirm three things. That cost of revenue includes inference, retrieval, evaluation, retries and human review, and not just the model invoice. That margin is reported per feature and per plan tier as well as blended, so the loss-making combinations are visible instead of averaged away. And that free and trial usage is costed and attributed rather than treated as marketing spend. An investor who asks how margin behaves at ten times the volume is asking whether anyone has done that work, and the honest answer is visible in about a minute.
Book a live walkthrough of CostCtrl and see how inference, retrieval, evaluation and review costs come together into a real margin per feature and per customer.
Related reading: AI gross margin and per-token economics and pricing AI products profitably.