Know the Cost per Answer Before You Ship an LLM Feature

“Price is what you pay; value is what you get.”
— Warren Buffett, quoting Benjamin Graham, 2008 letter to Berkshire Hathaway shareholders

Before an LLM feature ships, put two numbers in the launch document: what one answer costs, fully loaded, and what one answer is worth to the business. Then look at both for your heaviest users, not your average one. If the cost can exceed the value for the people who use the feature most, fix the pricing or the architecture before launch. After launch, the same fix means renegotiating with customers who already like the feature.

Most software trained product teams to ignore run cost. Serving one more page or running one more query is a rounding error, so cost was a build-time question. LLM features break that habit. Every answer carries a metered cost, the cost scales with usage, and under a flat seat price the heaviest users pay the least per use. That makes it entirely possible to ship a feature customers love that loses money every time they use it.

The argument for not worrying about it

The usual objection is that model prices keep falling, so any cost problem will solve itself, and that worrying about unit cost before product-market fit is premature optimization. Both halves have something to them. Per-token prices for a given level of capability have dropped sharply, and a seed company that spends its first quarter tuning inference cost instead of learning what users want has made a poor trade.

The trouble is that falling token prices and rising cost per answer happen at the same time. Teams spend the savings on longer context, more retrieved documents, agent loops and verification passes, because each one improves quality. The bill per answer often goes up while the price per token goes down. And a cheaper token does not change the shape of the problem: if cost grows with usage and revenue does not, a lower rate only moves the point where the feature starts losing money. The objection is right about when to optimize. It is wrong about when to know the number.

Count answers, not calls

The unit that matters is what the user experiences as one result: one answer, one summary, one drafted email, one resolved ticket. Model calls are the wrong unit, because a single answer rarely takes a single call.

A typical answer in a retrieval-backed assistant rewrites the user's question, retrieves and ranks documents, calls the main model with a long prompt full of retrieved context, and runs a second check on the output. An agent that uses tools may loop several times, and each loop resends the growing conversation, so the input for step ten includes everything from steps one to nine. Retries on timeouts and malformed output add more. Count all of it and divide by the answers actually delivered.

Then add the costs that are not tokens: the vector database, the embedding jobs that keep it fresh (the retrieval machinery described in retrieval-augmented generation in practice), logging and tracing, eval runs on every change, and any human review of outputs. These arrive on different invoices, which is why they get left out, and together they can be a meaningful share of the total.

A worked example

The numbers below are invented for illustration. They are not any provider's prices; use your provider's current rate card, such as Anthropic's published API pricing, and your own traffic.

Picture a B2B product that adds an assistant to a plan priced at $20 per seat per month. Each answer averages three model calls. The main call sends about 6,000 input tokens (system prompt, retrieved documents, conversation so far) and gets about 500 output tokens back; the other two calls are smaller. At an assumed $2 per million input tokens and $10 per million output tokens, the main call costs about $0.017. With the smaller calls, retries and a share of retrieval and logging, call it $0.04 per answer.

Four cents sounds like nothing. Now look at usage. A typical user asks five questions a working day, about 110 a month, which comes to roughly $4.40. A power user asks forty a day, about 880 a month, which comes to roughly $35. If the feature was meant to be a small part of what the $20 seat buys, the typical user is fine and the power user is a loss. If the power users sit in your largest accounts, the loss is concentrated where the revenue is, and those are the accounts least likely to accept a price change.

That is why the average misleads. Look at the distribution of answers per user, and at cost per answer by type of request, because a long-document summary and a one-line lookup can differ in cost by an order of magnitude.

What one answer is worth

The other side of the ledger is harder and matters more. Some features have a direct value per use: a support answer that deflects a ticket saves the cost of a person handling it, and the two can be compared directly, the kind of trade-off covered in running support with data and automation. Some drive revenue: an upsell, a conversion, retention in a higher plan that includes the feature. Some are table stakes, where the value is not losing deals to a competitor that has it.

Pick one and write it down, even roughly. A feature with a cost per answer and no stated value per answer cannot be judged, and it will be cut in the first budget review that asks. The thinking is the same as for any feature investment budget, except that the run cost keeps arriving every month.

Then divide by quality. If the eval set says 80% of answers are good, the cost per good answer is a quarter higher than the cost per answer. A cheaper model that passes fewer rows may not be cheaper at all, which is why cost and quality have to be measured on the same evaluation set that serves as the feature's spec.

The levers, in order of business impact

Once the number exists, there are a handful of ways to move it. Every one of them can lower quality, so each change runs against the eval set before it ships.

Pricing and limits. Usage caps, fair-use policies or a separate charge for heavy use are product decisions, and they are often the most effective lever for the power-user problem. They also have architectural consequences (metering, entitlements, usage billing), which is why pricing needs engineering in the room before the pricing page goes live.

Model choice and routing. The largest model is rarely needed for every step. Rewriting a query, classifying intent or checking a format usually runs fine on a smaller model. Routing goes further: send easy requests to a cheap model and hard ones to an expensive one, or try the cheap path first and escalate when a check fails. The eval set is what proves the router is not sending hard cases down the cheap path.

Prompt size. In retrieval-heavy features, input tokens tend to dominate the bill, as they do in the example. Retrieving twenty documents when five would do, resending a full conversation instead of a summary, and long system prompts full of untested instructions cost money on every call. Trim, then measure.

Caching. If many users ask the same questions, cache the answers. If every call shares a long, stable prefix such as the system prompt and fixed reference material, providers increasingly discount it: Anthropic, for example, bills cached prompt reads at a tenth of the base input price. Structure prompts so the stable part comes first.

Timing. Work that does not need an answer in seconds, such as nightly summaries or bulk classification, can run through batch processing, which Anthropic's Message Batches API prices at half the standard rate. The trade is latency for margin, and for back-office work it is usually a good one.

Before pulling any of these, check whether a cheaper method does the job without a model at all. Sometimes the right lever is not using the LLM for that request.

Telling finance

Cost per answer is also the number that makes the AI bill legible to a CFO. A monthly total that doubles with no explanation invites a freeze. A unit cost that held steady while answers doubled is a growth story. The case for talking to finance in unit costs, with forecasts as ranges, is made in explaining cloud and AI spend to the CFO, and LLM features are where it pays off fastest.

What good and bad look like

Good looks like a launch document with cost per answer, value per answer and the distribution of usage by customer, and a pricing decision that accounts for the heaviest decile. After launch, cost per answer sits on the same dashboard as usage, by feature and by account, and someone owns it.

Bad looks like learning the cost from the invoice. It also looks like a team that cuts the bill by moving to a cheaper model without rerunning the evals, so the savings show up in the finance report and the quality loss shows up months later in churn.

How this changes by stage

Seed. Do not optimize. Usage is small and learning matters more than margin. Do work out the cost per answer once, on paper, and check the shape: does cost grow with usage faster than revenue does? If it does, you want to know before the pricing page is set, because changing price later is far harder than setting it.

Series A and B. Usage grows and the bill becomes visible to finance. Put cost per answer on the dashboard by feature and by customer, and watch the heaviest accounts. This is the stage where routing and prompt trimming pay for themselves, and where packaging should account for heavy use.

Later stage. Finance asks for gross margin by product line, and the LLM bill becomes a line item someone owns. Cost per answer joins the release checklist next to the eval results: a change that raises it by half needs the same sign-off as a change that drops quality. Committed-spend contracts, alternative providers and running some models yourself become real options at this volume (see AI workloads beyond the big three clouds), and each one is a build, buy or wait decision that should be revisited as the company changes stage.

The question to take into your next planning meeting is simple: for each LLM feature we run, what does an answer cost our heaviest customer, what is it worth to them, and who in the room could answer that today?