Send less repeatedly
Summarize histories, retrieve only relevant chunks, cache stable prefixes, and lazy-load tool definitions.
Guide / LLM cost control
How to control LLM costs before production
Control recurring AI spend by estimating the architecture before merge, exposing uncertainty, and turning the largest cost drivers into engineering decisions.
The operating equation
LLM cost is an architecture outcome, not a token-price lookup.
A useful cost estimate begins with a baseline: how many user actions occur, how many model calls each action creates, how much context enters each call, how much output is allowed, and how often the workflow retries, falls back, or invokes tools. Model rates matter, but they multiply the design. A cheap rate attached to an uncontrolled workflow can produce a large bill.
For a simple workload, monthly cost can be approximated as request volume × calls per request × token quantity × model rate. Real systems add cached input, batch discounts, multiple models, context replay, tool schemas, failed attempts, human rework, cloud services, and uncertain adoption. The purpose of estimation is not to pretend those variables are exact. It is to make them explicit before they become production behaviour.
Eight cost levers
Control the variables that create recurring spend.
1. Model fitCompare cost per completed task, not only cost per token. A model that fails more often may create retries, fallbacks, longer prompts, and human rework that erase its lower rate.
2. Input contextMeasure the context sent on every call. Replaying full histories, documents, memory, and tool definitions can make input cost grow silently as the product matures.
3. Output ceilingsSet realistic output limits. A generous maximum is not free merely because average answers are shorter; it increases the possible tail and may encourage downstream workflows to consume longer responses.
4. RetriesRetries multiply completed-task cost. Classify the reason for each retry and cap automatic attempts. Fix validation, timeout, and prompt failures instead of budgeting for them forever.
5. Fallbacks and routingA fallback can protect reliability, but sequential attempts and escalations create correlated cost. Estimate the complete route, not only the first model.
6. Tools and agentsTool schemas, planning steps, reflection, memory, and multi-agent hand-offs add calls and context. Lazy-load tools and measure the number of model turns required to finish one task.
7. DemandSeparate user growth from calls per user. Adoption, automation frequency, and background jobs often dominate the bill after launch.
8. Price and structureFreeze the rate basis used for approval, including cache, batch, tier, and provider-specific pricing. A reproducible estimate must be explainable later when actual invoices arrive.
Why averages fail
Approve a distribution, not one optimistic number.
The expected monthly cost is useful, but it can hide a right tail created by demand spikes, longer outputs, retry storms, context growth, and fallback usage. P50 describes the median modelled outcome. P90 is a more conservative budget number: nine out of ten simulated outcomes fall below it under the declared assumptions. P95 exposes a deeper tail for systems where overruns are especially expensive.
A percentile is not a guarantee. It is conditional on the estimate basis. That is why Class1 reports estimate maturity separately from the P50/P90/P95 distribution. A mathematically precise simulation can still rest on weak volume assumptions. New workflows should be labelled as early estimates and calibrated against actual usage after deployment.
The pre-merge workflow
Make cost review part of architecture review.
1. Establish the baseline. Record the current models, calls, tokens, retries, fallbacks, tools, demand, and monthly cost basis.
2. Read the proposed change. Identify which cost levers the pull request changes and which inputs cannot be inferred from code.
3. Declare assumptions. Add expected request volume, adoption, token distributions, routing, and budget ownership instead of fabricating unknown values.
4. Simulate baseline and change together. Paired scenarios isolate the cost delta and make unchanged systems return an exact-zero delta.
5. Review the drivers. Show which variables create the P90 tail and which engineering control would lower it.
6. Approve, revise, or block. Use an advisory comment first. Introduce a blocking budget gate only when the team trusts the basis and has a clear exception process.
7. Reconcile with actuals. Compare the approved estimate with the real deployment window. Update assumptions without rewriting the historical estimate.
Controls that usually work
Reduce the tail without killing the feature.
Constrain the answer
Use structured outputs, realistic ceilings, stop conditions, and smaller responses for intermediate agent steps.
Eliminate blind retries
Retry only recoverable failures, add backoff, inspect failure classes, and stop cascades across models.
Match model to task
Route by difficulty and required capability. Escalate deliberately instead of using the most expensive model everywhere.
Budget per user action
Track calls per completed task and per active user so product growth does not hide architecture inflation.
Put a number in CI
Require review when a proposed positive P90 monthly delta exceeds the budget owned by the team.
Worked decision
A model upgrade is never only a rate change.
Imagine a pull request that moves from a smaller model to a larger one, raises the output ceiling from 1,024 to 8,192 tokens, allows five retries, and adds an expensive fallback. A spreadsheet that changes only the model rate will understate the proposal. The review must include the output distribution, retry tail, fallback frequency, request volume, and the cost of replaying context on every attempt.
The correct question is not “Is the new model more expensive?” It is “What monthly cost distribution does this architecture create, what benefit justifies it, and which control can reduce the tail?” The Class1 browser estimator demonstrates this decision with a fixed example and lets you paste a diff locally without uploading code.
FAQ
Questions teams ask about LLM cost control.
Should I optimize cost per token?
Use token rates as an input, not the final KPI. Cost per completed task captures retries, fallbacks, failed outputs, and the number of calls required to finish useful work.
When should a cost gate block a pull request?
After the team has a declared basis, a budget owner, and an exception process. Start in advisory mode and block only positive cost deltas that exceed an agreed P90 threshold.
Can code alone reveal the monthly cost?
No. Code can expose models, token ceilings, retries, fallbacks, tools, and added callsites. Demand and real call frequency require workload assumptions or actual telemetry.
How often should the estimate be updated?
Re-estimate when architecture, pricing, model routing, demand, or measured distributions change. Keep the original approved estimate so variance can be explained later.
What should I do first?
Choose one costly workflow, define the baseline and budget, run the worked estimator, and inspect the variables that drive P90 before adding a blocking policy.
Next step
Move the cost decision upstream.
Use the sandbox to see how model choice, output limits, retries, context, and budget change a pull-request decision. Then review the product workflow for applying the same method in CI.