Retry only recoverable failures
Separate transport, rate-limit, validation, quality and business-rule failures before choosing a response.
Guide / Reliability economics
How retries and fallbacks multiply LLM cost
Retries and fallbacks are reliability controls, but they are also cost multipliers. They can repeat context, regenerate output, call a more expensive model and extend an agent route. A forecast that prices only the first call will understate both expected spend and the monthly tail.
The basic mechanism
Every additional attempt consumes another route.
A retry is not free simply because the user sees one final answer. The system may resend the original prompt, conversation history, retrieved documents and tool schemas, then generate another output. If the retry follows a long timeout, the first provider may already have processed some or all of the request. If the retry changes models or providers, the new attempt may use a different rate and token pattern.
A fallback adds another branch. The system may call a primary model, inspect failure or low confidence, then call a stronger or alternative model. In some designs, both models run in parallel and the system selects a result. In others, a router, evaluator or repair model adds further calls. The cost estimate must represent the complete path rather than the advertised primary model.
Failure classes
Different failures require different cost assumptions.
Transport failureConnection errors, provider unavailability or gateway failures may trigger a new request before the original state is known.
TimeoutThe client stops waiting, but the provider may have consumed input and generated output. Retrying can create duplicate cost and duplicate business actions.
Rate limitImmediate retries can worsen congestion. Backoff may reduce repeated failures but increases latency and queue duration.
Invalid structureMalformed JSON, missing fields or schema violations can create repair prompts, repeated generation or escalation to a stronger model.
Low-quality resultAn evaluator, user or business rule rejects the output. The next attempt may use more context, a new prompt or a different model.
Tool failureAn agent may repeat planning and tool calls after unavailable data, invalid arguments or partial side effects.
Safety or policy rejectionA blocked response may route to a safer workflow, human review or a constrained model rather than a simple retry.
Retry multiplier
Do not model retries as a fixed whole number.
A configuration such as “maximum retries equals three” does not mean every call costs four attempts. It defines the upper bound. The estimate needs a distribution: the probability of zero retries, one retry, two retries and the deep tail. Those probabilities can differ by provider, model, task type, prompt version and traffic condition.
For an early estimate, declare a median and conservative retry factor rather than pretending to know the exact frequency. For an operating system, measure attempts per completed task and classify the reason for every retry. The mean shows overall inflation, while P90 or P95 exposes periods when correlated failures create a large monthly overrun.
Repeated attempts are not always identical. A repair prompt may be shorter than the original request, while an agent restart may replay a much larger history. A fallback model may have a higher rate but produce a shorter result. Model each route according to its actual quantities when the difference is material.
Correlated failure
The dangerous tail appears when failures are not independent.
A simple forecast may assume every request has the same small retry probability. Real incidents often affect many requests together. A provider outage, bad deployment, broken tool contract or overloaded dependency can create a retry storm across the entire workload.
Correlation changes the monthly distribution. Instead of a few independent extra attempts, a large share of traffic may enter the expensive route at once. Queues grow, timeouts increase and automatic retries add pressure. Fallback providers may also share upstream dependencies or regional constraints.
Budget cases should therefore include incident or regime scenarios, not only independent per-call probabilities. The P90 tail may be driven more by a few correlated periods than by normal daily variation.
Fallback patterns
Architecture determines whether a fallback saves or multiplies cost.
Sequential fallbackCall the primary route first, then call an alternative after failure or low confidence. Cost includes both attempts when fallback occurs.
Parallel hedgeRun two providers or models together to reduce latency or outage risk. Reliability improves, but both routes may be billed.
Capability escalationStart with an efficient model and escalate difficult tasks. This can lower average cost when routing is accurate.
Repair modelUse another model to fix structure or validate output. The repair route may be economical if it avoids repeating the full task.
Human fallbackRoute unresolved tasks to a person. Labour and delay belong in the completed-task economics.
Estimation method
Map routes, probabilities and quantities explicitly.
Begin with the first-pass route: model, input tokens, output distribution, tool cost and normal latency. Add each retry route and state what is repeated. Add fallbacks and identify whether they are sequential, parallel or conditional. Record the trigger for every transition and the probability that the route reaches an accepted completion.
Calculate the cost of each route using a frozen, dated rate basis. Weight routes by probability to estimate expected cost. Simulate uncertainty in demand, output, context, retry frequency and fallback rate to produce P50, P90 and P95 monthly totals. Keep estimate maturity separate from the percentile calculation.
For production calibration, record task identifiers, attempt numbers, failure classes, route transitions, tokens, cost and final completion. This evidence replaces generic assumptions with team-specific distributions and reveals which failure class creates the largest cost tail.
Controls
Reliability should be bounded, observable and economically justified.
Reduce synchronized pressure
Use bounded exponential backoff and jitter rather than immediate repeated attempts.
Prevent duplicate side effects
Attach stable task and operation identifiers so timeouts do not duplicate writes, purchases or messages.
Repeat the smallest useful unit
Repair invalid structure without replaying the complete agent workflow when the task result remains usable.
Stop retry storms
Open a circuit when a dependency is unhealthy and route or queue work according to a declared incident policy.
Set route and task limits
Cap attempts, tokens, tool calls, elapsed time and cost per task, then require an exception for further escalation.
Observability
A retry count without a reason is not enough.
Record the failure class, provider, model, route, prompt version, attempt number, elapsed time and final outcome. Tie all attempts to one business-task identifier. This allows the team to calculate attempts per completed task, cost by failure class and the share of tasks that enter each fallback route.
Monitor retry rate, fallback rate, first-pass completion, total completion, duplicate-operation prevention, human escalation and cost per completed task. Segment by task type and traffic regime. A stable average can conceal one difficult segment or an incident-driven tail.
Alerting should focus on economic and reliability impact. A small increase in retry rate may be harmless at low volume, while a correlated shift across a high-volume workflow can quickly exceed the monthly budget. Compare actual behaviour with the assumptions approved before deployment.
Pre-merge review
Retry policy is an architectural cost decision.
Code review should identify changes to maximum retries, timeout behaviour, fallback models, provider routing, agent turn limits and tool recovery. A pull request that changes any of those controls can materially alter recurring spend even if the primary model stays the same.
The cost report should show the baseline and changed retry distributions, fallback probabilities, rate basis and monthly demand. Review expected, P90 and the largest driver. If the positive P90 monthly delta exceeds the owned budget, the team can reduce the limit, improve failure handling or require explicit approval before merge.
FAQ
Retry and fallback questions.
How much do retries increase LLM cost?
The answer depends on retry frequency and what each attempt repeats. Measure attempts and route quantities rather than multiplying every call by the configured maximum.
Are fallbacks always more expensive?
No. A good capability-escalation design can keep routine work on an efficient route. Fallbacks become expensive when routing is inaccurate, failures are frequent or multiple routes run in parallel.
Should timeouts be retried?
Only with an idempotent design and a clear understanding of whether the original request may still be running. Otherwise, the retry can create duplicate cost and side effects.
What should I measure first?
Start with attempts per completed task, first-pass completion, fallback rate, failure class and cost per completed task. Those metrics reveal the largest economic problem.
Can Class1 detect retry changes?
Class1 scans code-visible retry and fallback signals, combines them with declared workload assumptions and estimates the monthly delta before merge. Try the browser example.
Next step
Turn reliability controls into a reviewable cost basis.
Map the normal, retry and fallback routes for one workflow. Add route probabilities and demand, then inspect which failure class drives the P90 tail. That gives engineering a concrete control target and finance an explainable budget case.