Failure margins should shape capacity decisions by determining how much headroom sits above the estimated throughput, when additional capacity should be considered, and when the estimate must be recalculated. A defensible baseline uses historical prompt and completion token metrics rather than a favorable isolated run.
The cited technical guidance identifies those two metrics as necessary for estimating system-level throughput. It also states that throughput varies across model, version, and workload combinations, so an estimate should not be treated as fixed when those conditions change.
Build the baseline from measured inputs
The first capacity decision is the baseline: what throughput can the intended workload reasonably be expected to support?
Use historical prompt and completion token metrics to estimate system-level throughput for the planned model, version, and workload combination. This baseline is an estimate, not guaranteed future performance.
The cited guidance does not prescribe a failure-margin formula or universal margin value. The margin therefore needs to be established separately rather than presented as part of the source’s throughput methodology.
Tie the margin to the estimate’s scope
A failure margin is most useful when it has an explicit operational meaning. It may govern how much capacity remains uncommitted, how much degradation can be tolerated, or what conditions trigger additional provisioning. Those choices are planning decisions, not values supplied by the cited guidance.
| Decision | Available basis | Capacity treatment |
|---|---|---|
| What is the baseline throughput? | Historical prompt and completion token metrics | Use both metrics to estimate system-level throughput |
| Which conditions does the estimate cover? | Throughput varies by model, version, and workload combination | Do not automatically reuse the estimate for a different combination |
| How large should the failure margin be? | No universal value is stated in the cited material | Establish it separately using operational risk, observed variation, and cost constraints |
| When should the estimate be reviewed? | Model, version, and workload conditions affect throughput | Revalidate the estimate when the relevant conditions change |
This structure keeps the measured estimate separate from the buffer. It also makes clear whether a proposed capacity figure represents expected throughput or throughput plus failure headroom.
What operators must still confirm
The available guidance does not establish a margin percentage, measurement window, revalidation schedule, approval process, or cost threshold. Operators must confirm those items against their own operating requirements.
They should also document:
- the precise model, version, and workload combination covered by the estimate;
- how prompt and completion token histories are collected and reviewed;
- whether observed failures, retries, or degraded runs are included in the planning record;
- what event would justify adding capacity; and
- when a historical estimate becomes stale after workload or configuration changes.
Until those checks are complete, the throughput figure should be reported as a conditional estimate for a specific combination—not as a promise of future capacity.