Operators should scale up when projected workload exceeds sustainable system-level throughput. A per-call latency miss calls for a separate review of workload and configuration rather than an automatic capacity increase. Operators should scale down only when a lower configuration can handle expected token demand while preserving required latency, reliability, and cost limits.
Measure throughput and latency separately
The cited sizing guidance treats system-level throughput, measured in tokens per minute (TPM), and per-call response time, also called latency, as separate sizing concepts. A satisfactory throughput result therefore does not establish that response times are satisfactory.
Historical prompt and completion token metrics provide the basis for estimating system-level throughput. Operators can compare that estimate with current and proposed configurations, but the available guidance does not establish universal TPM targets, latency targets, or scale-up and scale-down thresholds. Those values must come from each operator’s own requirements and validation.
Turn the measurements into a capacity decision
-
Set separate limits. Define the required throughput, acceptable per-call latency, and the measurement window for each. The evidence used here does not supply numerical targets.
-
Estimate the workload. Use historical prompt and completion token metrics to estimate system-level throughput, then compare expected demand with the capacity of the current and proposed configurations.
-
Evaluate throughput. If expected demand exceeds sustainable throughput at the current setting, proceed with a scale-up review. If throughput is adequate but latency is not, examine workload and configuration conditions separately before deciding whether more capacity is necessary.
-
Qualify a scale-down. Test the proposed lower setting against both expected token demand and the per-call latency requirement. A lower setting is suitable only if it satisfies both checks.
-
Check cost and reliability together. Use actual cost and reliability records to assess each configuration. A lower-cost configuration is not a successful scale-down if it pushes throughput, latency, or another operating requirement outside the operator’s limits.
What operators must still confirm
Before changing capacity, operators still need to confirm:
- The appropriate throughput and latency limits for the workload.
- Whether the historical token metrics represent the demand expected during the decision period.
- The actual cost and reliability effects recorded for each candidate configuration.
- Whether both measurements remain acceptable after the change.
- The conditions that would require the capacity change to be reversed.
No fee figure or universal operating threshold can be verified from the cited guidance, so any cost or reliability limits must remain operator-specific.