Production traces validate a capacity model by comparing its assumptions with the request-level latency and achieved throughput observed in production, organized by AI model and input/output token mix. Quota alone does not establish achieved throughput: per-call latency varies, so observed throughput may be lower than the quota suggests. Provisioned throughput also depends on the model and the input/output token mix in a given minute.
The practical check is therefore not whether a configured limit appears sufficient. It is whether the capacity model still predicts production behavior for the workload it is meant to cover.
How to check it
-
Turn assumptions into comparable cases. Define separate cases by AI model and input/output token mix. Quota and provisioned-throughput settings should remain assumptions until production traces show what was achieved.
-
Normalize the trace records. Capture or verify the model, input and output token counts, call timing, and outcome where available. Establish consistent definitions for a completed call, latency, and achieved throughput. A mismatch caused by inconsistent measurement is not evidence of a capacity-model error.
-
Compare like with like. For each model and token-mix case, compare modeled latency and achieved throughput with the corresponding production observations. A quota-only comparison can miss the effect of per-call latency variation. Changing the token mix during the comparison can also test a different workload from the one assumed by the capacity model.
-
Explain material mismatches. If observed throughput is lower, check whether the trace window contains a different token mix or slower calls than expected. If latency is higher within the same model and token-mix case, revisit the latency assumption. A single trace window should not be treated as a universal correction factor.
-
Check repeatability. Repeat the comparison across trace windows that represent the intended workload. The model gains support where its relationships remain consistent across relevant cases; it needs revision where the observed cases do not follow its assumptions.
What still needs confirmation
Production traces can test the relationships in a capacity model, but they do not supply missing planning targets. Before relying on the comparison, operators still need to confirm:
- Coverage: The traces represent the models and token mixes the plan is intended to serve. Agreement outside that coverage does not establish performance for unobserved workloads.
- Measurement: Timestamps, token counts, call boundaries, and throughput calculations are consistent across the capacity model and production records.
- Trace quality: The records are complete enough for the chosen comparison, with missing, failed, or retried calls handled according to a documented method.
- Acceptance criteria: The source points do not provide a universal pass/fail threshold, exact capacity target, or adjustment factor. The appropriate tolerance must be established separately rather than assumed.
- Currency: The deployment settings, model assumptions, and workload definition used in the capacity model still match the period represented by the traces.
In short, production traces are useful when quota and provisioned-throughput assumptions are converted into observed latency and achieved throughput for explicit model and token-mix cases. They show where the capacity model agrees with production and where further confirmation or revision is required.