Production AI capacity planning should be driven by three observed signals: system throughput, per-call response times, and completion times. Quota alone is not enough: Microsoft warns that per-call latency variation can keep achieved Azure OpenAI throughput below quota.
What Each Signal Reveals
| Signal | What to examine | Capacity-planning role |
|---|---|---|
| System-level throughput | The throughput actually achieved, measured for Azure OpenAI applications in tokens per minute (TPM) | Shows how much processing capacity the workload is achieving rather than merely how much quota is available. |
| Per-call response time | How response times vary between calls | Helps explain why achieved throughput may remain below quota even when sufficient quota appears to be available. |
| Completion times | How long application operations take to finish | Adds an application-level view that helps distinguish request-level activity from completed work. |
These signals answer different questions. Throughput shows the rate achieved; response-time variation helps expose uneven call behavior; completion times show how those calls translate into work completed by the application.
How to Check Capacity
Start with application instrumentation. Microsoft recommends observing throughput, latency, and completion times, primarily through instrumented code.
Then assess the measurements together:
- Record system-level throughput in TPM for Azure OpenAI workloads.
- Examine per-call response-time variation rather than relying only on an average.
- Compare achieved throughput with the applicable quota.
- Check completion times to understand the application-level result of the observed request behavior.
- Repeat the assessment when workload conditions change, because a capacity decision based only on nominal quota does not address the risk identified by Microsoft.
A throughput gap should prompt investigation of latency variation before it is treated solely as a request for more capacity.
What You Must Still Confirm
The cited Microsoft guidance does not provide a universal throughput target, acceptable latency range, or completion-time threshold. Those values must be established against your own service requirements and verified with current provider documentation and account-specific conditions.
Operators should also confirm that instrumentation captures all relevant production behavior and that the observed workload is representative of the demand the system must support. For platforms other than Azure OpenAI, do not assume that the same TPM and quota relationship applies without checking the provider’s own guidance.
The practical rule is to plan from measured throughput, response-time variation, and completion times together—not from quota in isolation.