There is no universal capacity figure in the cited Microsoft guidance. A variable Azure OpenAI workload needs a deployment-specific capacity envelope: compare its peak token demand with assigned rate limits, measure per-call response time separately, and track the allowances remaining during operation. Microsoft identifies system throughput and per-call response time as two distinct sizing concepts.
How to check the required capacity
-
Map the workload’s variability. Record token demand across normal activity and the busiest periods you need to support. An average alone can conceal short bursts that consume the available request and token allowances.
-
Check throughput in tokens per minute. Microsoft describes system-level throughput in tokens per minute, or TPM. Compare that measure with the workload’s actual token demand rather than selecting capacity from an assumed usage level.
-
Verify the deployment’s rate limits. Azure OpenAI quota allows rate limits to be assigned to individual deployments. Confirm the actual limit for each relevant deployment; the cited material does not state what limit any unspecified deployment has.
-
Measure per-call response time. Throughput does not answer the latency question by itself. Test representative calls under the expected load and compare the results with the response-time requirement set for the application. The cited facts provide no universal response-time target.
-
Inspect live allowance information. Microsoft states that every API call includes rate-limit information in its HTTP response headers, including remaining request and token allowances. Use this information to check how closely operation is approaching the applicable limits.
What you must still confirm
Before treating the capacity as verified, confirm:
- the peak token demand and burst pattern that must be supported;
- the TPM rate limits assigned to each relevant deployment;
- per-call response times under representative demand;
- the response-time and allowance thresholds required by your own application; and
- how often the workload profile and capacity assumptions need to be reviewed as usage changes.
A defensible capacity decision therefore connects observed TPM demand, configured deployment limits, measured response times, and remaining allowances. If any of those remain unknown, the required capacity is still unverified.