A recurring capacity-planning request for an internal LLM platform is:

We have selected a model. Do we have enough capacity, and how many GPUs would we need for N users?

The question is missing the information needed to answer it.

The model and its serving configuration determine the minimum resources required to start one replica. They don't tell us how many requests that replica can handle while meeting the product's latency and reliability requirements. That depends on prompt and response lengths, arrival rate, concurrency, cache reuse, calls per user task, traffic bursts, and competition with other workloads.

This guide describes how product and platform teams can replace capacity guesses with measurements. It is not a GPU calculator.

Three different capacity questions

Capacity planning combines three separate questions.

The first question is mostly about memory and execution topology. The second requires a performance experiment. The third includes production demand and reliability requirements.

This distinction also explains why an unallocated GPU isn't immediately available as serving capacity. Before the platform can handle traffic, it must place a replica, load the model, allocate its KV cache, warm the runtime, pass readiness checks, and attach it to the request path. Depending on the model size and configuration, this can take minutes. If the startup time is longer than the product can handle during a busy period, reactive autoscaling won't replace warm reserve.

Why user count is a weak capacity signal

Two requests to the same model can have very different costs:

• a short question with a short answer
• a summarization request with tens of thousands of input tokens
• an agent step containing tool schemas and a long conversation history
• a request that reuses a cached prefix
• a long-running generation that keeps KV-cache blocks active
• a user task that triggers ten LLM calls, several tool calls, retries, and context compaction

Modern inference engines such as vLLM combine requests through continuous batching. They schedule prompt processing (prefill) and token generation (decode) while managing a queue and KV cache. Prefill is usually more compute-intensive, while decode is often constrained by memory bandwidth and the number of active sequences. Changes in the input-length distribution can increase time to first token without changing the output throughput. Changes in concurrency or output length can make token streaming take longer and fill up the KV cache.

This is why the following two products may have the same number of users but require very different capacity:

• a chat assistant used a few times per day
• a coding agent that performs many LLM calls for every user task

Start with workload classes

Before collecting aggregate metrics, describe the operations the product performs:

Do not build the workload only from successful demos. Include long tasks, tool failures, retries, user cancellations, large contexts, and sessions that require compaction. These cases often determine tail latency and production capacity.

Model selection and capacity planning are related, but they shouldn't be combined into one step. External APIs can help compare model quality quickly, but their throughput and latency do not predict a self-hosted deployment. The final candidates should be benchmarked against the target runtime and hardware.

Define SLOs before measuring capacity

"Comfortable performance for 500 users" is not a measurable requirement. Define service-level objectives before testing.

For streaming LLM applications, the main latency signals are:

• time to first token (TTFT): how long the user waits before generation begins
• inter-token latency (ITL): the gaps between streamed tokens and the pauses a user can notice
• time per output token (TPOT): request-level decode time normalized by the number of generated tokens
• end-to-end request latency: how long the complete LLM call takes
• queue time: how long the request waits before execution
• task or agent-run duration: how long the user waits for the complete workflow

Reliability signals matter as well:

• error and timeout rate
• cancellation rate
• model availability
• SLO attainment percentage
• behavior during overload, rollout, and infrastructure failure

The product team should choose target values based on user experience and business requirements. The platform team can then test whether a model configuration meets them.

Raw throughput is not the right optimization target. A server may complete more requests per second while allowing TTFT or TPOT to become unacceptable. A better concept is goodput: the request rate the system can sustain while meeting the defined latency objectives. The DistServe paper uses this framing for LLM serving, and current vLLM benchmarking tools can report SLO-constrained goodput.

Observe the application and the inference platform

Proper capacity planning needs two complementary views.

Application view

For every LLM call, capture at least:

For an agentic application, one LLM call is too small a unit. The main unit should be the run: one execution of a user task. A run should record the number of LLM calls, tool calls, retries, and compactions. It should also record total token usage, maximum context length, models used, duration, final status, and a quality signal.

A tracing system like Langfuse can show LLM calls, retrieval, tools, and application logic as nested observations. Its data model also lets you group traces into sessions. Preserve the causal structure of the task regardless of the tracing system.

Inference-platform view

The inference layer usually exposes:

• running and waiting requests
• queue-time distribution
• prompt and generated token throughput
• TTFT, ITL, request-level TPOT, and end-to-end latency
• KV-cache usage, reuse, eviction, and preemption
• request failures and runtime errors
• GPU memory, utilization, throttling, and hardware health

Measure offered load at the gateway and completed throughput at the model server. During overload, completed throughput may remain constant while the queue and incoming demand continue to grow. Finished requests alone can make a saturated system appear stable.

For example, vLLM production metrics include queue depth, queue time, prompt and generation lengths, TTFT, inter-token latency, prefill and decode time, and KV-cache signals.

The application explains what workload was created. The inference platform explains how the serving system handled it. Without both views, it is difficult to distinguish between an overloaded model server, unexpectedly large contexts, excessive calls per agent run, slow tools, retries, or contention with another tenant.

Store high-cardinality identifiers in traces or logs, not Prometheus labels. Pseudonymize user and business identifiers.

Preserve workload shape, not only averages

Averages can hide the cases that cause queues and SLO violations. At minimum, analyze:

• requests per second and per minute
• concurrent LLM calls
• active user or agent runs
• input and output tokens over time
• p50, p95, and p99 input length
• p50, p95, and p99 output length
• p95 and p99 total context length
• LLM calls and tokens per run
• cache-hit and shared-prefix behavior
• retry and cancellation rates
• traffic by time of day
• peak-to-average ratio
• expected growth
• model mix

Percentiles are useful for dashboards, but independent percentiles cannot reconstruct a workload. Input length, output length, arrival time, scenario, and cache behavior are correlated. Retain sanitized request metadata or a replayable trace so the benchmark preserves these relationships.

Arrival patterns matter too. A closed-loop test, in which each virtual user sends a new request only after the previous one completes, reduces offered load when the server slows down. It can hide overload. For online serving, prefer an open-loop test that sends requests according to a controlled arrival rate and burstiness. vLLM's serving benchmark supports request-rate, burstiness, warm-up, detailed per-request output, and timed-trace replay. MLPerf's server scenario follows the same general principle by generating independent request arrivals and measuring the maximum rate that stays within a tail-latency constraint.

A defensible benchmarking process

Use the following loop for a new model or a material workload change.

flowchart LR
    accTitle: The LLM capacity-planning loop
    accDescr: Product scenarios produce application traces. Sanitized traces are replayed against a pinned model configuration. The SLO boundary determines the initial replica count, which is then validated in production and updated with new telemetry.

    Scenarios["Product scenarios"]
    Telemetry["Application telemetry"]
    Replay["Sanitized workload replay"]
    Benchmark["Single-replica benchmark"]
    Boundary["SLO-constrained capacity"]
    Plan["Replica and reserve plan"]
    Production["Production validation"]

    Scenarios --> Telemetry
    Telemetry --> Replay
    Replay --> Benchmark
    Benchmark --> Boundary
    Boundary --> Plan
    Plan --> Production
    Production --> Telemetry

Figure 1. Capacity planning is a feedback loop, not a one-time calculation.

1. Pin the complete serving configuration

Record:

• model checkpoint and revision
• tokenizer and revision
• precision or quantization
• maximum context length
• tensor, pipeline, expert, and data parallelism
• chat template and reasoning settings
• speculative decoding configuration
• inference runtime and version
• scheduler, batching, and KV-cache settings
• GPU type, count, interconnect, CPU, memory, and software stack

Changing any of these can affect memory usage, TTFT, TPOT, throughput, or quality. Rebenchmark after a material change.

2. Use the target hardware

Run a warm replica on the same hardware and topology as production. A benchmark on a different GPU, interconnect, runtime version, or parallelism plan can support comparisons, but not final sizing.

3. Replay representative work

Use real, sanitized request metadata when available. Preserve the mix of workload classes, input and output lengths, arrival times, generation settings, cache reuse, and retries. Send warm-up requests before collecting results.

Tools such as vLLM Bench and NVIDIA AIPerf expose the latency and throughput metrics needed for this experiment.

4. Increase offered load

Increase the request rate incrementally. GPU utilization is not the stopping criterion. Continue until the system violates one or more constraints:

• TTFT or TPOT crosses its target
• queue time grows without recovering
• SLO attainment falls below the target
• errors, timeouts, or cancellations rise
• KV-cache pressure causes preemption or instability

The highest stable rate before these failures is the sustainable capacity of that replica for that workload and SLO.

5. Repeat and record variance

Run each important point more than once. Record the benchmark duration, warm-up plan, random seed or trace, and complete configuration. Short tests may miss burst behavior, cache churn, thermal throttling, and long-tail requests.

6. Validate the full deployment

A single-replica result is an input to the plan. Run the test again with the planned number of replicas and the actual load-balancing strategy. Scaling can be non-linear due to cache affinity, routing, networking, distributed execution, and uneven request placement.

Test failure and maintenance scenarios: replica loss, GPU node loss, rollout, rollback, and reduced capacity in another failure domain.

Turning the benchmark into a capacity plan

For one stable workload mix, calculate:

sustainable_load_per_replica =
    highest offered load at which the required SLO attainment is preserved

base_replicas =
    ceil(projected_peak_offered_load / sustainable_load_per_replica)

reserve_replicas =
    capacity needed for failures, rollouts, and near-term growth

planned_replicas =
    base_replicas + reserve_replicas

This is shorthand, not a universal scalar formula. Request rate, concurrency, input and output lengths, cache reuse, and model mix must match the benchmark. If the product has very different workload classes, you should benchmark the production mix or establish separate capacity envelopes and admission rules.

The reserve should cover the actual failure topology. Losing one pod can remove a complete multi-GPU replica. Losing one node can remove several replicas or an entire model shard. During a rollout, old and new versions may need to coexist. The required reserve depends on placement, model topology, recovery time, and the product's permitted degradation mode.

Capacity on a shared platform

A multi-tenant inference platform introduces another variable: other workloads.

Products using the same replicas compete for:

• scheduler and gateway queues
• batching slots
• KV cache
• GPU compute and memory bandwidth
• generated-token throughput

Without per-tenant attribution, resource consumption cannot be measured reliably. Every project should use a stable project identity and correlation IDs from the gateway to the inference server.

There are three common operating models:

Maximum utilization and independent tenant SLOs conflict. Shared platforms need admission control, rate and concurrency limits, tenant quotas, priorities, and an overload policy. Otherwise, one product can consume the queue and KV cache needed by every other tenant.

Define the degradation policy explicitly. Options include rejecting excess requests with a retryable error, restricting maximum context or output length, reducing agent parallelism, routing to a smaller model, pausing batch traffic, or accepting a lower SLO for a defined workload class.

A practical sizing request

Product teams can use this template when requesting capacity from the platform team.

1. Product and scenarios

• product or service name
• owner and technical contact
• environments
• workload classes
• interactive, asynchronous, batch, or agentic execution
• expected launch date and growth stages

2. Model candidates

• exact model identifiers and revisions
• tokenizer revision
• quality-evaluation results
• context-length requirement
• reasoning, multimodal, tool-calling, or structured-output requirements
• acceptable fallback models

3. Workload profile

• expected active users or jobs
• requests or agent runs per time window
• LLM calls per run
• input tokens p50, p95, and p99
• output tokens p50, p95, and p99
• context length p50, p95, and p99
• concurrent LLM calls
• peak-to-average ratio and burst duration
• expected growth over the next 6-12 months
• representative trace or dataset

4. SLO

• TTFT target and attainment percentage
• ITL target and attainment percentage
• request-level TPOT target and attainment percentage
• end-to-end LLM latency
• agent-run or job duration
• queue-time limit
• error and timeout rate
• overload behavior

5. Reliability and isolation

• shared, dedicated, or reserved capacity
• required availability
• acceptable capacity after a replica, GPU, node, or failure-domain loss
• rollout and rollback requirements
• maximum recovery time
• allowed degradation modes

6. Observability and data handling

• tracing system
• token-usage collection
• correlation IDs
• prompt and application versioning
• pseudonymization
• retention and access rules
• confirmation that sensitive prompts, outputs, tool arguments, source code, and documents are not recorded by default

If important data is missing, the platform team can only give a preliminary estimate with explicit assumptions and a wide range.

Common sizing mistakes

«We will have 400 users.»

This describes the audience, not the load. Specify active-user behavior, scenario frequency, LLM calls per task, token distributions, concurrency, and peaks.

«The model fits on eight GPUs, so one node is enough.»

Eight GPUs might be the minimum for one replica. That does not indicate how much SLO-compliant traffic the replica can handle or whether the product can survive a node failure.

«We tested with 30 users and multiplied the result.»

LLM inference is non-linear because of batching, queueing, cache pressure, and the distribution of request lengths. Control the request rate and replay the workload instead.

«Average latency is fine.»

The average hides the long tail. Capacity is usually limited by p95 or p99 latency, queue time, errors, and SLO attainment. For a low-volume workload, use a measurement window long enough to make tail percentiles meaningful.

«The external API was fast.»

External APIs can help evaluate model quality. Their hardware, batching, routing, and load are unknown, so their performance does not predict a local deployment.

«GPU utilization is below 100%, so capacity is available.»

GPU utilization alone does not show queueing, KV-cache pressure, memory headroom, TTFT, TPOT, or the impact of another long request.

«Our tracing tool will tell us how many GPUs to buy.»

Application tracing explains the workload. Pair it with benchmarks and inference-server metrics to determine GPU capacity.

«We will log everything and investigate later.»

Capacity planning rarely requires storing raw prompts, responses, documents, source code, tool arguments, or terminal output. Excess telemetry can create security and privacy risks without improving the calculation.

Further reading