Andrey Krisanov

LLM inference, AI infrastructure, and distributed systems

Understand How Services Run and Operate in Production

Software engineers should understand the runtime model, resource requirements, and capacity of the services they operate.

2 min read
#software-engineering #production #devops #platform-engineering

Over the past few years, I have interviewed dozens of software engineers who could not explain how the services they developed ran in production. Infrastructure or platform teams often manage deployment, but that division of responsibility does not remove the need to understand a service's runtime behavior.

A typical conversation goes like this:

Interviewer: How do you ship your service to production?

Candidate: We build Docker images and run containers.

Interviewer: What are the resource requirements for a container?

Candidate: I don't know the details. The DevOps team handles that.

Interviewer: Which application server does the Python service use?

Candidate: Application server? Do you mean WSGI?

Interviewer: Yes, the thing that handles web requests and runs your Python code.

Candidate: Let me check the repository. It uses Gunicorn.

Interviewer: Can you estimate how many requests the application can handle?

Candidate: No. We don't run load tests.

Interviewer: Can you make a rough estimate?

Candidate: No.

Interviewer: What happens at the process and thread level when the application server handles a request?

Candidate: I can't explain it.

Not knowing a configuration value from memory is reasonable. Being unable to explain the runtime model or how the team measures capacity is a problem. For synchronous and asynchronous services, the chosen Gunicorn server model affects concurrency, resource allocation, and throughput.

Platform and SRE teams can manage the infrastructure, but engineers still need to understand the processes and threads that execute their code, the CPU and memory those processes require, and the service's approximate capacity.