We've hit the ceiling on optimizing a single process: caching (Part 4), the database (Part 5), and concurrency within a single process (Part 6). All of that matters, but there's a hard limit that no amount of tuning can get past — a single process can only use so many CPU cores and so many threads. Now we step outside that limit entirely: running several instances of task-tracker-api at once, behind a load balancer.
Containerizing the App With Docker
First step: package the app into a Docker image so it can be run identically, over and over.
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app ./app
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
Nothing exotic here — install dependencies, copy the code, run uvicorn. What matters: this image contains NO state at all (no database file inside it) — exactly the stateless principle covered in Part 2. All state lives outside the container: Postgres and Redis.
Docker Compose — Running 3 Instances + a Load Balancer at Once
services:
postgres:
image: postgres:16-alpine
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: tasks
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
redis:
image: redis:7-alpine
healthcheck:
test: ["CMD", "redis-cli", "ping"]
api1: &api
build: .
environment:
DATABASE_URL: postgresql+psycopg2://postgres:postgres@postgres:5432/tasks
REDIS_URL: redis://redis:6379/0
INSTANCE_ID: api-1
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
api2:
<<: *api
environment:
DATABASE_URL: postgresql+psycopg2://postgres:postgres@postgres:5432/tasks
REDIS_URL: redis://redis:6379/0
INSTANCE_ID: api-2
api3:
<<: *api
environment:
DATABASE_URL: postgresql+psycopg2://postgres:postgres@postgres:5432/tasks
REDIS_URL: redis://redis:6379/0
INSTANCE_ID: api-3
nginx:
image: nginx:1.27-alpine
ports:
- "8080:80"
volumes:
- ./docker/nginx.conf:/etc/nginx/nginx.conf:ro
depends_on: [api1, api2, api3]
A few important points:
&api and <<: *api — a YAML anchor feature, so we don't rewrite the same config three times. api2 and api3 inherit api1's configuration, only differing in INSTANCE_ID.DATABASE_URL and REDIS_URL — this is what makes them genuinely count as one system: whichever instance handles a given request, they all see the same data, because the cache and database are shared (exactly the concept from Part 2).INSTANCE_ID — purely for observability, so we can see which instance answered a request (used in the response header and metrics, covered further in Part 8).nginx has port 8080 open. All traffic MUST go through the load balancer; there's no shortcut straight to any particular instance.Nginx as a Load Balancer
http {
upstream task_tracker_api {
server api1:8000;
server api2:8000;
server api3:8000;
}
server {
listen 80;
location / {
proxy_pass http://task_tracker_api;
}
}
}
The upstream block registers the three instances as one group. Nginx's default: round-robin — the first request to api1, the second to api2, the third to api3, then it repeats. No extra configuration is needed for this most basic strategy.
Running It and Proving Round-Robin
docker compose up --build -d
The /health endpoint in task-tracker-api deliberately returns instance_id, so we can see who answered:
for i in 1 2 3 4 5 6; do curl -s http://localhost:8080/health; echo; done
{"status":"ok","instance_id":"api-1"}
{"status":"ok","instance_id":"api-2"}
{"status":"ok","instance_id":"api-3"}
{"status":"ok","instance_id":"api-1"}
{"status":"ok","instance_id":"api-2"}
{"status":"ok","instance_id":"api-3"}
Exactly as promised — perfectly round-robin. Now the more important part: prove the data stays consistent even when answered by different instances:
curl -X POST localhost:8080/api/tasks -H "Content-Type: application/json" -d '{"title":"Task from the load balancer"}'
curl -X PATCH localhost:8080/api/tasks/1 -H "Content-Type: application/json" -d '{"done":true}'
for i in 1 2 3; do curl -s localhost:8080/api/tasks/1; echo; done
The last three requests — which could each land on a different instance — all show "done":true. This proves the cache invalidation we built in Part 4 works ACROSS instances, because Redis is shared — the moment one instance updates a task and deletes its cache entry, ALL instances immediately see the fresh data, not just the instance that made the update.
Measuring the Impact — and a Lesson About the Measuring Tool Itself
Now for the numbers. Our first experiment runs as usual: load_test.py with concurrency 100, comparing one instance (direct access, bypassing the load balancer) against three instances (through Nginx). At a glance, the result is surprising — almost no difference at all, both showing a p95 of around 5 seconds.
Before concluding "horizontal scaling doesn't help," remember the principle from Part 3: read the data, don't just trust the final number. We check docker stats while the load test is running:
CONTAINER CPU %
bench-client 101.97% ← the load test generator itself!
demo-app-api1-1 4.75%
demo-app-api3-1 4.01%
demo-app-api2-1 4.22%
demo-app-postgres-1 1.08%
demo-app-redis-1 1.84%
Our own load-testing script is the bottleneck — its CPU is pegged at 100%, while EVERY server instance (and the database) is still relaxed under 5%. Because Python (and asyncio in a single process) is limited to one CPU core for CPU-bound work, our load generator can't send requests as fast as the server can actually handle them — so both one instance and three instances look equally "maxed out," except it's the client that's maxed out, not the server.
This is an important, easy-to-fall-into lesson: the measuring tool itself can become the bottleneck, and if you don't check for it, you can wrongly conclude something "doesn't help" when it was never actually properly tested.
The fix: run the load generator from MULTIPLE parallel processes, so the CPU load doesn't pile up on a single process. With three parallel client processes (each at concurrency 34, ~100 concurrent total combined):
| One instance | Three instances (Nginx) | |
|---|---|---|
| Combined throughput | ~261 req/s | ~365 req/s |
| P95 latency (averaged across all three clients) | ~650 ms | ~525 ms |