Observability: Knowing When to Scale

Throughout this series, we found out about problems in a fairly manual way: running load_test.py, reading its output, checking the server log in another terminal. That's great for learning and controlled experiments, but in a real production system, we can't just sit around running manual load tests all the time. We need a system that continuously tells us the application's condition — this is observability.

The Three Pillars of Observability (the Ones That Matter Here)

Observability is usually discussed in terms of three pillars: logs (records of events), metrics (measured numbers over time), and traces (the path of a single request across many services). For a task-tracker-api this simple in scale, we'll focus on the first two — logs and metrics — since traces only become truly necessary once a single request crosses many different services (covered as a roadmap item in Part 9).

Structured Logging — Logs a Machine Can Process

A plain log (print("User logged in")) is pleasant for a human to read but hard for a program to process. Structured logging writes logs in a parseable format (we use JSON), so they can be searched, filtered, and analyzed by tooling.

class JsonFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        payload = {
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
        }
        extra_fields = getattr(record, "extra_fields", None)
        if extra_fields:
            payload.update(extra_fields)
        return json.dumps(payload)

Every request that finishes processing produces one log line like this:

{"timestamp": "2026-07-10T13:41:56.695851+00:00", "level": "INFO", "logger": "app", "message": "request_completed", "request_id": "eec43850-6da2-4cd5-91a1-300a105438c8", "method": "POST", "path": "/api/tasks", "status_code": 201, "duration_ms": 72.09, "instance_id": "api-1"}

Compare that to a plain-text log like "POST /api/tasks - 201 Created" — the JSON version above can be queried directly: "show every request with duration_ms > 500," or "count how many status_code >= 500 responses came from instance_id: api-2 in the last 5 minutes." This is what makes it useful once logs from many instances get collected in one place (a log aggregator like Loki, the ELK stack, or a cloud logging service).

Also notice request_id — a unique ID per request. If a user reports an error, we can ask for their request_id (shown in a response header, or in the error message the user sees) and jump straight to the exact relevant log line, without having to dig through thousands of other log lines happening at the same time.

Metrics — Numbers Summarized Over Time

If logs are a detailed record of every event, metrics are a numeric summary over time — request counts, latency distribution, error rate. We use prometheus-client, with two kinds of metrics:

REQUEST_COUNT = Counter(
    "http_requests_total",
    "Total HTTP requests",
    ["method", "path", "status_code", "instance"],
)
REQUEST_LATENCY = Histogram(
    "http_request_duration_seconds",
    "HTTP request latency in seconds",
    ["method", "path", "instance"],
)

Our middleware records both metrics on EVERY request, then exposes them via a /metrics endpoint:

@app.middleware("http")
async def observability_middleware(request: Request, call_next):
    start = time.perf_counter()
    response = await call_next(request)
    duration = time.perf_counter() - start

    route = request.scope.get("route")
    path_label = route.path if route is not None else request.url.path

    REQUEST_COUNT.labels(request.method, path_label, response.status_code, settings.instance_id).inc()
    REQUEST_LATENCY.labels(request.method, path_label, settings.instance_id).observe(duration)
    ...


@app.get("/metrics")
def metrics():
    return Response(generate_latest(), media_type=CONTENT_TYPE_LATEST)

curl localhost:8000/metrics

http_requests_total{instance="api-1",method="GET",path="/health",status_code="200"} 1.0
http_requests_total{instance="api-1",method="POST",path="/api/tasks",status_code="201"} 1.0
http_requests_total{instance="api-1",method="GET",path="/api/tasks/{task_id}",status_code="404"} 1.0
http_request_duration_seconds_bucket{instance="api-1",le="0.025",method="GET",path="/health"} 1.0
http_request_duration_seconds_bucket{instance="api-1",le="0.05",method="GET",path="/health"} 1.0
...
http_request_duration_seconds_sum{instance="api-1",method="GET",path="/health"} 0.0138

This format (the Prometheus exposition format) is designed to be scraped (pulled periodically) by a Prometheus server, which stores its history and enables visualization via Grafana. We won't run an actual Prometheus in this series to stay focused, but our /metrics endpoint is already in a format ready to plug into those tools whenever needed.

The Cardinality Trap — Why We Use a Route Template, Not the Raw Path

Notice this line in the middleware:

route = request.scope.get("route")
path_label = route.path if route is not None else request.url.path

This is a small detail that's easy to get wrong, with big consequences. If we used the raw request.url.path as a metric label, then /api/tasks/1, /api/tasks/2, /api/tasks/3, and so on would each be treated as a DIFFERENT label by Prometheus. With thousands of tasks, that's thousands of distinct time series for just one endpoint — this is called cardinality explosion, one of the most common causes of a metrics server (Prometheus) running out of memory or becoming very slow in real production.

The fix: use route.path, which gives the TEMPLATE (/api/tasks/{task_id}), not the actual value. Every request to any task counts as the SAME single label — we can still tell which endpoint is slow, without exploding the number of time series.

Connecting Metrics to Scaling Decisions

Now let's close the loop back to Part 1: "when do we actually need scaling?" With the metrics we've built, the answer is no longer a guess. A few concrete signals from /metrics that can drive real decisions:

This is why this entire series started with "measure first" (Part 3) — the baseline numbers we collected along the way aren't just for that article, they become the real reference point for judging production metrics: if the p95 on a dashboard is approaching the number that made the system "crack" during a load test, that's a signal to act BEFORE users start complaining — not after.

Summary