Hands-On: Measure First, Then Optimize

The previous two articles covered the concepts. Now it's time to get hands-on — but not with caching or database scaling just yet. Before that, we need to answer a question that's often skipped: how fast or slow is our task-tracker-api actually right now? Without this number, any "it's faster now" claim in later articles is just a feeling, not a fact.

Why Measuring First Is Non-Negotiable

Optimizing without data is like fixing a car blindfolded — you might get lucky, but you're more often wrong. Teams that rush to add a cache, upgrade a server, or split into microservices without knowing where the real bottleneck is often end up with a system that's more complex BUT NOT ANY FASTER — because the actual problem was somewhere else entirely, never touched.

The principle is simple: measure, then optimize, then measure again to validate. We'll repeat these three steps in every hands-on article from here on.

A Simple Load-Testing Script

We'll use one small Python script, benchmark/load_test.py, for the rest of this series. No need to install any external tool (locust, hey, etc.) — just httpx, already in requirements.txt.

async def run_load_test(url: str, total_requests: int, concurrency: int) -> None:
    latencies: list[float] = []
    errors: list[str] = []

    semaphore = asyncio.Semaphore(concurrency)
    async with httpx.AsyncClient(timeout=30.0) as client:

        async def bounded_worker() -> None:
            async with semaphore:
                await _worker(client, url, latencies, errors)

        start = time.perf_counter()
        await asyncio.gather(*(bounded_worker() for _ in range(total_requests)))
        total_duration = time.perf_counter() - start
    ...

It works simply: send total_requests requests to url, capped at a maximum of concurrency requests running at the same time (via asyncio.Semaphore) — this simulates many users hitting the API in parallel, instead of one at a time in sequence. The script records each request's latency, then at the end computes throughput (requests per second) and the latency distribution (average, median, p95, max).

Why p95, not just the average? An average is easy to hide problems behind — if 95 out of 100 requests finish in 50ms but the remaining 5 take 5 seconds, the average still looks "fine" even though 1 in 20 of your users is having a bad experience. P95 (95th percentile) tells you: "95% of requests finished in THIS long or faster" — a far more honest picture of the worst-case-but-still-common user experience.

Baseline: Measuring task-tracker-api As-Is

We start from where the FastAPI for Beginners series left off: one instance, SQLite, no cache. Run the app as usual:

cd fastapi-for-beginner/demo-app
uvicorn app.main:app --port 8000

Seed a bit of data first so the list endpoint has something to work with:

for i in $(seq 1 50); do
  curl -s -X POST http://localhost:8000/api/tasks \
    -H "Content-Type: application/json" \
    -d "{\"title\": \"Task seed $i\"}" > /dev/null
done

Then run a load test at a reasonable load — 200 requests, 20 of them running concurrently:

python benchmark/load_test.py --url http://localhost:8000/api/tasks --requests 200 --concurrency 20

The result, on the machine used to write this series:

Total requests    : 200
Concurrency       : 20
Success / Failed  : 200 / 0
Total duration    : 1.09s
Throughput        : 182.7 req/s
Average latency   : 104.9 ms
Median latency    : 100.9 ms
P95 latency        : 158.6 ms
Max latency        : 203.3 ms

Not bad — every request succeeded, and latency stays under 200ms even in the worst case (p95). Save this number — this is our baseline, the reference point for every optimization in the articles ahead.

Pushing Further: Finding the Breaking Point

The numbers above look healthy, but that's just one data point at one load level. The more important question: at what point does this system start to crack? Let's crank the load way up — 500 requests, 100 of them concurrent:

python benchmark/load_test.py --url http://localhost:8000/api/tasks --requests 500 --concurrency 100

The result, on the same machine:

Total requests    : 500
Concurrency       : 100
Success / Failed  : 3 / 497
Total duration    : 150.77s
Throughput        : 3.3 req/s
Average latency   : 29943.2 ms
Median latency    : 30129.1 ms

From 182.7 requests/second down to 3.3 requests per second. From ~100ms latency to ~30 SECONDS (and most of those ended in a timeout, not an actual completion). This isn't gradual degradation — this is total collapse. Something far deeper than "the server's a bit slow" is happening here.

Reading the Logs to Find Out What's Actually Happening

Check the server log running in another terminal, and there it is — the same error, repeated over and over:

sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached,
connection timed out, timeout 30.00

This is a textbook real-world example of the "a resource assumed to be shareable turns out to be locked" point covered in Part 1. Our database connection pool (default: 5 connections + 10 overflow = 15 max concurrent connections) runs out of slots the moment 100 requests arrive at once — request #16 onward has to queue up waiting for a free connection, and many of them time out before ever getting a turn.

This isn't a CPU problem. It isn't "the server isn't powerful enough" either. It's purely a resource configuration problem — the number of available database connections isn't enough to handle the concurrent load coming in. If we just blindly upgrade the server (vertical scaling) without knowing this, the problem will NOT go away — more CPU and RAM don't add slots to the connection pool. We'll fix this properly in Part 5.

Why This Example Matters

Notice the order we just followed: measure at a reasonable load (healthy), measure at an extreme load (collapse), THEN read the logs to diagnose the root cause. If we'd jumped straight to a "solution" the moment we heard the word "scalability" — say, immediately containerizing and spinning up 5 instances — this connection pool problem would still be there, just now showing up in 5 different places instead of one. Adding instances does NOT fix a problem whose root cause is a misconfigured resource on a single instance.

Summary