At the end of Part 5, we found something surprising: enlarging the database connection pool barely changed performance under high load. There's another bottleneck limiting concurrency, happening BEFORE our request ever touches the database at all. In this article we dig into what that is, and why async/await is the answer.
The Thread Pool — A Hidden Bottleneck Behind def Endpoints
Recall the concept from Part 2 of the FastAPI for Beginners series: a regular endpoint (def) is run by FastAPI in a separate thread pool, so blocking code (like a synchronous database query) doesn't freeze the entire event loop. That explanation is correct, but there's an important detail we skipped: this thread pool has limited capacity.
import asyncio
from anyio import to_thread
async def main():
limiter = to_thread.current_default_thread_limiter()
print(limiter.total_tokens)
asyncio.run(main())
40
The default is 40 concurrent threads, shared by EVERY regular def endpoint across the entire application. The 41st concurrent request has to WAIT for one of those 40 threads to finish before it gets a turn — exactly like the queue at the database connection pool, but one layer earlier, and it often goes unnoticed because it doesn't show up as an explicit error like QueuePool does — it just feels "slow," with no clear error message.
Proving It With an Isolated Demo
To keep this clean of database, cache, or other influences, we use examples/blocking_vs_async_demo.py — two endpoints simulating a 2-second I/O call, one blocking, one async:
@app.get("/slow-blocking")
def slow_blocking():
time.sleep(SIMULATED_IO_DELAY_SECONDS) # blocking — runs in the thread pool
return {"ok": True}
@app.get("/slow-async")
async def slow_async():
await asyncio.sleep(SIMULATED_IO_DELAY_SECONDS) # async — runs on the event loop, NO thread pool
return {"ok": True}
Run it, then send 80 concurrent requests (double the thread pool's limit) to each:
uvicorn examples.blocking_vs_async_demo:app --port 8001
python benchmark/load_test.py --url http://localhost:8001/slow-blocking --requests 80 --concurrency 80
python benchmark/load_test.py --url http://localhost:8001/slow-async --requests 80 --concurrency 80
The result:
/slow-blocking (def) | /slow-async (async def) | |
|---|---|---|
| Total duration | 4.31s | 2.37s |
| Median latency | 3138 ms | 2173 ms |
| P95 latency | 4257 ms | 2307 ms |
The async endpoint never touches the thread pool at all. All 80 "I/O calls" (asyncio.sleep) are scheduled directly by the event loop, which can handle thousands of concurrently waiting I/O operations — because all it's doing is "waiting efficiently," not allocating an expensive operating-system resource (a thread) for every single request.
The "Band-Aid": Enlarging the Thread Pool
If a full migration to async isn't feasible yet (say, because the database driver you're using doesn't support async), there's a shortcut: explicitly enlarge the thread pool's capacity.
from anyio import to_thread
@asynccontextmanager
async def lifespan(app: FastAPI):
to_thread.current_default_thread_limiter().total_tokens = 200
yield
Try re-running with ENLARGE_THREAD_POOL=true uvicorn examples.blocking_vs_async_demo:app --port 8002, then load-test /slow-blocking again at the same load:
Total duration : 2.48s
Median latency : 2275 ms
P95 latency : 2419 ms
Nearly identical to the async version. But this isn't a long-term solution — every OS thread is expensive (a few MB of memory each, context-switching overhead), while a single event loop can handle thousands of coroutines with far less overhead. Cranking this number too high just relocates the problem elsewhere (running out of memory instead of running out of "queue slots"). Treat this as first aid, not a substitute for an actual migration to async.
Why Our task-tracker-api Isn't (Fully) Async
Notice that the endpoints in task-tracker-api itself still use plain def, not async def — deliberately, and not an oversight. If you mark an endpoint async def but call blocking code inside it (like psycopg2, the synchronous Postgres driver we're using), that's FAR WORSE than sticking with plain def — because that blocking code freezes the ONE AND ONLY event loop handling EVERY request in the entire application, not just one thread out of the 40 available. This is exactly the warning given in Part 6 of the FastAPI for Beginners series, and now we understand the concrete consequences firsthand.
To safely use async def for real, EVERY I/O operation inside that endpoint — database queries, external API calls, file reads — needs its own async version (e.g. asyncpg for Postgres, httpx.AsyncClient for HTTP calls). A migration like this requires swapping database drivers and re-testing the entire application — a significant investment, so it makes sense to exhaust the cheaper fixes first (caching, indexing, correctly sized connection pools, as we've already done) before jumping to a full async migration.
Summary
def endpoints run in a thread pool with limited capacity (default 40 in anyio/FastAPI) — a hidden bottleneck limiting concurrency BEFORE a request ever touches the database.async def endpoint (whose I/O is also async) doesn't use the thread pool at all — it's handled directly by the event loop.async def if it contains blocking code — that freezes the entire event loop, far worse than staying with plain def.