In the previous article, we covered why applications stop scaling. Now it's time to break down what components actually make up a scalable system. Think of it like getting familiar with the puzzle pieces before we start scaling task-tracker-api in practice — so that once we get to Part 3 and beyond, the terminology that comes up won't feel unfamiliar.
The Big Picture: From One Instance to Many Instances
A simple system that hasn't scaled yet usually looks like this — one client talks directly to one application instance, which talks directly to one database:
Client ──► Application (1 instance) ──► Database (1 instance)
A system that has scaled usually looks like this — a few new components enter the picture:
┌──► Application (instance 1) ──┐
Client ──► Load Balancer ──► Application (instance 2) ──┼──► Cache (Redis)
└──► Application (instance 3) ──┘ │
▼
Database
Every new box in this diagram has a specific role. Let's go through them one by one.
Load Balancer — Spreading Traffic Across Many Instances
A load balancer is the component that receives all incoming requests, then distributes them to one of several available application instances. A few common distribution strategies:
/health endpoint), and automatically stops sending traffic to any instance that isn't responding.A load balancer's benefit isn't just traffic distribution — it's also what removes the single point of failure at the application layer. If one instance dies, the load balancer automatically redirects traffic to the instances that are still alive.
Stateless vs. Stateful — Why This Is Crucial for Horizontal Scaling
This is a concept that often trips up beginners. Stateless means an application instance doesn't hold any user-specific data in memory between requests — every request is processed independently, and all the data it needs (user identity, application data) is fetched from outside that instance itself (a database, a centralized cache, or re-sent by the client, e.g. via a token).
Why does this matter for horizontal scaling? Because the load balancer might send a user's first request to instance 1, then that same user's next request to instance 2 — round-robin doesn't guarantee one user always lands on the same instance. If an application instance stores important state in local memory (say, a login session stored in a plain Python variable), the second request that lands on a different instance won't find that state — the user suddenly gets "logged out" at random.
The task-tracker-api we've been building since the start of the FastAPI series is already correctly stateless: every endpoint fetches data from the database (and, later, the cache) on every request, instead of storing it in a Python variable that lives for the duration of the process. This is why, in Part 7, we'll be able to run 3 instances at once without changing the application's logic at all.
Caching Layer — Absorbing Load Before It Reaches the Database
A cache is temporary storage that's much faster to access than the original data source (usually a database), used to store results that get requested repeatedly. Instead of every request querying the database, a request whose data has already been fetched before can simply be answered from the cache.
Redis is a popular choice for this — a very fast in-memory data store that (importantly, for horizontal scaling) can be shared across all application instances. This is the key difference from a local cache in each instance's own memory: if instance 1 caches something in its own memory, instances 2 and 3 have no idea — they still have to query the database again. With a centralized cache like Redis, the moment one instance populates the cache, ALL instances can immediately benefit from it. We get hands-on with this in Part 4.
Connection Pooling — Managing a Limited Number of Database Connections
Every connection to a database isn't free — there's a cost to opening a new one (TCP handshake, authentication), and a database has a limit on how many simultaneous connections it can handle. If every request opens a new connection and closes it when done, at high traffic that becomes significant overhead and easily hits the database's connection limit.
A connection pool solves this by preparing a number of connections up front, then "lending" them to requests that need one, and returning them to the pool when done — rather than closing the connection. This is exactly what we've already been using implicitly via SQLModel/SQLAlchemy in the FastAPI series (Depends(get_session)), and we'll explicitly tune its size in Part 5, because a pool sized wrong — too small or too large — can just as easily become a new bottleneck once many application instances are sharing one database.
Putting It All Together
These four concepts — load balancer, stateless applications, shared cache, and connection pooling — are the same puzzle pieces we'll assemble one by one starting in Part 4. The order is deliberately arranged from the cheapest/simplest (caching, database tuning) to the most complex (horizontal scaling with Docker), because that's also the most sensible order to try things in the real world — don't jump straight to the most complicated architecture when the problem might actually be solved by a missing database index.