Hands-On: Test Coverage & Its Limits

We now have a lot of tests — unit, integration, parametrized, with mocking for external dependencies. The natural next question: how much of our code is actually covered by tests? In this article we measure that with test coverage, while being honest about what that number CANNOT tell us — so we don't end up chasing the wrong number.

Measuring Coverage with pytest-cov

pip install pytest-cov
pytest --cov=app --cov-report=term-missing

The result, from an actual run on task-tracker-api:

Name                      Stmts   Miss  Cover   Missing
-------------------------------------------------------
app/__init__.py               0      0   100%
app/config.py                 16      0   100%
app/database.py               12      6    50%   15-18, 22-23
app/dependencies.py           16      1    94%   28
app/logging_config.py         17      1    94%   17
app/main.py                   29      2    93%   18-19
app/models.py                 20      0   100%
app/notifications.py           8      2    75%   23-24
app/rate_limit.py             14      1    93%   10
app/routers/auth.py           26      0   100%
app/routers/tasks.py          54      3    94%   23, 107-108
app/schemas.py                29      0   100%
app/security.py               21      0   100%
-------------------------------------------------------
TOTAL                        262     16    94%

94% overall — looks good. But let's dig into app/database.py, sitting at just 50%, because that's where the most important lesson lives.

Reading --cov-report=term-missing

The Missing column shows the LINE NUMBERS that were never executed by any test. For app/database.py, that's lines 15-18 and 22-23:

def create_db_and_tables() -> None:      # line 14
    try:                                    # line 15 — NOT covered
        SQLModel.metadata.create_all(engine)   # line 16 — NOT covered
    except (ProgrammingError, IntegrityError):  # line 17 — NOT covered
        pass                                    # line 18 — NOT covered


def get_session():                        # line 21
    with Session(engine) as session:        # line 22 — NOT covered
        yield session                          # line 23 — NOT covered

Why are these functions not covered, when they're clearly part of the code that runs in production? Because in the previous article, we OVERRIDE get_session with a test version via app.dependency_overrides — the REAL implementation of this function is NEVER actually called while tests run, because FastAPI always uses the override.

The Core Lesson: A High Number Can Hide Code That's Completely Untested

This is the single most important point in this article: 94% coverage does NOT mean 94% of the application's BEHAVIOR has been tested. Coverage only measures which lines of code were EXECUTED while tests ran — it doesn't care whether the result was actually CHECKED (assert) or not, and it has no idea about code that's deliberately "bypassed" via mocking or a dependency override, like the database.py case above.

The implication: if create_db_and_tables() has a bug (say, catching the wrong exception name), our coverage report will NEVER tell us — because that function is genuinely never run by any test. This isn't a flaw in our test suite (the dependency override itself is CORRECT — we really don't want tests touching a real Postgres), but it's a reminder that functions like this need a DIFFERENT kind of verification — for example, tried manually once during initial project setup (like we did in earlier series when verifying that docker compose up actually succeeded in creating the tables).

Why 100% Coverage Isn't the Goal

Chasing 100% coverage as a target is a common trap, for two reasons:

A healthier target: focus on BUSINESS LOGIC and critical paths (authentication, authorization, validation, important calculations) — exactly what we've been building since the earlier articles. High coverage in those areas is far more valuable than uniformly high coverage across every file regardless of how important it is.

A Quick Look: Mutation Testing — Going Further Than Coverage

If coverage only tells you "this line was executed," there's a stricter technique: mutation testing. Tools like mutmut for Python deliberately BREAK your code in small ways (turning == into !=, > into <, and so on) and then run the test suite — if the tests still PASS even though the code has been broken, that's a sign the test isn't really checking that behavior, even if coverage says 100%. This is beyond the scope of this beginner series, but it's good to know about as a next step (also mentioned in the roadmap of the closing article of this series) once your test suite matures enough.

Summary