Test Suite Gaps as a Deployment Risk Signal

A green pipeline tells you your tests passed. It does not tell you your product works. Those are two different claims, and the gap between them is where deployment risk lives, quiet and unmeasured, until the wrong combination of retry and latency raises it in production.

Take the classic Friday-afternoon release. Unit tests pass. Integration checks pass. The smoke suite runs clean. Everyone signs off, the deploy goes out, and by evening the tickets start rolling in: checkout is failing, but only for a subset of users, and only under a specific condition nobody tested for, a retry firing while a downstream payment service is running slow. Nothing in the suite caught it, because nothing in the suite looked anything like that scenario. That's not bad luck. That's a structural problem with how most teams still think about testing: as an administrative box to check, pass counts and signoffs, rather than an engineering feedback loop built to catch the ways modern systems actually break, at service boundaries, under concurrency, during retries, with test data that went stale months ago.

This piece works through that gap: what it is, how to measure it, and how to read it as a risk signal before code ships rather than as a postmortem after it breaks.

What a test gap actually is, and why coverage alone does not close it

Start with a distinction that gets flattened constantly: passing tests, covered code, and tested behavior are three different things. A test suite can hit 100% of your lines and still leave a documented behavior completely unverified. It's a measured finding with three component gap types.

Coverage gap. Code that no test touches. This is the one most teams already track, and it's the easiest to see on a dashboard.

Oracle gap. Code that a test does reach, but where no assertion actually checks whether the output was correct. The test runs, nothing crashes, the line lights up green in the coverage report, but nothing verified the behavior. Researchers studying this call it the "covered but not checked" problem, and a 2025 paper from Maton, Kapfhammer, and McMinn (presented at ESEM) framed the whole issue under the phrase "beyond coverage and kill scores," pointing out that neither metric on its own proves a test suite is actually checking anything meaningful.

Behavioral gap. A documented, expected behavior, something written into an API contract or a method's own documentation, that no test validates, regardless of how much coverage or how many assertions surround it.

That third category is the one that should worry teams most, because it's invisible to the metrics most CI pipelines already trust. Research out of UBC (Paul and Holmes, 2026) analyzed 8,922 documented methods across ten widely used open-source Java libraries, extracting 20,729 distinct expected behaviors with 93.1% precision. The finding: 17.5% of those behaviors were entirely untested, in mature, actively maintained, developer-curated codebases. And that's a floor, not a ceiling.

It gets worse when you look at automated test generation, the tools teams increasingly lean on to close coverage gaps quickly. EvoSuite, one of the more established automated generators, still left at least 20.6% of expected behaviors unvalidated. ASTER left at least 27.1%. Automated generation raises coverage numbers. It doesn't close the behavioral gap, and treating the two as the same thing is where teams get burned.

The study found that most untested behaviors showed up in methods that already had high line coverage, and more than half persisted even in methods with high mutation kill scores (a metric specifically designed to measure whether tests would catch injected bugs). In other words, the code that looked the safest by every structural measure teams already trust was often hiding the least-tested behavior. That's not a rounding error. That's the metric actively pointing you away from the risk.

So if coverage percentages and kill scores can't reliably flag these gaps, what can? That's the question the next body of research actually answers.

The research linking specific gap types to defect probability

A 2025 study in IEEE Transactions on Software Engineering took a more industrial angle. Instead of scanning open-source libraries, the researchers went into real industrial companies and analyzed 31 historical test gap reviews across eight production systems. They backed that up with semi-structured interviews with six quality engineers who'd actually authored those reviews, people who'd sat with the gaps and made judgment calls about which ones mattered.

What came out of it: code criticality and complexity metrics hold up as real predictors of which gaps carry the highest defect probability. And maybe more useful than the finding itself is the framing behind it. The motivation named directly in the paper is that exhaustive testing has become impossible in rapidly changing systems. Triage by risk isn't a nice-to-have anymore, it's the only approach that scales.

Three metrics kept surfacing as carrying real signal:

Code criticality. How heavily other parts of the system depend on this module. A gap in something many services call into is not the same animal as a gap in something nothing else touches. Code complexity. More branching logic means more distinct paths through the code, and more of those paths slip through untested even when the line-coverage number looks fine. Change frequency. Files that get modified often carry more defects than files that sit untouched, making active churn a meaningful risk signal.

The operational takeaway here is blunt: a gap in a low-complexity utility function that three files import is not the same deployment risk as the identical gap sitting inside a billing calculation or an authentication check. Teams that review both with the same level of scrutiny are spending review time in the wrong place. Lightweight prioritization, according to the study's own quality engineers, was enough to sort the gaps that actually needed blocking review from the ones that could wait.

That only matters, though, if getting the sorting wrong actually costs something real. It does.

What undetected gaps cost when they reach production

The cost of a defect doesn't rise in a straight line as it moves through the pipeline. It compounds. A bug caught during design costs somewhere around $100 to fix. The same bug, caught after it's live in production, costs closer to $10,000, roughly a hundred times more for the identical mistake, just found later. IBM's Systems Sciences Institute found something similar from a different angle: an error caught after release costs 4 to 5 times more than one caught during design, and up to 100 times more than one caught during maintenance.

Zoomed out to the national level, the number gets hard to hold in your head. The Consortium for Information & Software Quality, co-founded by OMG and Carnegie Mellon's Software Engineering Institute, put the aggregate cost of poor software quality in the US at an estimated $2.41 trillion. At the individual company level, that shakes out to over $1 million a year in average losses, and 45% of US businesses report losses above $5 million annually from software quality problems alone.

Downtime itself carries its own separate price tag. Gartner estimates the average cost at $9,000 per minute of downtime. Against a two-hour outage, that rate puts the cost well past a million dollars before anyone's even started debugging.

Then there's the tail risk, the kind of event that reminds everyone why this conversation matters. When CrowdStrike shipped a faulty update in July 2024, it crashed 8.5 million Windows machines and produced a financial impact that exceeded $5 billion. That's what happens when a change with untested failure modes reaches scale before anyone catches it.

None of this is edge-case territory, either. 40% of companies report at least one critical software failure every quarter. That's not a disaster scenario, that's a routine operating condition for close to half the industry.

And speed is making it worse, not better. The 2024 DORA report found that AI adoption correlated with a 7.2% decrease in delivery stability. Teams are shipping faster. Testing coverage isn't keeping pace with that speed, and the gap between velocity and verification is widening, not closing.

None of this is about scaring anyone into paralysis. It's about recognizing where in the process a dollar spent on catching a bug buys the most protection. Which raises the actual operational question: how do you read the signals before the deploy, not after the incident report?

How to read test gaps as deployment risk signals before a release

The shift starts with framing. Testing isn't a signoff ritual where the goal is a clean report at the end of a sprint. It's a feedback loop, and its job is to surface regressions fast and classify them correctly, before they're customer tickets.

That means reading three layers of signal together, not in isolation:

Code-level signals. Cyclomatic complexity on the files that changed. Lint debt piling up. The historical defect rate of whatever got touched in this particular pull request.

Workflow signals. How long the PR sat before merge. How long the test suite took to run. Whether any test stages got skipped or muted to hit a deadline tracks that muting decision as its own signal.

Business-impact signals. How often this module has triggered incidents before. The recent change-failure rate on it. Whether the code sits on a path a paying customer actually walks through every day.

The IEEE research supports prioritizing gaps by risk: gaps in high-criticality, high-complexity, frequently-changed modules warrant the most urgent attention, while gaps in stable, low-dependency utility code can be addressed later. Treating every gap identically isn't caution, it's just inefficient use of review time that could go toward the gaps that actually matter.

If the gap surface feels too large to tackle all at once, don't try. Pick one critical path. One pipeline stage that keeps flaking. One test area that generates noise without catching anything real. Tighten those first. Trying to instrument everything simultaneously usually means nothing gets done well.

According to the World Quality Report 2025-26 from Capgemini, teams automate only 33% of their test cases on average. The large majority of testing across the industry is still manual. That's not a minor footnote, it's a structural constraint that gets worse every time deployment frequency goes up, because manual testing doesn't scale the way automated testing does.

Quality gates are the enforcement mechanism that makes all of this systematic instead of ad hoc: automated checks that block a merge when coverage drops below a threshold, when complexity spikes, or when a critical path is missing assertions. Not bureaucracy for its own sake. The thing that makes gap-reading a repeatable process instead of a judgment call someone has to remember to make.

For behavioral gaps specifically, the audit looks a little different. Pull the documented expected behaviors out of API contracts, method-level docs, and user-facing specs. Map each one against the tests that currently exist. Whatever's left over, behaviors with no assertion covering them, are the highest-priority gaps on the list, regardless of what the line-coverage report says.

That last point matters even more once the thing you're testing isn't code your team wrote.

Silent updates and behavioral drift: the gap problem in LLM-dependent systems

Hosted LLM services introduce a governance problem that traditional test-gap thinking wasn't built for. The API endpoint doesn't change. The version number doesn't change. But the model behind it can shift anyway, silently, on the provider's own schedule, and a test suite built against yesterday's behavior has no way to know today's behavior is different.

This isn't theoretical. Anthropic's own postmortem from September 2025 documented three separate bugs affecting Claude Sonnet 4, none tied to any API modification, including a routing error that affected up to 16% of requests and an output-corruption bug that injected non-ASCII characters into responses. Separately, research from Chen et al. in 2024 tracked GPT-4's rate of producing directly executable code and found it dropped from 52% to 10% over three months, March to June 2023, with no version change announced anywhere in between.

Standard regression testing doesn't catch this well, and there's a specific reason why. A model's aggregate performance can improve on the benchmarks the provider publishes while regressing hard on the narrow set of tasks a particular application actually depends on. Echterhoff et al. (2024) made this point directly: global metrics don't guarantee behavioral compatibility at the level of a specific workload. The averages can look fine while your use case quietly breaks.

A 2025 paper out of NTNU (Chishti, Oyinloye, and Li) proposes a governance framework built around three pieces:

Production contracts. Explicit, written rules for how the model has to behave inside a given application: authentication code has to pass its security tests, JSON output has to validate as JSON, no exceptions. Risk-category regression suites. Targeted tests aimed at the tasks where drift is most expensive: security logic, billing calculations, anything where a malformed output has real financial consequence. Compatibility gates. Automated checkpoints that block a model update from going live in production until it passes the contracts defined for that application, forcing a structured review instead of a silent swap.

The parallel back to conventional test gaps is almost exact. A behavioral gap in an LLM-dependent system, like a behavioral gap in code someone actually wrote, does not appear on a coverage dashboard. It requires someone to write down what "correct" means for this specific use case, and then test against that definition on purpose.

Zoom out from the mechanics of any single gap type, though, and these problems tend to originate at the MVP stage.

Why MVPs are where test gaps are born and where they compound fastest

Early on, manual QA is genuinely faster than building out automated coverage. So teams ship without unit tests, without integration tests, without a regression suite, because speed to first users lets them reach the market before test infrastructure that isn't paying off yet would. That's not a mistake at the MVP stage. It's a reasonable trade.

The debt is usually deliberate, at least at first. The team knows what corner it's cutting and plans to circle back once things stabilize. Then things don't stabilize on schedule, and undocumented, constraint-based debt turns into accidental debt fast: nobody left on the team even remembers why the shortcut exists anymore, only that touching that file makes everyone nervous.

The real inflection point arrives with product-market fit: real users with real money on the line replace the experiment. That's when the interest payments on skipped testing stop being theoretical.

• Every new change now carries an unknown risk of breaking something else, because there's no regression safety net to catch it.
• Releases get more fragile. Confidence in shipping declines. Onboarding a new engineer slows to a crawl, because every change now needs manual re-verification instead of a test suite doing the checking.

Healthy engineering teams typically run a maintenance ratio, the share of engineering time spent maintaining existing code versus building new features, somewhere between 15% and 25%. Healthy engineering teams typically run a maintenance ratio, the share of engineering time spent maintaining existing code versus building new features, somewhere between 15% and 25%. Once that number crosses 35%, the debt has reached a point where it's actively slowing down new development, not just adding friction around the edges.

And this isn't a niche problem. A 2022 McKinsey study found technical debt can account for up to 40% of a company's entire technology estate. A 2024 survey of technology executives found that for more than half of companies, technical debt eats up more than a quarter of the total technology budget. That's real money, sitting there every year, just to keep old shortcuts from collapsing.

AI-assisted development is speeding this cycle up, not slowing it down. Building by iterating until something "works," the vibe-coding approach that's become common with AI-assisted MVPs, tends to produce systems that lack unit tests, integration tests, and regression coverage by construction, because the loop that produces the code was never built to produce tests alongside it. The DORA finding on AI adoption and the 7.2% decrease in stability correlated with AI adoption is the macro version of this same pattern, playing out across the industry at once.

None of this means ripping out the codebase and starting fresh. It means putting down the testing infrastructure, contracts around expected behavior, coverage on the paths that actually matter, gates that enforce it automatically, that makes every future change safe to make.

That dynamic doesn't play out the same way everywhere, though. Inside nonprofits, the same test-gap problem occurs wrapped in a different set of constraints.

How the same gap problem plays out differently inside nonprofits

Budget and staffing set the ceiling here before a single line of code even gets written. 77% of nonprofits say their current budget couldn't support hiring a full-time technologist, and fewer than 1 in 5 have a dedicated technology staff member. That's not a testing gap yet. It's the condition that makes testing gaps almost inevitable downstream.

Training compounds it. 68% of nonprofit staff report they aren't adequately trained on all the hardware and software they're expected to use day to day, and 98% say they want their staff to build deeper technical skills, a want that budget rarely lets them act on. Then there's the data side: 97% of nonprofits collect data of some kind, but only 6% rate their own ability to analyze and use that data highly. Collection without capacity just produces a pile of numbers nobody has time to turn into a decision.

The spending pattern makes the imbalance concrete. Smaller nonprofits, those under $1 million in budget, spend roughly 13% of their total budget on IT. Their larger peers spend just 1.5%. Smaller organizations are carrying a technology burden that's wildly out of proportion to what they actually have the staff or budget to support. Testing infrastructure, the thing that requires dedicated time nobody's paying for, is almost always the first casualty.

Bhaskar Jayakrishnan, Cisco's SVP of Engineering for Customer Experience, has described technical debt in this context as a hidden tax on mission delivery. The framing holds up under scrutiny: systems are interconnected, so an unpatched operating system or a legacy integration sitting quietly in one corner of a nonprofit's tech stack doesn't stay contained. It becomes the reason a donation form fails during a fundraising push, or a volunteer database locks up during an event that took months to plan.

That's the same test-gap logic running through every section of this piece, just under tighter constraints and higher mission stakes. The gap doesn't announce itself in a code review. The system carries more weight than anyone tested it for the day this occurs.