The Negative Control: Why a Test Suite of Only Positive Matches Cannot Catch Over-Matching

Twenty-two greens and one invoice that should have failed

I had just generated two consulting invoices for Headspin, June and July 2026, and written a 22-check PDF verifier to make sure I never shipped stale bank details again. I ran the verifier on both new PDFs. June came back PASS (22/22). July came back PASS (22/22). Every banking field matched the current TD config, the hour caps held, the totals were right, the stale-account check was green.

I did not stop there. I pointed the same verifier at an old Chase-era invoice from December 2025, the one that still carried account 3902643775 and routing 021000021, against the current TD config. That PDF was supposed to fail hard. It did fail on several bank-name and total checks. It also reported PASS No stale account/routing numbers on an invoice that contained exactly those numbers. The check designed to catch the thing I was most afraid of had silently waved it through.

That is the shape of a negative-control failure: not that the suite is empty, not that the code is untested, but that every assertion in the suite only ever asked "does this find the good thing?" and never "does this refuse the bad thing?"

What the verifier was supposed to catch

The skill I was building, invoice-generation, pulls git history, spreads hours across weekdays, renders a PDF, and appends a ledger entry only after the PDF succeeds. The dangerous failure mode is not a crash. It is a clean render with the wrong bank footer. I had switched from JPMorgan Chase to TD ESSENTIAL BANKING. The generator read invoice.config.json, which was already correct. Project docs still listed the Chase numbers. I wanted a gate that would refuse any PDF whose extracted text still carried the old account or routing digits, even if every other field looked fine.

The stale-number check lived in verify_invoice.py. The idea was simple: collect the current account and routing numbers from config, find every 9-to-12 digit run in the PDF text, and fail if any found number is not in the current set. The original code looked like this:

current_nums = {norm(str(bank.get(f, ""))) for f in
                ("account_number", "routing_number") if bank.get(f)}
found_nums = set(re.findall(r"\b\d{9,12}\b", t))

norm stripped whitespace so PDFKit's hard wraps would not break substring checks. PDFKit on macOS routinely splits words across line boundaries, so HOURS comes back as HOU\nRS. Stripping whitespace made those checks reliable. It also, I would learn, destroyed the word boundaries the digit regex depended on.

On the two fresh TD invoices the check passed for the right reason: the only 9-to-12 digit runs in the text were the current account and routing numbers. On the Chase invoice it also passed, for the wrong reason. The numbers were sitting in the PDF. The regex never saw them.

The negative control that exposed it

After the two green runs I wrote, in the session itself: "22/22 on June. Testing July, plus a negative control to confirm the checks can actually fail." That sentence is the whole method. A green suite on known-good inputs proves the happy path. It does not prove the detector can discriminate.

The control was a real artifact, not a fixture I invented in memory. Invoice_202512-005 - Nick Krzemienski.pdf still lived in the invoices directory. It was an old Chase-era bill for the same client. I ran:

python3 ~/.claude/skills/invoice-generation/scripts/verify_invoice.py \
  "Invoice_202512-005 - Nick Krzemienski.pdf" \
  --config invoice.config.json \
  --invoice-number 202512-005 --hours 50 2>&1 | tail -18

The interesting lines in the output were:

  FAIL  Bank account_name: TD ESSENTIAL BANKING
  FAIL  Bank account_number: 4458021618
  FAIL  Bank routing_number: 026013673
  PASS  No stale account/routing numbers
  FAIL  No day exceeds 4.0h  (over-cap days: [8.0, 7.5, 10.0, ...])
  VERDICT: *** FAIL *** (12/18)
EXIT=0

Two separate bugs showed up at once. The | tail pipeline swallowed the process exit code, so EXIT=0 even though the verdict was FAIL. And the stale-number check, the one I cared about most, had passed on a PDF that still printed Chase account 3902643775 and routing 021000021. Banking-name mismatches failed correctly. The digit matcher did not.

A suite that only ever saw good invoices would have shipped this. The negative control is what made the green stale check look suspicious instead of reassuring.

What \b actually did wrong

I pulled the raw PDF text and ran the same normalization the verifier used:

Chase acct 3902643775 present in normalized text: True
Chase routing 021000021 present: True
regex \b\d{9,12}\b hits on normalized text: []

The digits were in the string after norm. The word-boundary pattern found nothing. The raw context around the account line was clean in the unnormalized extract:

'Account Number: 3902643775\nBank Name: JPMorgan Chase'
'Routing Number: 021000021\nPayment Terms: NET30'

After full whitespace stripping, those regions collapsed into forms like ...3775BankName: and ...021000021PaymentTerms:. In Python's regex engine, \b is the boundary between a word character (\w, which includes digits) and a non-word character. Once you glue 3775 directly to Bank, there is no boundary between the last digit and the following letter. Both sides are word characters. The pattern \b\d{9,12}\b requires a boundary on both ends. With the boundaries gone, the match set is empty, found_nums is empty, and "no unexpected numbers" becomes a vacuous truth.

That is the precise failure mode. The positive tests never stressed it. Every good invoice's current account and routing numbers still had enough surrounding punctuation or spacing, after normalization, for \b to fire. The over-match case, more accurately the under-match case here, only appears when you feed the detector a string that contains the forbidden digits in a shape the pattern cannot see.

The session note I left myself after the diagnosis was blunt: after stripping all whitespace, \b boundaries vanish, so the regex never matches. The fix is not "stop using \b." The fix is to stop destroying the boundaries the pattern needs.

The fix that keeps the boundary and still heals PDF wraps

PDFKit still hard-wraps mid-number. I had seen routing digits split as things like 0210 00021 in other extracts. Fully stripping whitespace was the wrong hammer for that. The right hammer removes only whitespace that sits between two digits, and leaves every other space alone so \b still has something to anchor against:

current_nums = {norm(str(bank.get(f, ""))) for f in
                ("account_number", "routing_number") if bank.get(f)}
# Search a copy where only *inter-digit* whitespace is removed. Fully
# stripping whitespace would glue digits to adjacent letters and destroy
# the \b boundaries this pattern relies on, silently matching nothing.
digit_safe = re.sub(r"(?<=\d)\s+(?=\d)", "", raw)
found_nums = set(re.findall(r"\b\d{9,12}\b", digit_safe))

That is the verbatim before and after from the session edit on verify_invoice.py. The lookaround (?<=\d)\s+(?=\d) collapses 0210 00021 into 021000021 without turning 3775Bank into a single word token. The comment in the patch is the durable part: fully stripping whitespace would glue digits to adjacent letters and destroy the \b boundaries this pattern relies on, silently matching nothing.

I re-ran the negative control without piping through tail, so the exit code would survive:

  FAIL  No stale account/routing numbers  (unexpected: ['021000021', '3902643775'])
  VERDICT: *** FAIL *** (11/18)
EXIT=1  (expect 1)

Both Chase numbers now failed the check, and the process exited 1. Then I re-ran the two real invoices:

202606-012 -> exit=0 |   VERDICT: PASS (22/22)
202607-013 -> exit=0 |   VERDICT: PASS (22/22)

That pair of results is the minimum proof the detector discriminates. Known-good inputs still pass. A known-bad input fails on the exact field it is supposed to fail on. Without the second half, 22/22 is a rubber stamp.

Why a suite of only positive assertions cannot catch this

Every check in the original green runs was a positive assertion in the logical sense that mattered here. "Does the PDF contain the current account number?" "Does the PDF contain the current routing number?" "Does the found-digit set equal the current-digit set when the input is a correct invoice?" Those questions all answer yes on a good PDF whether or not the matcher can see a bad number on a bad PDF.

An over-matching pattern, or in this case a pattern that matches nothing after a destructive normalize, is invisible to a test that only feeds it cases where the expected answer is "found." If the expected answer is always "found," a detector that always returns "found" and a detector that always returns "nothing, so the set comparison happens to pass" can look identical. The negative control flips the expected answer. It is a case designed so that a correct detector must FAIL. If it passes, the detector is broken, full stop.

I had a second, quieter version of the same lesson later in the same session. A timesheet-hours parser returned an empty list on an inline-layout template it was not written for, and "no violations found" on an empty list is another vacuous pass. Empty-result-means-clean is the same family of bug as empty-match-set-means-no-stale-numbers. Both are green because the code path that would have produced a finding never ran.

Lab science has a name for the missing piece. A control that always passes proves nothing about the assay. You need a sample that must come back negative, or must come back positive against a blank, before you trust the instrument. Software verification that only ever asserts the happy path is an assay without a negative control.

Where this generalizes past one regex

I keep hitting this shape outside invoice PDFs.

Validation gates that only assert "required field present" will greenlight a form that accepts " " or a SQL fragment in a field the schema types as a string. The positive check passes. The negative check, "this value must be rejected," was never written.

Security filters that only assert "blocked payload is blocked" on a single known bad string will miss the encoding of that string that slips through. The positive block test is necessary. It is not sufficient. You also need the cases the filter is supposed to let through, and the near-miss encodings it is supposed to still catch, or you cannot tell discrimination from a hard-coded special case.

Any matcher, allowlist, denylist, classifier, or lint rule has the same requirement. Assert the match. Assert the non-match. Prefer a real artifact that should fail over a synthetic string you invented to make the test pass. In my case the real artifact was sitting in the same directory as the good invoices. Running one extra command against it was cheaper than the invoice I would have sent with the wrong bank footer if the gate had stayed vacuous.

The durable rule I took out of the session is small enough to keep next to every verifier I write: after the suite is green on known-good inputs, feed it one known-bad input and require a fail. If the known-bad input passes, the suite was never testing discrimination. It was testing that the code path runs. Those are not the same claim, and only one of them is worth shipping behind.