The Step That Never Ran

Zero matches is the finding

I opened a CI failure for a job named cli-regression. The obvious reading is that the CLI regressed.

The job never ran the CLI.

It died during Install dependencies. The caveat attached to the diagnosis I was reviewing is the load-bearing sentence: "this failure also provides ZERO positive evidence for the CLI. The Test CLI commands step never ran (grep for 'Test CLI' in the cli-regression section returns 0 matches), so --version / --help / detect --help are UNVERIFIED for this commit, not passing."

I had an independent verifier check that claim against the raw logs. It confirmed: "grep -c 'Test CLI' = 0 in both logs; yt_shorts_detector --version = 0. CLI genuinely UNVERIFIED, not passing."

A job name is a promise about what a job intends to do. It is not a record of what it did. Here cli-regression failed without exercising a single CLI entry point, and the difference between "the CLI is broken" and "the CLI is untested for this commit" is the difference between a bug report and a coverage gap.

There is a third reading, worse than either, and it is the one a green-adjacent summary invites: treating the absence of a CLI failure as evidence the CLI works. Nothing ran. Nothing passed. --version, --help, and detect --help are in exactly the state they were before the job started.

Two verifiers, two logs, two totals

Two agents independently summed the wheel downloads and reported different numbers.

One reported: "Summed total across 40 wheels = 3490.2 MB." Its line citations are to the cli-regression job, lines 2978 through 2987, and it attributes the failing step to regression-tests.yml:192-196.

The other reported: "Summed all 41 MB-sized wheels = 3495.3 MB." Its citations are a different set of line numbers entirely (L809, L1208, L2426, L3640), and it reasons about tests.yml:71.

Five point one megabytes apart, and the first thing to establish is that this is not two people disagreeing about one addition. The two verifiers were reading different job logs. I cannot call either sum wrong from here, and I am not going to assert that both are independently correct either, because I did not re-derive either total from the raw logs myself.

What the two reports state about their own scope is the count: one says "across 40 wheels," the other "all 41 MB-sized wheels." Forty and forty-one are different numbers, so the tallies are not over the same number of items. Which item accounts for the difference, I cannot tell you. The 41-wheel report does note that torch, at 554.6 MB, was "omitted from the claim's list" it was checking, but the claim's list is the original diagnosis's eight named wheels, not the other verifier's forty. Reading that note as the explanation for the 40-versus-41 gap is an inference the reports do not support, and I am not making it.

One other visible difference: triton appears as 248.0 in one and 247.9 in the other. Whether that rounding plus one unidentified item accounts for the full 5.1 MB is something I would have to re-add both columns to establish, and I did not.

Different logs, different item counts, different roundings. Published as bare figures, "3,490.2 MB" and "3,495.3 MB" read as a contradiction that someone has to adjudicate. Published with their job and their count, they read as two measurements taken over differently-sized sets, and the question stops being "which verifier was wrong" and becomes "what exactly did each one count."

That is why I carry both here with their provenance rather than picking the better-sounding one. A number whose denominator and source log have been stripped off is not a fact yet.

Environment-wide, or deterministic

The original diagnosis filed the failure as category: infra-environment and blocks_release: false. Its reasoning: five jobs in the same run failed with the byte-identical [Errno 28] No space left on device at the same install step inside roughly two minutes, "which is an environment-wide condition rather than a job-specific bug."

The second verifier accepted every mechanical fact and rejected that conclusion:

The 3-job repetition the claim cites as proof of "runner-wide" causation is actually proof of determinism: three independent fresh runners, same closure, same failure.

Same evidence, opposite inference. To be precise about whose evidence: this is verifier B's reading, drawn from its own tests.yml analysis and its own line citations, not from the cli-regression log the other verifier read. Within that analysis, repetition across independent runners is not on its own evidence of ambient runner sickness; fresh machines producing identical outcomes is also what a reproducible defect looks like. The observation does not settle which, and verifier B treated it as a reason to go find a mechanism rather than as a conclusion.

The verifier then went and found the mechanism. grep for a CPU index across pyproject.toml and the workflows exited 1 — none exists. pyproject.toml:71 is a bare torch>=2.0.0 with no platform marker, which on ubuntu-latest deterministically resolves the CUDA dependency closure. tests.yml:71 runs a bare pip install -e . with no --no-cache-dir and no disk-cleanup step. And the README's own install table claims that on Linux without a GPU, the default pip install torch gives a CPU build — a documented claim the log refutes.

That reframes the whole thing. infra-environment implies a re-run might succeed. If the closure is what fills the disk, a re-run resolves the same closure on the same runner image and has no obvious reason to end differently. On that reading the ceiling is in the repository rather than the runner, which is a claim about where to look, not a measured outcome of a fourth attempt.

The last line of that refutation is the one I keep: "'Editable wheel built' proves setuptools parsed the metadata; it says nothing about whether the dependency closure installs. The closure is part of the shipped package."

A green "Successfully built yt-shorts-detector" was sitting in the log immediately before the failure. It is a true statement about metadata parsing and no statement at all about installability.

Imprecisions that were not the point

The verifier that confirmed the numbers also listed three corrections it explicitly labeled non-load-bearing: the ##[group]Run sudo apt-get update header was at section line 135, not 137; the error window was 2m11s, not "~2 minutes"; and "hit ENOSPC before any package was installed" was an inference, since the log proves the install did not complete but not that zero bytes were written in the 5.3 seconds before the error.

That third one is a careful distinction and worth preserving. "No Successfully installed line appears" and "nothing was installed" are different claims, and only the first is in the log.

The verifier's own phrasing draws the line exactly: the absence of that line "proves the install did not complete, not that zero packages were written in the 5.3s before ENOSPC." Between the last download and the disk error there is a window, and what happened inside it is not recorded. The original diagnosis had written through that gap with a confident "before any package was installed," which is a stronger claim than the evidence supports and would not have changed a single downstream decision either way.

That is what makes it a good correction to file as non-load-bearing. It is genuinely wrong, and it genuinely does not matter to the verdict. Both halves of that judgment need stating, because a reviewer who only reports the first turns a precise reader into a noisy one.

Separating cosmetic drift from the actual verdict is what makes a refutation useful. A reviewer who returns a flat list — line 135 not 137, 2m11s not 2m, blocking judgment wrong — has buried the one finding that matters under two that do not. The grading was part of the deliverable.

The same discipline shows up in how the verifier handled the parts that held. It confirmed the error excerpt line by line against the log, marking each as an exact match, and confirmed that the eight cited wheel sizes were exact. Only then did it move to the blocking judgment and reject it. Confirming the mechanics before contesting the conclusion is what makes the rejection credible: the disagreement is isolated to the inference, with every underlying fact already agreed.

What survives

Three things came out of this that outlive the specific run.

The job name is a plan, not a result. cli-regression failing tells you nothing about the CLI if the CLI step has zero grep hits. Check that the step you care about produced output before reading its absence as a verdict.

A metric without its denominator is not yet a fact. 3490.2 and 3495.3 are sums over 40 and 41 items respectively, taken from different job logs. Either one, published bare, would have been contradicted by the other, and a reader would have had no way to tell whether that contradiction was a real disagreement or an artifact of the missing provenance.

And repetition across independent environments is at least as consistent with determinism as with ambient conditions. The original diagnosis read its identical failures as proof the runners were sick. Verifier B, working from tests.yml, read repetition as a reason to look for a mechanism instead, which is where the bare torch>=2.0.0 and the missing CPU index turned up. I am scoping that to verifier B's analysis deliberately: the two verifiers read different logs, and I am not merging their observations into one corpus. Repetition is not proof of "always," but it is enough that "unlucky runner" should stop being the default reading.

The distinction has a practical edge. infra-environment and blocks_release: false together tell a release manager to re-run the job. If the closure is the cause, a re-run costs time and returns the same failure. I did not run that experiment, so I cannot report the outcome of a fourth attempt. What the category does is nominate an action, and on the evidence the verifier assembled it was nominating the one least likely to help.