Here is a short, opinionated one. A security check that quietly passes when it cannot do its job is worse than having no check at all. Not equally bad. Worse. Because at least with no check you know you are unprotected. A check that fakes green tells you that you are safe while you are not, and you make decisions on that lie.
The choice every check has to make
Consider a dependency scanner whose first step is to fetch the code it is meant to inspect. One day that fetch fails. Maybe a project was renamed and the old location no longer resolves. The scanner now faces a choice, and it is a choice every check in your pipeline faces at some point.
It can silently skip the project it could not fetch, scan whatever it did manage to reach, and report “all clear.” Or it can stop, mark the run incomplete, and refuse to claim anything is clean when it never actually looked.
Those two behaviours produce the same green tick on a good day and wildly different outcomes on a bad one. The first manufactures confidence out of a failure. The second tells you the truth.
What happened
We hit exactly this. A project rename broke the step that fetches the code, so one project in the sweep could not be scanned. The scanner did the right thing: it produced an incomplete report rather than a false all clear. The run visibly said “this project was not scanned, and here is why,” instead of leaving it out and letting the summary read as if everything had been checked.
That is the whole story, and that is the point. The failure was loud. Nobody had to notice a project was missing from a list. The report announced its own gap.
The principle
Any check that degrades to “pass” on error is actively dangerous, because it converts an operational problem, we could not run the check, into a false statement about safety, the check passed. Those are not the same thing and they must never render as the same result.
So build your checks to fail loud. When a check cannot complete, its outcome should be visibly not green: incomplete, errored, needs attention, whatever your pipeline calls it, as long as nobody can mistake it for success. A red state you can trust is worth more than a green one you cannot, because the entire reason the check exists is to be believed.
When you review your own tooling, ask one question of every check: what does it do when it cannot do its job? If the answer is “reports success,” you do not have a safety net. You have a decoration that will let you fall.
Leave a Reply