Green lights that lie

The most dangerous dashboard is a green one over a broken system. In one week last August I worked three incidents that shared a signature: a stale build image, a live critical CVE, and days of silent data loss - and in every one of them, every existing check, counter, and dashboard reported healthy the entire time. The failure was never "nobody was watching." Everything was watching. The problem was that each watcher observed a name for the thing, a label on it, or a neighbor of it - and never the thing itself.

A check that watches half a system and reports on the whole is worse than no check, because it is trusted. So I now put every check, alert, probe, and CI gate through six questions before I trust it. Each one came out of a real incident, and each one prevents a specific way a green light lies.

Does it read the artifact, or a name for it?

Our build server runs a custom image that bakes its plugin set from a file in git. Six plugins got upgraded through the UI, then faithfully recorded in git

• and the image was never rebuilt. For three days, three observers agreed everything was consistent: the file in git listed the new versions, the running instance served the new versions, and the weekly check diffed the running instance against itself and passed. The only dissenting observer was the image, and nothing asked it. The drift would have surfaced only on a volume wipe - the exact disaster the baked image exists to survive - as a silent downgrade.

A name - a tag, a git path, a version string - is a claim about content, and claims drift. The gate has to read the content. The fix here was two-part: the image tag now carries a hash of the plugin file's actual contents, so CI can verify the tag names the right bytes with pure text comparison, and a daily job inside the running system re-hashes the baked file against what git says. One gate proves the tag points at the right content; the other proves the content is behind the name. Each states in a comment what it cannot see.

Does it cover the whole claim its name makes?

We had a daily security check on that same build server. It queried the official advisory feed and reported "no advisories" every day. Then the vendor published a critical remote-deserialization CVE against the exact core version we were running - and the check kept passing, because it read advisories for the plugins only. It said nothing about the core they ran on. The warning sat on the instance's own admin page for days. We found it by accident.

A check is trusted at the level of its name, not its code. There are two honest exits: extend the check to cover the whole claim, or shrink the name to match the implementation - "plugin update check" is honest about what it does. What's not acceptable is the middle state, broad name over narrow query. Audit a gate by reading its name aloud and then its query. Where they diverge, you've found where the next incident will hide.

Does it check aged state, or only fresh state?

This one cost the most. A tracing store's volume filled, the write-ahead writer wedged on a corrupt block, and from then on every trace was accepted, queryable for a few minutes, then destroyed. Ingest counters: healthy. Error counters: zero. And the trap inside the trap: a synthetic test span sent through the pipeline got a success response and was queryable seconds later - served from live memory - before evaporating. A send-and-query probe reported a healthy pipeline while the store lost everything. That fresh probe "passing" sent the investigation in the wrong direction for over an hour.

Any system with a buffer, cache, or write-ahead log in front of durable storage can serve you a fresh answer from the part that works. The probe has to ask for something old - old enough that it can only exist if the durable path works. The gate we shipped sends a span through the real production path every thirty minutes, checks it fresh, and then queries for the span from three runs ago, about ninety minutes old, past every in-memory window the system has. Deterministic IDs derived from the clock make that stateless: any run can compute what an earlier run must have written.

Does the alert self-clear, and does absence fire?

Two expression-level rules, both learned by tripping over their opposites. Alert on the most recent success, not on the existence of a failure - a failed-job object lingers after the underlying problem is fixed and latches the alert red until a human cleans it up. "How stale is the newest success" self-clears the moment the next run goes green. And a dead watchdog must not read as a quiet one: if the probe itself is deleted or stops scheduling, most alerting sees no data and treats it as fine. Silence is not an allowed state for a gate. Every watchdog needs a wired answer to "who notices when the watchdog dies?"

Has it ever fired?

A gate that has never fired is unproven coverage. Before trusting one, break its target on purpose and watch it catch the break. Edit the input and keep the tag - does CI fail? Force the runtime condition with a wrong expected hash - does the job go red with the right message? For alert rules, history is a free regression corpus: evaluate the expression at a timestamp inside the actual incident it was built from. A rule that returns the right answer at the incident's own timestamp has passed the only test that matters - it would have caught the thing it exists to catch.

Does it fail only when the thing is broken?

The first version of our core-version check read a vendor URL that exists per released version - and stops existing once you are the newest release. So the check's first real run failed precisely because everything was fine. A check that goes red when the system is healthy is worse than the hole it covered, because it teaches everyone that red is noise, and the next red will be real. Corollaries: an unreachable dependency reports "inconclusive," loudly, never a silent pass; shape-check external answers before comparing them, so an HTML error page never gets compared as if it were a version string; and match the check's cadence to the rate of change of what it watches, not what you control. Our config changed when we rebuilt it. Advisories publish on no schedule of ours. The check runs daily now.

One more thing on cost, because honest gates multiply. Order them cheap to expensive, and make the cheap ones eager to reject and reluctant to accept. A cheap check that confidently rejects junk saves the expensive verification for the cases that genuinely need it - but a cheap check that confidently accepts on weak evidence corrupts everything downstream. Keep the expensive check as the gate. Just feed it less.

The limit worth stating plainly: these six questions harden a gate against the failure modes someone has already imagined - usually because someone already paid for them. They don't enumerate the failure modes nobody has imagined yet. Every test above was tuition from a real incident, and the next incident will find the gap none of them name. Which is why the last rule is a posture, not a test: when a gate you wrote passes, treat it with the same suspicion as a gate you inherited. The core-version check above was itself the fix for the CVE blind spot, and it shipped with its own bug. The fix for one gate's lie is the most likely place the next one starts.