The arrival test

A committed configuration file in my infrastructure repo almost took down a production service, and the dangerous thing about it was that everything anyone needed to know was already documented.

During a migration, the live routing for a geocoding service had been repointed by hand to a new set of workloads. The committed manifest was never updated - it still carried the old selector, pointing at workloads that no longer existed. Anyone reconciling the repo to the live system would have applied that file and pointed production traffic at nothing, instantly. Worse: a diligent reader who noticed the mismatch would conclude that the live system had drifted and that the right move was to restore the committed version - the file's own contents argued for the destructive action.

The fact that would have prevented this existed. It lived in the cluster's state, in one person's memory, and in a long operations doc - about 52,000 characters away from the file where the damage would start. It was written down. It just wasn't going to arrive.

That's the failure mode I now test for deliberately, and I gave the test a name: the arrival test.

The question is not "is it documented"

Documentation reviews check whether a document is correct, complete, current, and readable. There's a fourth way to fail that none of those reviews can see: the document is fine, and unreachable at the instant of the decision. The research literature backs this up - a large study that mined hundreds of documentation issues found "findability" is its own distinct failure branch, separate from correctness and staleness, and put it bluntly: if existing documentation cannot be found, it conceptually does not exist.

I run a related check I call the stranger test: hand an artifact to readers with no context and see if they can reconstruct what it means. The Arrival Test is its inverse. The stranger test asks: given this document, is it legible? The arrival test asks: given this moment, does the document arrive? An artifact can pass the first perfectly and fail the second totally.

How the test works

You don't hand anyone a document. You hand them a task.

Drop a reader - a person or an AI agent - at the moment of need: the file they're about to change, the alert that just fired, the diff awaiting review. Tell them nothing about what they're supposed to find. Blinding is the whole instrument: the probe must not know the governing fact exists, because the moment you say "there's a doc about this," you're measuring search-when-told, not discovery. And start them at the work, never at the wiki or the search box - let someone start at the docs portal and you've tested the docs portal.

Then watch. Whatever they act without is siloed, no matter how well-written the document containing it is.

I score each probe with one of five verdicts. REACHED: the fact arrived unprompted, within a small hop budget. REACHED-LATE: it arrived, but after the wrong path or after the mistake. SILOED: the fact exists and the probe acted without it. ABSENT: the fact doesn't exist anywhere. And STALE-TRAIL: the probe reached something, and it was wrong or out of date.

Two of those distinctions carry most of the value. SILOED and ABSENT have opposite fixes - a siloed fact needs a pointer at the point of need, an absent fact needs to be written - and conflating them is how someone writes a third document about a thing that was already documented twice, making the corpus less reachable, not more. And STALE-TRAIL ranks worst of all, worse than the fact not existing. A probe that finds nothing keeps looking. A probe that finds a confident, outdated answer stops - and one stale hit can poison a reader's trust in an entire channel, siloing everything else in it.

The fix, when a probe fails, is almost never a rewrite. The manifest that nearly caused the outage was fixed with a comment block at the top of the file itself: this selector is intentional, here's why, do not "correct" it. The next probe reached the fact at hop zero - it was in the artifact the work starts in. The long operations doc didn't change at all. Arrival failures are path problems. You fix the path.

Coverage numbers lie without validation

One measurement from building this convinced me that the obvious metric is a trap. A widely deployed open-source monitoring stack ships its alerts with runbook links - and scores 100% on coverage, 139 alerts out of 139 carrying a link. Perfect, until you follow the links: about 21% returned a 404, and another 16% resolved to stubs whose diagnosis section literally read "to be added." Nearly 37% dead or hollow, on a stack running in thousands of production clusters, and the coverage metric reads 100% the whole way. That's STALE-TRAIL at scale. If you measure whether pointers exist without checking where they land, you are auditing your own good intentions.

Agents make this measurable - and urgent

Here's why I care more about this now than I did five years ago: most of the readers in my systems are AI agents, and for an agent, arrival is mechanical. A person who can't find the rule can ask a colleague. An agent can't. If a constraint isn't loaded into its context by its own mechanics before it writes, the constraint does not exist for it - and vendor documentation says plainly that instruction files are advisory context with no guarantee of compliance.

Placement, it turns out, dominates. One published result from the model vendors' own research: adding a single scaffolding line to a prompt raised a model's needle-retrieval score from 27% to 98%. The information was in the context window the entire time. Presence is not retrieval - for machines any more than for people. Which is why my hard constraints don't live in prose an agent might or might not weigh. They live in hooks that inject the fact at the exact moment of the edit, or block the edit outright. The manifest comment is for readers. The hook is for writers.

None of this is a new instinct, and I won't pretend otherwise. Google's disaster-testing program was running this probe at company scale in 2012 - their first exercise found exactly one person able to locate the documented emergency plan; after fixing the paths, more than a hundred could. What I've added is only the instrument: the blinding rule, the verdicts, the hop budget, the target validation.

The honest limitation

The strongest attack on this method comes from inside it, and I can't answer it yet. The arrival test prescribes putting anchors at the point of need, then measures whether anchors exist at the point of need. Run the test, add an anchor, re-probe, and improvement is guaranteed by construction - you moved the fact to where the measurement looks. My manifest story has exactly this shape: SILOED before, REACHED after, and zero direct evidence the anchor has prevented anything. Proving that would take longitudinal outcome data - do anchored repos actually see fewer incidents and fewer bad reverts - and I don't have it. Every result I've produced so far is an internal-consistency result, and I'd rather say that than imply otherwise.

And one finding at my own expense, because the recursion is the point: when I audited my own documentation corpus's link graph, 36% of documents were unreachable by link from any entry point - including, at the time of the audit, the very guide describing this method. A document arguing that unreachable knowledge doesn't exist was, as published, unreachable.

The test still earns its keep. The one-line version: if it matters that someone knows a thing, don't ask whether it's written down. Put someone at the moment they'd need it, tell them nothing, and watch whether it arrives.