I measured 47 days of talking to AI agents

Over 47 active days this summer, my AI agents delivered 2,435,810 words of output to me. That's 51,825 words a day - about 3.9 hours of reading at a normal reading speed, every day, just to keep up with my own team. I was losing real hours to it, and I kept catching myself typing the same three characters into a session that had gone quiet: "eta?"

So I did what I do with anything that costs me hours: I measured it. This essay is about what the data actually supported, and - more usefully - about the two headline claims I published in the first draft that my own review process killed.

The method

I mined every agent transcript on my machine for the period: 345 session files, 2,724 analyzable turns. A turn is one prompt from me plus everything the agent said back until my next prompt. For each turn I characterized the reply - word count, headings, tables, bullets - and then classified my own next message. Was it a bare poll ("eta?", "continue", "?"), a correction ("that's wrong", "you didn't"), or a normal continuation?

The classifiers are regexes, saved alongside the writeup so the measurement can be repeated without drift. Significance is Fisher's exact test, with the threshold Holm-corrected for the roughly 15 comparisons I ran - because if you run fifteen tests and celebrate whatever clears 0.05, you're doing astrology with extra steps.

What survived

Two findings. That's the honest yield from 2.4 million words.

First: replies under 75 words are a problem. When an agent's reply was under 75 words, my next message was a poll 32.9% of the time. When the reply was 200 to 500 words, 5.2%. That's a 27.7 point gap, and the p-value is below one in a million. Short replies from an agent read like silence - they don't tell you what changed, what's next, or when, so you end up asking.

Second: headings matter above 200 words. Holding reply length constant, replies that carried a heading were followed by a poll 2.6% of the time, versus 6.6% without. Smaller effect, still clears the corrected threshold.

Everything else I believed about reply structure turned out to be noise at this sample size. Tables looked like they cut my corrections threefold - five events, doesn't survive correction. Bullets looked like they reduced polling - p of 0.0497, which is the statistical equivalent of a photo finish you lost. Both dropped.

I turned the two survivors into standing rules for every agent I run, with the measured numbers written directly into the rule so future agents know these aren't style preferences.

The stranger test that killed my own claims

Here's the part I actually want you to take away. Before publishing the writeup, I ran it through what I call a stranger test: four readers with no context - a newcomer, an executor, a skeptic, a domain expert - each got the document cold, in isolation, with no statement of what I meant. Their job was to reconstruct the intent and attack the claims.

All four reconstructed the intent correctly. The document was legible. It was not correct. Two of its loudest claims died under review, and the final version says so in its header: the retractions are the most useful part of the artifact - read them before the protocol.

Retraction one: "cap your agent fleet at four." The first draft computed a sustainable fan-out of about 4 concurrent agents from a supervisory-control formula, and recommended a hard cap. Review found the calculation was circular - I had derived both inputs from the same distribution, in a way that returns roughly 4 for any dataset, including someone who never runs two agents at once. Worse, the direct test contradicted the recommendation: when I stratified my turns by how many sessions were actually concurrent, the poll rate was flat from 1 agent to 9-plus, and corrections actually fell. There is no fleet-size signal in my data. The cap was the loudest recommendation in the draft, and it was wrong.

Retraction two: "37.7% of replies were answered faster than I could have read them." This was the dramatic stat - proof I was skimming a third of everything. It measured the gap between the last word of a reply and my next prompt. But agent output streams: it arrives progressively, and I read it while it generates. Counting the generation window as reading time, the figure drops to 13.6%, and the gradient inverts - long replies, which looked worst, were actually fine, because they take longest to generate. The line "half of everything written to you is written into a gap that cannot hold it" was a good sentence. I deleted it, because it was false.

I also went looking for the famous claim that an interruption costs "23 minutes and 15 seconds" to recover from, since it gets cited in every essay about focus. The paper everyone attributes it to never says 23, never measures resumption lag, and actually reports interrupted tasks finishing faster - the measured cost is stress, not minutes. Don't build policy on it. I nearly did.

Associations, not effects

One more thing the writeup says out loud, because most writeups in this genre don't: these are associations in observational data, not proven effects.

There's a confound I can't remove. Task ambiguity plausibly drives both sides of my findings - well-specified work formats cleanly into headings and rarely needs a follow-up, while exploratory work does neither. Holding length constant doesn't control for that. So the correct reading is not "adding a heading will cause fewer polls." It's "replies that had headings were followed by fewer polls, for reasons that may include the heading." Testing it properly means randomizing - having agents flip a coin on heading presence at constant content for a week. I haven't run that experiment. Until someone does, these are strong hypotheses with honest error bars, not laws.

The limitations don't stop there, and I'd rather list them than have you find them: the classifiers are unvalidated regexes with no inter-rater check, the comparison family wasn't pre-registered, and word counts include code blocks, which inflates length for code-heavy replies. Any of these could move the numbers by an amount I can't bound.

Why publish the failures

It would have been easy to publish the two clean findings and quietly drop the rest. The draft with the fleet cap and the 37.7% stat was a better read - more dramatic, more quotable, more wrong.

But I run a team of agents whose single worst habit is confident claims that collapse under verification. I can't hold them to a standard I don't hold my own analysis to. The retraction list is the proof that the review process has teeth - and it's the only reason you should believe the two findings that survived it.

If you supervise AI agents and take one thing from 47 days of my data, take this: don't accept a reply under 75 words, expect a heading above 200, and test your own favorite claim before you publish it. Mine didn't make it.