Documentation drift: how to detect it and fix it automatically

A team I know renamed a prop. variant="solid" became variant="gradient", the PR was clean, the changelog entry got written. The docs page for that component kept saying solid for eleven weeks.

Nobody was negligent. That page had been edited three times in the window — a typo, a new example, a reordered section — so its "last updated" badge was green and every audit that walks a sitemap hunting for old timestamps skipped straight past it.

The page was fresh. It was also wrong. Documentation drift is the gap between a documentation page and the source code it describes, measured from the exact commit that page was last written against. Freshness is a property of the page. Drift is a property of the relationship between the page and the code, and if your docs stack doesn't store that relationship, drift isn't something it can detect at all — only something someone eventually notices.

That distinction is the whole post. You cannot automate a fix for a condition you cannot measure, and most setups have no way to measure this one.

What is documentation drift, exactly?

The precise version: a page is in sync when the source files it documents are at the same commit they were at when the page was last written or verified. It has drifted when those files have moved on. That's it. Drift is a state you can compute, not a feeling you have about a page.

Note what falls out of that definition. Drift is not "the page is old." A five-year-old page documenting a function nobody has touched in five years has zero drift and is perfectly correct. Drift is also not "the page has never been reviewed." A page reviewed this morning against a file that changed this afternoon has drifted, six hours old.

And drift has a size. If the source file changed by one line in a comment, that is a different situation from 42 lines added and 7 removed across the function you documented. Both are drift. Only one is worth a rewrite.

Why most teams can't detect documentation drift

Because the link doesn't exist. In a normal docs repository, docs/button.mdx is a file. src/components/Button.tsx is a different file. Nothing in either one says they're related. The relationship lives entirely in the head of whoever wrote the page, and it leaves the company when they do.

So teams reach for proxies, and the proxies are all worse than they look:

• Git timestamps answer "when was this file last touched," which the typo fix ruins.
• Sitemap audits find pages nothing has happened to, which is decay, not wrongness. I wrote about the workflow patterns that actually move freshness — PR-blocking doc reviews, file-level ownership, runnable examples — and every one of them is real. They're also all pressure applied to humans. None of them detects anything.
• Search over the docs for the old symbol name works exactly once, for the drift you already know about.

The cost of the miss has gone up sharply. A stale page used to produce a confused developer who filed a support ticket. Now it produces an AI assistant that reads the page, believes it, and writes variant="solid" into a customer's editor with total confidence. When your documentation is the interface AI tools program against, an undetected drift is a defect with a distribution channel.

The four things drift detection needs

Exact drift detection needs four pieces, and it does not work with three:

1. A link from page to source. Explicit and stored: this page documents these files, in this repo. A file, a directory, or a glob.
2. A pinned commit. The SHA the page was generated or verified against. Without it you can tell that source changed, but not whether it changed since the page was written, which is the only question that matters.
3. A change signal. Something that tells you the source moved. A push webhook if you administer the repo, a scheduled comparison against the pinned SHA if you don't.
4. A size measurement. Lines added and removed between the pinned SHA and current head, so you can tell a rename from a comment fix and decide what deserves action.

Store those four and drift stops being a judgement call. A linked page is in one of five states — in_sync, drifted, updating, error, or unlinked — and every one is computable without a human reading anything.

{
  "docPath": "docs/button.mdx",
  "repoFullName": "acme/ui",
  "sourcePath": "src/components/Button.tsx",
  "lastSyncedSha": "9f3c1ab",
  "status": "drifted",
  "driftLines": { "added": 42, "removed": 7 }
}

That record is the entire idea. Everything else in this post is what you do with it.

What should happen when a page drifts?

Detection is the hard half. Once you have it, you have to decide what the system does, and there are exactly three options:

Flag it. Put the page in a queue with the changed commits attached. Cheapest, safest, and it degrades into a backlog nobody reads, same as every other queue.

Rewrite it and stage the change for review. The system regenerates the page from the current source and hands you a diff. You read it, publish or discard. This is the right default for almost everyone.

Rewrite it and publish. No human in the loop. Correct for high-volume reference pages generated from code in the first place; genuinely risky anywhere a person has invested prose.

The decision that ruins this feature if you get it wrong is the third one applied indiscriminately. Auto-publishing over a page a human carefully edited is the single failure mode here that destroys work rather than merely being wrong — and it does it silently, which means you find out weeks later, from the git history, when someone asks where their paragraph went.

How Agent Sync handles documentation drift in Doccupine

This is the feature I've been describing in the abstract, so here are the specifics.

Links are created for you. When the doc agent writes a page, it records the source files that page came from, pinned to the commit it read them at. You can also link by hand from the editor — "this page documents that code" — which works on hand-written pages that were never generated. For an existing site with nothing linked yet, Scan for sources proposes links across the whole docs directory. That scan is heuristic, not an LLM crawl: it costs no AI credits, pre-checks strong matches (the page explicitly references the file path) and leaves weak ones (name overlap) unchecked for you to approve.

Detection runs two ways. A push webhook on repos you administer marks links drifted the moment you commit. For repos you don't administer — webhook installation needs the admin:repo_hook scope and returns 403 without it — a background poll compares against the pinned SHA every 30 minutes. The source repositories manager labels each repo "Instant" or "~30 min" so you know which you're getting.

The publish decision is yours, at three levels. A project-wide default, a per-page override, and hand-edit protection that overrides both toward review. Hand edits are detected by sha256 of the page content, normalized for line endings, trailing whitespace, and runs of blank lines so a Prettier pass doesn't read as a human edit. When the flag fires you can keep your version or take the regeneration; either choice re-baselines, so the flag is never a one-way trap.

Review is a change set, not a mystery commit. You get a diff per file, and you can discard a whole file or a single line before publishing. Frontmatter is preserved on an update — a source change rewrites the body, it doesn't renumber or recategorize your page. Every generated page is MDX-validated before it can ship, because an invalid page fails the whole site build.

Cost has hard ceilings. Pushes aren't something the platform controls, so automatic regeneration is capped at $0.50 of AI spend per run, 15 pages per run, and 200,000 input tokens per run, billed against the AI budget on your plan rather than metered per message. Drift below one changed line never buys an LLM call. And review-mode pages don't auto-regenerate at all — that check runs first, before any API call, so a push can't spend your budget on a page you never opted in.

You get an email when pages are ready to review, or when pages were updated and published. Not on every push.

The full feature page is at self-updating docs if you want the walkthrough version.

What automatic updates don't fix

I'd rather you hear these from me than discover them.

Renames read as a broken link. A moved or renamed source file looks like a delete plus a create. The link goes to error and needs repointing by hand. This is the most common piece of manual upkeep the feature has.

One push, one run. If a single commit drifts eight pages, one regeneration starts and the other seven wait in the drift inbox for you to trigger them. Runs are serialized per project on purpose; parallel runs would race on the same staged changes.

Regeneration replaces, it doesn't merge. Ask the agent to update a page you've hand-edited and you get a rewrite from source, not a merge of your prose with the new facts. Review sits in front of it, so nothing ships unseen — but "accept" means accepting a replacement.

It only knows what's in the code. A drift-triggered rewrite fixes the signature, the default, the prop name. It cannot fix your architecture explainer, because the reason you chose that design was never in the diff. Conceptual documentation is a human job and will stay one.

Latency without a webhook is real. A freshly linked file on a repo you don't administer may not get its first poll for 30 to 60 minutes.

And the honest scoping question: if you have twelve pages documenting a surface that changes twice a year, none of this is worth configuring. Automated drift detection earns its keep on reference documentation generated from a codebase that moves weekly. Below that, a calendar reminder is a perfectly good system.

One more thing about where this runs

The free CLI generates and serves your docs site — content, search, AI chat, MCP server, the whole thing from a directory of Markdown files, self-hosted, no account. Agent Sync is not part of it, and the reason is structural rather than commercial: drift detection needs a service that's awake when you push, holding webhooks, pinned SHAs, and link state between deploys. A build-time generator has nowhere to keep any of that, so it lives in the hosted platform, which is the part that stays running.

The component library team found their eleven-week-old solid because a customer filed a bug against an example that didn't compile. The information needed to catch it existed the entire time: a commit had touched Button.tsx 42 lines' worth, and a page claimed to describe Button.tsx. Nothing in their stack held both of those facts in the same place, so nobody could ask the question.

Storing the link is the whole intervention. Everything after it is plumbing.

I'm collecting a specific number: the longest gap you've personally found between a code change and the docs catching up, and what finally surfaced it. Eleven weeks is my current record and I suspect it's not close. Email [email protected] — I want to know whether the thing that catches drift is ever a process, or whether it's always a customer.