The first time I saw this go wrong, it wasn't dramatic. A team had put their staging docs behind a login screen before a launch, felt appropriately careful about it, and moved on. Three weeks later someone searched for an unreleased feature name out of idle curiosity and found their own internal page sitting in Google's index, cached, with the full API surface of a product that hadn't shipped.
The login screen worked exactly as designed. It just wasn't the only way in. The crawler had indexed the site before the gate went up, and nothing about adding the gate told Google to forget what it already had. Meanwhile the site's own search endpoint was still happily returning document chunks to anyone who called it directly, because that endpoint had never been told about the password either.
That's the thing about making documentation private: the login screen is the part everyone gets right, and it's maybe a quarter of the job. A modern docs site — especially one built to be read by AI agents — is not one door. It's four. This post is about all four, how to close them, and when a shared password is the wrong tool entirely.
Three tiers of "private," and where a password actually sits
Before touching any config, it's worth being precise about what you're asking for, because "make the docs private" covers three genuinely different requirements with three different costs.
Tier 1 is obscurity. An unlisted URL, a noindex tag, a Disallow in robots.txt. Nothing is enforced — anyone with the link gets in, and robots.txt is a polite request that well-behaved crawlers honor and badly-behaved ones read as a directory listing of what you'd rather hide. This is fine for a staging site nobody knows about and worthless for anything you'd be upset to see leaked.
Tier 2 is a shared password. One secret, everyone who has it gets in, no accounts and no database. This is the sweet spot for private previews, client-facing docs during an engagement, and internal docs where the threat model is "don't let this show up in a search result," not "defend against a determined attacker." It's genuinely enforced, and it takes about a minute to set up.
Tier 3 is per-user identity. SSO, individual accounts, revoking one person's access without disrupting anyone else, and an audit trail of who read what. This is what compliance means when it says "access control." It's also an order of magnitude more work, and most teams asking for it actually need tier 2.
The honest guidance: if you can't name the specific person whose access you'd need to revoke individually, you want tier 2. Reach for tier 3 when you can, and don't let anyone sell you tier 3 complexity for a tier 2 problem.
The four surfaces you actually have to close
Here's the part that gets missed, and it's the reason the story at the top of this post happened. When you gate a documentation site, these are the four independent surfaces that serve your content:
1. The pages themselves. The obvious one. Every route renders a login screen instead of your documentation. If your gate runs in a layout or a page component rather than in middleware, check it carefully — anything that bypasses that component bypasses your gate.
2. The content APIs. This is the one that bites. A docs site with search and an AI assistant has endpoints — typically something like /api/search and /api/rag — that return your documentation as structured data. They are not pages. They do not render your login screen. Unless they were explicitly taught about the password, they will keep answering, and someone can reconstruct a substantial portion of your docs by calling search with common terms in a loop. A gated site with an open search endpoint is not gated.
3. Search engines and crawlers. Three separate mechanisms, and you want all three: robots.txt disallowing everything, a noindex, nofollow meta tag on every page, and an X-Robots-Tag: noindex, nofollow response header. The header matters because it applies to non-HTML responses too, and because it can't be missed by a crawler that never parsed your markup.
4. The machine-readable layer. This is new, and it's the one almost nobody thinks about. If your docs site is AI-native, it also publishes llms.txt, llms-full.txt, per-page Markdown mirrors, and quite possibly an MCP server that AI coding tools query directly. Every one of those is a full-fidelity copy of your documentation in a format designed to be easy to consume in bulk. Gating your HTML while leaving llms-full.txt public is a rounding error away from publishing a text file of everything you were trying to hide.
Surface 4 is the one worth auditing by hand. After you turn on any gate, open a private browsing window and request /llms-full.txt, your MCP endpoint, and one page's raw Markdown. If any of them returns content, the gate is incomplete.
Doing it yourself with the CLI
If you're running the open-source generator, this is one environment variable:
SITE_PASSWORD=choose-a-strong-shared-passwordThat's the whole configuration. Everyone who visits shares that one password. When the variable is unset or empty, the site is fully public and behaves exactly as it did before — which also means removing it is how you turn protection off.
Setting it closes surfaces 1 through 3 together: pages render a login screen, /api/rag and /api/search return 401 Unauthorized without a valid session, and the site advertises itself as unindexable through robots.txt, the meta tag, and the X-Robots-Tag header simultaneously.
Surface 4 is deliberately separate. The MCP server has its own authentication through a DOCS_API_KEY bearer token and is not affected by SITE_PASSWORD. That's the right design — agents authenticate differently from humans — but it means you set both when you want the docs and the agent endpoint locked down. Setting only one is the most common mistake here.
Because the check reads the variable at request time, flipping protection on or off is an environment change and a redeploy. Your content doesn't rebuild.
How the gate works, and why those details matter
I want to walk through the mechanics, partly because "we set a cookie" covers a wide range of implementations from fine to alarming, and you should know which one you're running.
When a visitor submits the correct password, the site sets an httpOnly cookie that unlocks it for 30 days. The cookie never contains the password. It holds an HMAC-SHA256 derived from it, so a stolen cookie doesn't hand over the secret itself, and reading the cookie tells an attacker nothing they can type into the login form.
That comparison runs in constant time. A naive string comparison returns faster on a wrong first character than a wrong last character, and that timing difference is enough to recover a token byte by byte given enough requests. Constant-time comparison closes it.
Password attempts are rate limited by IP, which is what stands between a shared password and an afternoon of automated guessing. The cookie is marked secure in production, so it only travels over HTTPS.
And rotation works the way you'd want: changing SITE_PASSWORD invalidates every existing session immediately, because every previously issued cookie was derived from the old value. When a contractor's engagement ends, you rotate, and everyone re-enters the new password once.
Two things this design does not give you, stated plainly: no record of who viewed what, and no way to revoke one person without rotating for everyone. Those are tier 3 properties, and no shared-password scheme has them.
When a shared password is the wrong tool
Skip this approach when:
- You need to know who read what. Regulated environments, security reviews, anything where "we can't tell you who accessed it" is an unacceptable answer.
- Access is per-customer and overlapping. If customer A must see their docs and not customer B's, you need per-user authorization, not one shared secret with two audiences.
- Revocation has to be surgical. High-turnover contractor access where rotating for everybody each time someone leaves is genuinely disruptive.
- The content is the product. If your docs are a paid deliverable, gate them at your identity provider alongside the rest of your product, not at the docs layer.
In each of those cases, put the site behind your own identity provider or your host's access controls. A shared password isn't a weaker version of that — it's a different tool for a different problem, and using it as a substitute is how teams end up believing they have access control when they have a speed bump.
The managed version
If you'd rather not wire environment variables by hand, the hosted platform ships the same gate as a toggle in the project's Authentication settings. You set a password, it's stored as an encrypted environment variable, and the platform prompts the redeploy that activates it. Same three-layer enforcement, same 30-day sessions, same DOCS_API_KEY caveat for the MCP endpoint.
The distinction is only ever about who runs the deploy pipeline. The gate itself is in the open-source generator, which means you're never locked into the hosted version to keep your docs private — self-host the identical setup for free, and move between the two whenever it suits you.
The short version
Making documentation private is four jobs, not one:
- Decide your tier. Obscurity, shared password, or per-user identity. Most teams need the middle one.
- Gate the pages in middleware, not in a layout component.
- Gate the content APIs. Search and chat endpoints return
401, or your gate is decorative. - Tell the crawlers three ways.
robots.txt, thenoindexmeta tag, and theX-Robots-Tagheader. - Close the machine-readable layer.
llms.txt,llms-full.txt, raw Markdown, and the MCP endpoint — the last one usually needs its own key. - Verify from a private window. Request all of the above while logged out. Anything that returns content is a hole.
- Remember what was already indexed. A gate stops new crawling. Removing content Google already has is a separate request through Search Console.
That last one is what turned a careful team's staging site into a search result. The gate was correct. It just went up second.
If you've hit a surface I didn't list here — some other endpoint or artifact that kept serving content after a gate went up — I'd really like to hear about it. Email [email protected]. The list of doors keeps growing as docs sites get more machine-readable, and the ones that catch people are never the front one.