TIL: Cloudflare was prepending Disallow rules above my robots.txt
I authored a robots.txt that explicitly invited AI crawlers. Production served a Cloudflare-managed block above it that disallowed eight of them. The repo never showed it, no API can turn it off, and RFC 9309 group-merging is the only thing standing between your allowlist and invisibility.
I hand-wrote a careful robots.txt for a production site of mine that explicitly allowed GPTBot, ClaudeBot, Google-Extended, CCBot and friends. Inviting AI crawlers was the entire point. The repo said allow, the deploy was green, and production was serving something else: Cloudflare had quietly stapled its own block on top of mine.
What Cloudflare actually does
Cloudflare's managed robots.txt setting does not replace your file, and it does not warn you. It prepends a managed block above your file and serves the concatenation. This is documented behavior, in Cloudflare's own words: "Cloudflare will prepend our managed robots.txt before your existing robots.txt, combining both into a single response." Nothing in your repo, your build output, or your deploy logs reflects it.
The managed block disallows eight crawlers outright:
# BEGIN Cloudflare Managed content
User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
# ...plus Amazonbot, Applebot-Extended, Bytespider, meta-externalagent
# END Cloudflare Managed Content
# ...your file starts here, saying Allow: /
So the served file contains two groups for the same user-agent saying opposite things. Which brings up an RFC corner I had never needed before.
The RFC 9309 nuance
My first instinct was "first matching group wins, so the Disallow kills my allowlist." That is not what RFC 9309 says. Section 2.2.1: if more than one group matches a user-agent, the groups MUST be merged into one. Section 2.2.2: the most specific rule wins, and if an allow and a disallow are equivalent, "the allow SHOULD be used."
Disallow: / and Allow: / are the same one-octet path, so they are equivalent. A spec-perfect parser merges the groups and lets the allow win. My allowlist should survive.
Should. That is a SHOULD, not a MUST, sitting behind a group-merging step plenty of real-world parsers skip. Empirically the bet does not pay: the AEO scanner I use scored crawler access 58/100 with the managed block on, and 100/100 the moment I turned it off. Its parser did not merge the groups. If a purpose-built AEO tool reads my site as blocked, I am not going to assume GPTBot reads it as allowed.
No API can turn it off
I went looking for an endpoint to flip the setting. There is none. The zone settings endpoint returns 9109 Unauthorized for it, and the AI-audit robots endpoints are read-only; even the POST .../ai-audit/robots/bulk is a bulk read. It is a dashboard toggle or nothing.
Two traps inside the dashboard itself:
- Two different settings both sound like "block AI crawlers." The per-crawler allow/block actions in AI Crawl Control's Crawlers tab are not the managed robots.txt toggle. Mine had every crawler set to allow in one tab while the other was emitting
Disallow: /for the same crawlers. - The docs and the dashboard disagree on where it lives. The docs point at Security Settings → Bot traffic. I found mine under AI Crawl Control → Signals → "Managed robots.txt". Check both.
The one-command lesson
diff <(curl -s https://yourdomain.com/robots.txt) public/robots.txt
Your repo is not the served artifact. Anything the edge can inject, headers, redirects, robots.txt, needs a production assertion, not a local one. I had authored the allowlist, reviewed it, and deployed it, and it was dead on arrival at the edge.