--- title: "Law Firm AI Crawler Policy: The robots.txt Citation Trap - Carlos Arias" description: "Your law firm AI crawler policy was probably set by default, not by decision. What GPTBot and ClaudeBot rules actually do to your firm's visibility." url: "https://carlosarias.com/blog/law-firm-marketing/law-firm-ai-crawler-robots-txt" --- [Law Firm Marketing](/blog/categories/law-firm-marketing) # Law Firm AI Crawler Policy: The robots.txt Citation Trap Your law firm AI crawler policy was probably set by default, not by decision. What GPTBot and ClaudeBot rules actually do to your firm's visibility. [Carlos Arias](/blog/authors/carlos-arias) · September 22, 2026 · 5 min read ![A black ink-brush torii gate above pale grey wash clouds on cream paper, with a small red seal.](/_astro/cover.EXi_jEkO_9lpDI.webp) *A black ink-brush torii gate above pale grey wash clouds on cream paper, with a small red seal. AI-generated illustration by Carlos Arias .* Open yourfirm.com/robots.txt. If nobody at your firm has looked at it since the site launched, you don’t have a law firm AI crawler policy. You have whatever an agency pasted in three years ago, plus whatever your CDN decided last month. Two authors, one file. They answer the same question and they may not agree. Which machines can read your practice pages, and which of them are allowed to cite you. Principals find out sideways. Rankings look fine. Intake looks fine. Then a prospect mentions they asked ChatGPT for an estate-planning attorney in your city and heard three other names. Legal has less runway than most industries. Martindale-Avvo’s 2026 State of the Legal Consumer report put AI Overview trigger rates for legal queries near 78%. That was the highest of any category they measured. One vendor’s sample, so treat it as a direction, not a constant. ## Blocking a scraper and blocking a citation source are different acts The industry is sloppy about this distinction and the sloppiness is expensive. A crawler that collects text to train a model and a crawler that fetches your page to answer a live question are not the same program. They don’t share a user-agent string. They don’t need the same rule. ### One vendor, three bots, three jobs OpenAI runs three of them. GPTBot collects training data. OAI-SearchBot builds the index ChatGPT search cites from. ChatGPT-User fetches a page when a user’s question triggers it. Anthropic splits the same way. ClaudeBot trains. Claude-SearchBot indexes for search. Claude-User fetches when a person’s question points at your page. ### The accidental opt-out So Disallow on GPTBot is a coherent position for a law firm site: decline to be training data, stay retrievable. Disallow OAI-SearchBot and you have done something else. You have asked to be left out of the answer. Firms do the second one by accident constantly. A September 2026 sweep of the Tranco top 5,000 found 535 sites disallowing GPTBot, and 238 of them, 44.5%, disallowing OAI-SearchBot alongside it. The search bot went out with the training bot. Nobody chose that. Somebody copied a blocklist off a forum. ## Cloudflare now writes the file, and the defaults moved on 15 September This is the part that makes an old, unread robots.txt genuinely dangerous: you are no longer the only author of it. ### Bot Preference Sync writes the directives Cloudflare published Bot Preference Sync on 21 August 2026. It generates robots.txt directives from the AI bot policy already set in the dashboard and prepends them to your existing file between # BEGIN Cloudflare Bot Preference Sync and # END markers, leaving your own rules underneath. Every plan gets it, free to Enterprise. The dashboard reduces the whole question to three categories, each with the same three options: allow, block everywhere, or block only on pages that display ads. - Search. Crawlers that index you so you can be found and cited. - Agent. Crawlers acting on a specific user’s request, mid-task. - Training. Crawlers feeding model weights, with no attribution back to you. ### The new-domain defaults flipped Then the defaults changed. On 15 September 2026, Cloudflare set Training and Agent to blocked on ad-bearing pages for new domains, with Search left allowed. That reaches new customers and any new site added on an existing account. It reaches every existing free customer too. And Cloudflare has been blocking AI crawlers by default for newly onboarded domains since 1 July 2025, so the account may already have been answering this question for a year before anyone asked. Most firm sites carry no ads, so the ad-page clause is usually inert. The part that isn’t inert is simpler and worse: somebody, at some point, clicked through an onboarding screen about AI bots on your account, and you have no idea what they picked. ## What “disallow training” is actually asking for Cloudflare’s newer position is the interesting one, and I think it’s defensible. A crawler that does both search and training can’t just assert good faith. To keep crawling sites that disallow training, Cloudflare requires the operator to: - Respect a no-training preference. - Give site owners a route out of AI summaries. - Report at URL level which pages were made available for training. - Publicly demonstrate that disallowing training doesn’t degrade search results. Miss those and the crawler gets treated as opaque and blocked. Read that as what it is. It’s one company’s enforcement policy at the edge, not a standard. The standards work is at the IETF, where the AIPREF working group is trying to give these preferences a real grammar. Cloudflare’s Content Signals Policy, launched 24 September 2025, is a robots.txt extension pointed the same way. Useful, early, not settled. ## The crawlers that were never going to read your file Here is the honest limit. Robots.txt is a request. It constrains the companies that have decided to be constrained, and nothing else. Bytespider, ByteDance’s crawler, is the standing example. Its own documentation claims it follows the exclusion protocol. Fortune reported on 3 October 2024 that it ignores robots.txt while scraping about 25 times faster than GPTBot, and independent operators keep logging requests to disallowed paths. Its reach swings hard. In May 2024 Bytespider was 42% of all AI crawler requests Cloudflare saw, touching more than 40% of the properties Cloudflare protects. A year later it was 7%. Through all of that it stayed one of the crawlers least often named in robots.txt at all. The same report found it in under 5% of the files checked on top domains. So a firm’s rules do two different jobs and need two different mechanisms. Preferences for the crawlers that honor preferences, which is robots.txt. Enforcement for the ones that don’t, which is a WAF or bot-management rule at the edge. If your vendor’s answer to scraping is a line in a text file, that answer is decorative. ## What I’d check in your law firm AI crawler policy Four things, in order, and none of them require a vendor. Load yourfirm.com/robots.txt in a browser. Search it for GPTBot, OAI-SearchBot, ClaudeBot and Claude-SearchBot. If the training bots and the search bots are being treated identically, somebody made a decision for you and it wasn’t a decision. Look for those # BEGIN Cloudflare Bot Preference Sync markers. If they’re there, the file you’re reading is being written upstream, and editing it by hand will not hold. Then open your server logs and grep 90 days for OAI-SearchBot and Claude-SearchBot. Zero hits against a permissive robots.txt means something in front of your origin is refusing them, and that thing is usually a bot-management default nobody documented. The wider diagnostic sequence, retrieval before citation, is laid out in the four retrieval pipeline tests. Last, find out who actually controls the CDN account. On most small-firm sites it’s a hosting reseller or the agency the principal already fired. None of this makes a model name your firm. That’s not a promise anyone can make honestly. What it does is stop you from being excluded by a file you’ve never read, which is a smaller claim and a real one. Fix access first, then architecture. If the raw HTML is empty when the crawler arrives, permission was never the bottleneck, and that problem is structural. Open the file. If what you find doesn’t match what you were told you were paying for, write us at hi@carlosarias.com and tell us what the lines say. Tags [#Law Firm SEO](/tags/law-firm-seo/)[#AI Search](/tags/ai-search/) #Robots Txt [#Technical SEO](/tags/technical-seo/)[#Local SEO](/tags/localseo/) Share Written by [Carlos Arias](/blog/authors/carlos-arias) Marketing Engineer for law firms. I combine digital marketing, software, data, automation and AI to improve the whole system — from first click to signed case. On this page - Blocking a scraper and blocking a citation source are different acts - One vendor, three bots, three jobs - The accidental opt-out - Cloudflare now writes the file, and the defaults moved on 15 September - Bot Preference Sync writes the directives - The new-domain defaults flipped - What “disallow training” is actually asking for - The crawlers that were never going to read your file - What I’d check in your law firm AI crawler policy ## Continue reading [Law Firm Marketing](/blog/categories/law-firm-marketing) · September 21, 2026 [### Law Firm AI Search Indexing: Four Retrieval Pipeline Tests](/blog/law-firm-marketing/law-firm-ai-retrieval-pipeline/) Law firm AI search indexing, tested in an afternoon. Four checks that tell you whether ChatGPT and Claude can retrieve your practice pages at all. Carlos Arias · 7 min [Law Firm Marketing](/blog/categories/law-firm-marketing) · September 20, 2026 [### Measuring AI Search Visibility for Law Firms: Your Report Still Leads With Rankings](/blog/law-firm-marketing/ai-search-measurement-law-firm-marketing/) Measuring AI search visibility for law firms starts where the ranking report stops. What to track, what to drop, and the one number nobody can attribute. Carlos Arias · 7 min [Law Firm Marketing](/blog/categories/law-firm-marketing) · September 18, 2026 [### Law Firm Website AI Visibility: It's an Architecture Problem](/blog/law-firm-marketing/machine-first-architecture-law-firm-website/) Law firm website AI visibility is an architecture problem, not a content problem. The four pillars, what breaks on a practice-area page, and the fix. Carlos Arias · 8 min ## Stay in the loop. One email when it’s worth it — new posts and updates, no spam. Thanks — check your inbox to confirm. Free. Unsubscribe in one click.