Can AI search reach your site?

Your robots.txt is the plain-text file at yoursite.com/robots.txt that tells crawlers what they may fetch; your CDN or host is the service in front of your server, such as Cloudflare. Google names both in its advice for appearing in AI Overviews and AI Mode: "Ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure" [1].

Simple Web Audit, which we make, runs step 2 for eleven of the crawlers below; you can do every step by hand. The table follows each company's own documentation as it stood on 12 October 2026, and every row cites its source.

Which crawlers do AI companies run?

Most AI companies run separate crawlers for search, for training models, and for fetching a page when a user asks about it, and robots.txt addresses each by its own name, its user agent. Blocking a search crawler can keep your pages out of that company's answers. User fetchers often ignore robots.txt, because a person asked for the page. Google-Extended and Applebot-Extended are names only: they never visit, and exist so you can opt out of training (and, for Google, Gemini grounding).

Step 1: open the live robots.txt

Type yoursite.com/robots.txt into a browser and read what comes back: that is the file crawlers get, and it may not match the copy in your CMS or code. Each subdomain, and http and https, has its own file [13], and Anthropic asks for an opt-out on every subdomain you want covered [7].

Make sure the address returns the file, not an error. Google treats a 4xx error other than 429 as if there were no robots.txt, and after a server error it stops crawling for 12 hours, then uses the last copy it read [14]. The robots.txt standard tells crawlers to assume they may fetch nothing while the file is unreachable [15]. Of the other companies in the table, only Amazon documents its handling: it treats a file it can't fetch as absent [9].

Search Console's robots.txt report shows the file Google last fetched for each of your main hosts, with any errors [13], and Bing Webmaster Tools has a tester for Bingbot [16]. Neither tests AI crawler names, so for those, read the groups yourself.

Step 2: work out which rules each crawler follows

Each crawler obeys one group of rules: the group that names it, or, when no group does, the group for every crawler, which starts with User-agent: * [15]. Google says it never combines a named group with the * group [14].

Two crawlers borrow other rules: Applebot follows your Googlebot group when no group names Applebot [4], and Amazon's search crawler follows your rules for other search bots when you don't name it [9]. Several names can share a group, one User-agent line each, above the rules they share [15]. This robots.txt keeps AI search open and opts out of training by OpenAI, Anthropic, Google and Apple:

OAI-SearchBot, Claude-SearchBot and PerplexityBot aren't named, so they follow the * group and can fetch everything except the cart. Watch for the opposite pattern: a * group with Disallow: / beside a group that opens the site to Googlebot alone. Google can crawl, and Applebot follows the Googlebot group, but Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot are all shut out.

Simple Web Audit does this step for you: it checks Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot on each page in the audit (for the four besides Googlebot, skipping pages you have already kept out of Google Search) and flags a block for you to review. For GPTBot, ClaudeBot, Google-Extended, ChatGPT-User, Claude-User and Perplexity-User, it lists what your homepage allows without flagging it.

A robots.txt that returns 404 or 410 means no rules for every crawler in the audit. For any other 4xx error except 429, such as 403, Google treats the site as having no rules [14], and the robots.txt standard allows any crawler to do the same [15]; but of the other companies only Amazon says what its crawlers do, so the audit reports Googlebot as allowed and the rest as unknown rather than guess. It doesn't cover Apple, Google-Agent, Amazon, Meta or DuckDuckGo, or your CDN settings, which come next.

Step 3: check your CDN or host's bot settings

A CDN can turn a crawler away before robots.txt comes into it. On Cloudflare, the AI bot policies under Security Settings have separate settings for Search, Training and Agent bots (user fetchers), each set to Allow, Block, or Block on pages with ads [17]. Since 15 September 2026, blocking reaches search engines too:

For a site that wants search but not training, Cloudflare points to its new Disallow AI Training setting, which states the preference in robots.txt and blocks crawlers that collect only training data [18]. New domains get a preset based on whether the site shows ads: with ads, training is disallowed and user fetchers are blocked on ad pages; without ads, everything is allowed [18]. If you had the older "Block AI bots" switch on, Cloudflare moved it to the new settings, and it now also blocks user fetchers on pages with ads [18].

Vercel's AI Bots ruleset is off by default; set to Deny, it blocks all traffic Vercel identifies as AI bots [19]. Vercel's page doesn't list which bots that covers, so if you turn it on, watch your logs for search crawlers afterwards (step 4). On any other host, look in its security or firewall settings for anything that blocks bots or AI crawlers.

To opt a crawler out, use robots.txt rather than blocking its IP addresses:

Step 4: check which crawlers actually arrive

Your server or CDN logs show which crawlers reached you: search them for each name in the table, such as OAI-SearchBot, Claude-SearchBot, PerplexityBot and bingbot. On Cloudflare, AI Crawl Control shows which AI services visit and whether they follow your robots.txt [20]. Google-Extended and Applebot-Extended never appear, because they make no requests [3][4].

Anyone can send a crawler's name, and Google warns that user agent strings can be faked [3]. To confirm a visit is real, match its IP address against the company's published list or a reverse DNS lookup: Google [21], OpenAI [2], Anthropic [22], Bing [23], Apple [4], Perplexity [8], Amazon [9] and DuckDuckGo [10] each publish one. Anthropic's single list covers all three of its crawlers [22], and Meta's crawler page gives none [11].

Sending a request yourself under a crawler's name, for example with curl's -A option, doesn't show what the real crawler gets. Cloudflare recognizes a verified bot by a cryptographic signature, a published IP list or reverse DNS [24], none of which your request has, so your result can differ from the crawler's in either direction. That is our reading of how verification works; the Cloudflare documentation we read doesn't say what it does with such a request.

A change can take days to show in what crawlers do: OpenAI says about 24 hours for ChatGPT search [2], Amazon about 24 hours [9], Perplexity and Meta up to 24 hours [8][11], and DuckDuckGo 72 hours [10]; Google caches robots.txt for up to a day [14]. Check your logs again a few days after a change.

What about Google's AI features and Copilot?

AI Overviews and AI Mode have no crawler of their own; they cite pages Googlebot has crawled. A page has to be indexed and eligible to show with a snippet, and Google says there are no other technical requirements [1]. To keep pages out of them, use nosnippet or max-snippet on the page, or the Search Console setting Google rolled out to every site on 31 August 2026, which applies to a property: a whole site, or a folder added as its own property [1][25]. Blocking Google-Extended does not take pages out of Search [3]. The state of SEO in 2026 covers when opting out makes sense.

Copilot draws on Bing's crawling and index, so allow Bingbot. Bing's controls for Copilot are robots meta tags on the page: NOARCHIVE keeps its content out of Copilot answers, and NOCACHE limits Copilot to the URL, title and snippet [6].