Check if your robots.txt blocks AI crawlers

Your robots.txt file quietly controls which bots can access your site. Many sites ship with defaults that block AI crawlers like GPTBot, keeping them out of AI answers without you realising. Here’s how to check if your robots.txt blocks AI crawlers and what to do about it.

Is your robots.txt blocking AI crawlers?

Open your site’s robots.txt file by visiting yoursite.com/robots.txt. Look for lines that include GPTBot, CCBot (Common Crawl), or Google-Extended. If you see Disallow: / or specific paths under these user agents, those bots are blocked. If the file doesn’t mention them at all, they’re allowed by default.

Should you allow AI bots?

Blocking AI crawlers means your content won’t appear in AI-generated answers or tools. Some owners prefer this to avoid third-party use of their content. Others want the visibility and traffic AI answers can bring. There’s no universal answer, but consider:

• Visibility: Allowing bots means your content may appear in AI search results or chatbots.
• Control: Blocking bots keeps your content out of AI training data, but also out of helpful answers.
• Traffic: AI answers can drive traffic if they link back (though not all do).

If you monetise content or rely on direct visits, blocking may make sense. If you want broader reach, allowing bots is likely better.

How to change your robots.txt

To block or allow AI bots, edit your robots.txt file. Most hosting providers or CMS platforms let you edit this directly. Examples:

• Block GPTBot:

  User-agent: GPTBot
  Disallow: /

• Allow GPTBot but block specific paths:

  User-agent: GPTBot
  Disallow: /private/

• Allow all AI bots (default if not mentioned): Simply omit any AI bot rules.

After saving, test with tools like Google’s robots.txt tester or by revisiting the file in your browser. Changes take effect immediately but may take time for bots to respect them.

For more on how AI visibility works, see our guide. If you’re unsure, start by allowing bots and monitor traffic or attribution. You can always adjust later.

Want help checking your site’s AI readiness? Ask us.