Insights · Trend · SEO
GPTBot in robots.txt: Content protection or digital suicide?
Traditional SEO is no longer enough in the age of Answer Engines that provide direct responses to users. Blocking crawlers like GPTBot or PerplexityBot isn't just about content protection; it's a risk of disappearing from the AI radar entirely. Here is what is changing and what you can do better.
What's changing right now
LLMs are becoming the new entry point for users seeking specific solutions. If you block AI bots, your brand won't appear in citations or sources generated by ChatGPT or Claude. The battle for organic traffic is expanding into a battle for presence within the memory of large language models.
What you can do better
Concretely, starting today
- Don't block all AI bots by default; evaluate the referral traffic value from platforms like Perplexity.
- In robots.txt separate the search-side bots (OAI-SearchBot, ChatGPT-User, PerplexityBot) from the training crawlers (GPTBot, ClaudeBot, Google-Extended) — only the former sends people to your site.
- Monitor how your brand is described by LLMs instead of just tracking Google rankings.
- Treat AI bots as an opportunity for free distribution of expert knowledge within the AI ecosystem.
Proof from our work
At DIGITAL DTL our robots.txt explicitly welcomes 11 AI crawlers — both search-side and training — alongside an llms.txt, a site map written for models. We also built our own visibility checker: it queries Gemini for a set of phrases and shows whether the brand comes up in the answer at all. That is one model, not the whole market — but even that is enough to see the gap between “we are in the index” and “we are in the answer”.
Frequently asked questions
Does blocking GPTBot affect Google rankings?
Directly, no – Googlebot is a separate crawler. However, if your content is missing from AI models, users utilizing AI Search may never find your site.
Which AI bots is it worth allowing?
First separate two groups, because they do different things. Citations and referral traffic come from the search-side bots: OAI-SearchBot (the ChatGPT index), ChatGPT-User and Claude-User (fetching a page on a user’s request) and PerplexityBot. Training crawlers — GPTBot, ClaudeBot, Google-Extended — are a separate group: they feed the models but send nobody to your site. If you care about showing up in answers with citations, allow the first group first; consenting to training is a separate decision you may settle differently.
Check your AI visibility
Don't let your company stay invisible to ChatGPT and Claude users. We can help you configure robots.txt to protect data while building reach.
