I’ll make three arguments for leaving your website completely unblocked for AI bots:
-
Acquisition of current traffic
-
Brand display today
-
Future developments in LLM-based tools
Acquisition of current traffic
I recently saw Wil Reynolds speak at SMX Munich, and in his talk, he made a strong case for ChatGPT (or similar tools) as a significant acquisition channel today or in the near future. He outlines a similar topic in this blog post, along with a methodology for getting an idea of how much your business might be affected. I recommend you take a look. (He will also speak at MozCon in June.)
This will definitely vary from company to company. That said, my current experience is that:
I’ve argued elsewhere that I don’t think generative AI is an identical replacement for search. It’s a different tool with different uses. But you should evaluate it for your business.
In the case of “Google-Extended” we also need to consider whether we believe this impacts Google Search as an acquisition channel. For now, Google says no, a claim some people are understandably skeptical about. In any case, the situation could change quickly if and when Google introduces generative AI search capabilities.
Brand display today
Additionally, at SMX Munich, I saw Rand Fishkin (also speaking at MozCon in June) argue that digital marketers are too focused on bottom-of-the-funnel attribution, which is increasingly difficult, and should instead take a leaf out of the book of pre-web marketers who evaluated impressions, number of visitors, and similar “vanity metrics.” I agree! In fact, I wrote about it in 2019. That post has since fallen victim to a somewhat questionable site migration, but I’ve reuploaded it here.
On a similar basis, maybe I shouldn’t only worry about whether ChatGPT (or even other outputs from LLM models, such as AI-written content) drive traffic to me, but also simply whether they mention my brand and products. Preferably, mentioning it the same way I would.
If I block these models from accessing the pages where I talk about my products, and if I also agree with the above argument that blocking access significantly affects the content that models can import, then I will make them less likely to be mentioned accurately, or even at all.
This could have a particular impact if I launch a new product or rebrand: anything new will only be acquired through external sources, which could be less than positive or, again, inaccurate.
Future developments in LLM-based tools
What if we accepted that current generative AI-based tools are not the primary acquisition channels? It will always be like this? What happens if I block GPTBot now and then, in a year or two, OpenAI launches a search engine based on the index it created?
Perhaps at that point a rapid reversal could be made. Will it be fast enough? Often these models aren’t quite at Google’s level when it comes to quickly importing new content. Presumably, though, to be a competitive search engine, they should be? Or would they use Bing’s index and crawler? It could also be argued that these models could use the content itself (originality?) as a signal of authority, as opposed to (for example) links, user signals or brand search volume. Personally, I find this impractical and, as such, unlikely, but it’s all a big unknown, and that’s the point: uncertainty itself is not an attractive proposition.
Beyond that, a search engine is only one (most likely) possibility: a few years ago we wouldn’t have imagined that ChatGPT would have such a big impact.