Charlotte, one of the main characters in E. B. White’s well known children’s book Charlotte’s Web, was a spider with a gift for getting noticed. The internet runs on spiders too, called “crawlers”.
A crawler is a computer program that visits websites and reads them automatically, page by page. Think of it as a very fast librarian moving through the shelves of the internet, finding and cataloguing information.
Companies like Google, OpenAI (ChatGPT) and Anthropic (Claude) all have crawlers which use website content to answer people’s questions and to train their own AI tools.
Last week, while reviewing a website, I found that two of the internet’s librarians were being turned away at the gate. OpenAI’s and Anthropic’s crawlers were halted with the message “You have been blocked”, whilst the red carpet was being rolled out for Google’s crawler.
What had caused this was a setting in the security service protecting the website. Security services such as Cloudflare, Akamai and Amazon CloudFront work quietly in the background doing their job, but I’ve noticed that some services now block AI crawlers by default unless some fabulous AI visibility wizard like me changes the setting.
Checking the gate to a website is essential. If it stays shut, millions of people using ChatGPT and Claude are far less likely to see the company behind the gate in their AI search results.
This does not mean the company will disappear from every AI answer. Information may still be available elsewhere. But it could be incomplete, out of date or wrong, and an AI tool may point someone towards another provider instead.
The fix may be a single setting. The hard part is knowing that it is there.
So here is one simple question to ask whoever looks after your website: “Which AI crawlers are blocked from reading our website?”
Caroline Langley is a barrister and HKMAAL accredited mediator at Justice Chambers, Admiralty, with more than thirty years at the Bar across four jurisdictions, and the founder of Just Law Education. Read more about Caroline.
See SEO, AEO and GEO services.
- AI Visitors Are Not The Same As Human Visitors
- Consent Is Not a Signature: Why Mediation Matters in Hong Kong
- "Informed Refusal" or "Non-Compliance"? Who Gets to Decide?
- "It's Just the Baby Blues": How the DSM Failed Lindsay Clancy Before a Jury Ever Could
- A Search Result Is a Starting Point, Not a Conclusion
This article first appeared on LinkedIn.