What does robots.txt do?
Your robots.txt is a public file that asks crawlers to stay out of certain parts of your site. It is a polite sign on the door. It does not stop anyone. Your firewall is the actual lock, and it decides, request by request, whether to hand over a page at all.
What happens when they disagree?
When our free check runs, one of the first things it does is ask for your home page four times, using the names of four AI crawlers: OAI-SearchBot, GPTBot, ClaudeBot and PerplexityBot. Then it looks at what came back.
On one small health clinic's site we looked at, robots.txt welcomed every AI crawler. But when we asked for the home page by their names, each of the four got a reply of five bytes. Not a web page. A challenge.
That is the worst kind of problem, because nothing looks wrong. If you only read robots.txt, everything looks fine. And if an AI crawler is turned away, nothing else on the site can be read, however good it is.
How can you check it yourself?
Ask your own site for its home page using a crawler's name, and look at how much comes back.
curl -s -A "GPTBot" https://yourbusiness.com.au/ | wc -c
A normal home page is tens of thousands of characters. A handful means something is turning the request away.
What should you do?
- Ask your web host or developer whether bot protection challenges AI crawlers.
- Decide on purpose which crawlers to allow. Some fetch pages to answer a question. Others collect training data. They are separate decisions.
- Allow the crawlers you want, then test again by asking for your pages by name.
- Test again after any change to your hosting, your CDN or your security settings.
What does this not tell us?
- It does not tell us whether an AI would cite you if it could read you.
- Blocking some crawlers is a legitimate choice, especially ones that collect training data. The problem is blocking by accident.
- One test on one day is a snapshot. Bot protection settings change.
- This is our own observation from a live test, not a published study.
Is your website ready for AI?
Our free check reads your site the way AI tools do, tests four AI crawlers live and scores nine dimensions out of 100.
Sources
- Live crawler test, Score by Aura8 · Aura8 (our own check), 27 October 2026.
We keep what a source says separate from our reading of it. If a source is from another country or a different date, we say so. We are not giving advice, and nothing here is a promise of rankings, citations or sales.