A blocked crawler cannot cite you
When someone asks an AI tool who solves their problem, it answers from pages it was allowed to read. If yours was off limits, a competitor's page fills the slot.
All tools
Free tool
Can ChatGPT read your site?
Most sites never checked. Enter a URL and we read your robots.txt the way each AI crawler does, then tell you which of 10 platforms are allowed through and which rule is turning them away.
Why a blocked crawler costs you
The block is usually an accident
Most blocks we find were copied from a template robots.txt or added by a plugin default. Very few were a decision anyone made on purpose.
Permission is not the same as readable
Allowing every crawler achieves nothing if the page builds itself in the browser. Crawlers that skip JavaScript see an empty shell, so the check tests both.
Questions about AI crawlers
How do I know if ChatGPT is blocked from my site?+
Enter your URL above. The check reads your robots.txt and reports each OpenAI agent separately: GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for live fetches during a conversation. Blocking any one of them costs you a different thing.
Which AI crawlers does this check?+
10 platforms: ChatGPT, Claude, Perplexity, Gemini and AI Overviews, Microsoft Copilot, Apple Intelligence, Common Crawl, Meta AI, Amazon, and ByteDance. Each is tested by its own user-agent strings against the path you entered.
Should I allow AI crawlers?+
It depends on your business. If you sell something and want to appear in AI answers, blocking these crawlers removes you from that surface entirely. If your revenue comes from people landing on your pages to read the content itself, blocking training crawlers while allowing search ones is a defensible middle path.
What is the difference between GPTBot and OAI-SearchBot?+
GPTBot collects pages to train the model. OAI-SearchBot builds the index ChatGPT uses when it searches the web. Blocking GPTBot keeps you out of training data but leaves you citable in search. Blocking OAI-SearchBot removes you from ChatGPT's answers.
Does Google-Extended affect my normal Google rankings?+
No. Google-Extended only controls whether Gemini and AI Overviews may use your content. Blocking it leaves ordinary Googlebot crawling and ranking untouched.
My robots.txt allows everything. Why does the check still warn me?+
Permission is only half of it. Most AI crawlers do not run JavaScript, so a page that builds its content in the browser reads as blank to them. The check counts the words in the HTML your server returns and flags the page when there is almost nothing there.
What is llms.txt?+
A proposed file at your domain root that lists your key pages and what each one covers, so a model can find the important material quickly. Adoption is still early and it is not a ranking factor, so treat it as cheap insurance rather than a priority.