Website Health Check
Find out which search and AI crawlers your site lets in. We read your robots.txt for 11 crawlers and test whether your firewall turns AI crawlers away, as part of a free full health check of your site.
about a minutepart of the full health check
01What we check
Googlebot, Bingbot and OAI-SearchBot (ChatGPT search). Blocking them keeps your pages out of search results and AI answers, so it fails the check.
PerplexityBot, Claude-SearchBot and ChatGPT-User, which fetch pages when someone asks an AI tool a question. Blocking them is a warning.
GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot collect data for training AI models. Blocking them is your choice: we list them and they cost no points.
We ask for your home page under the names of OAI-SearchBot and PerplexityBot. If your firewall or bot protection refuses those requests, it may turn the real crawlers away too.
That is your choice. GPTBot collects training data; blocking it does not keep you out of ChatGPT search, which uses OAI-SearchBot. Our check scores the search and answer crawlers, not the training ones.
Firewalls and bot protection (for example on Cloudflare) can refuse requests by name before robots.txt matters. That is why we also test with the crawlers' names.
No. We send the requests from our own server, so a firewall that checks addresses may treat the real crawlers differently. The report says so when it matters.
03More free checks