About the Perceivable crawler
What our crawler does, how to identify it, and how to control it.
Identification
Our crawler requests pages with a user agent containing Perceivable/1.0 and a link back
to this page:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/151.0.0.0 Safari/537.36 Perceivable/1.0 (+https://perceivable.eu/bot)What it does
- Requests public pages over HTTP or HTTPS and renders them, exactly as a browser would.
- Runs accessibility rules against the rendered page in memory.
- Follows same-origin links to discover further pages, up to the audit's page limit.
What it does not do
- Submit forms, create accounts, or attempt to log in.
- Follow links to other domains.
- Request media or font files — these are blocked to reduce load on your server.
- Store page content beyond the markup of elements that failed a rule.
Politeness
One page at a time per site, with a pause of at least 250 ms between requests, and longer if your
robots.txt sets a Crawl-delay. A free scan is a single page. The largest
audit is 100 pages, spread over several minutes.
Controlling it
We honour robots.txt. To exclude us specifically while allowing other crawlers:
User-agent: perceivable
Disallow: /To exclude a section from everyone including us:
User-agent: *
Disallow: /private/If our crawler is causing a problem, email hello@perceivable.eu and we will block the domain outright.
If we are being blocked
Bot-protection services sometimes challenge our crawler. If a scan of your own site fails with a loading error, allowing the user agent above will let it through. We do not attempt to evade bot protection.