Skip to content
FreeWhere is your business recognised? Try the Entity Check

← All glossary terms

GlossaryCrawlability

Crawlability

A page is crawlable when a crawler can find it (through links or a sitemap), is permitted to fetch it (robots.txt and firewall rules), receives it successfully (no errors or blocks) and can read its content (ideally in the initial HTML).

Example

User-agent: OAI-SearchBot
Disallow: /

This robots.txt rule makes the whole site uncrawlable for that crawler, however good the pages are.

Why it matters

A page that cannot be crawled cannot be used by that search or answer system. Crawlability is a prerequisite, not a guarantee of indexing, ranking or citation.

Check crawler access with the AEO Audit · See crawler activity