GEO · content access
GEO: bot protection blocks AI search crawlers on every page
- Where we checked
- https://example.com/ and 40 checked pages (categories, products, contact).
- What we tested
- Requests with the GPTBot, ClaudeBot and PerplexityBot user agents, compared with a regular browser request.
- Finding
- The browser gets the content. All three AI crawler user agents get a 403 error on all 40 pages. robots.txt does not mention them; the block comes from the hosting provider's bot protection.
- Evidence
Browser: HTTP 200, page content GPTBot: HTTP 403 (40 of 40) ClaudeBot: HTTP 403 (40 of 40) PerplexityBot: HTTP 403 (40 of 40)
- What it means
- AI search tools cannot read the content, so their answers cannot cite the offer, prices or contact details from this site. The block can be switched off in the host's protection settings. The finding applies to test requests with these user agents on the day of the audit.