Free test — 30 seconds, no sign-up
Can AI agents see your site?
We query your page live using the user agents of GPTBot, ClaudeBot, PerplexityBot and Google, then show you exactly what they receive. Not an estimate: your server's actual response.
- Reading robots.txt
- Querying with each AI crawler
- Rendering the page without JavaScript
- Looking for structured data
Accessibility score
Your server's response, per crawler
What the agent can read
This test detects blocks based on user agent and robots.txt. IP-range filtering — common with Datadome or Cloudflare — can only be observed in your own server logs. That is what the full audit is for.
Want to know what this test cannot see?
The full audit opens your server logs and WAF rules, template by template, and puts a cost on every fix.
Questions
What people ask before they sign
Isn't allowing AI crawlers just letting them scrape my content?
It is the most common objection, and a fair one. But blocking means disappearing: when a customer asks an agent where to buy something, only readable merchants get named. The right answer is neither "block everything" nor "open everything" — it is to choose: open the category and product pages that make you money, and close what has no business being there. That trade-off is decided rule by rule, and it is exactly what we work through.
How do I find out whether my bot protection already blocks GPTBot?
One hour of reading server logs. Filter by user agent over the last thirty days and count the 403 and 429 responses. In most cases we have seen, the block had been in place for months without a single alert firing: bot protection does its job, it does not ask whether the decision is good for the business. The free test at the top of this page already gives you a first signal.
What is an llms.txt file?
A Markdown text file at the root of your site describing your business and main pages for language models. It replaces neither robots.txt nor your sitemap: it complements them by giving context where the others only give permissions and addresses. It takes about an hour to put in place. Ours is at /llms.txt.
My site is a React or Vue SPA. Is that a problem?
It is the second most common problem. Agent crawlers do not execute JavaScript: where your browser shows a complete product page, they receive an empty container. You do not need to rebuild everything — server-side rendering or pre-generating just the category and product templates is usually enough to restore readability.
Which platforms do you work with?
This plays out in infrastructure, not in the CMS: reverse proxy, CDN, WAF rules, rendering layer. So we work platform-independently — Magento, Salesforce Commerce Cloud, PrestaShop, Shopify Plus, bespoke builds. What matters is your delivery chain, and we know how to read it.
How long before I see a result?
The diagnostic takes five to ten days and gives you a costed picture immediately. Unblocking work shows an effect within days — the time it takes crawlers to come back. Server-side rendering and structured data take longer, typically two to eight weeks depending on your deployment pipeline.