Your bot protection probably blocks GPTBot. How to check in an hour

Why the block is almost always accidental

No e-commerce director ever decided to block GPTBot. What happened is more ordinary: a bot protection product was deployed to contain price scraping and credential stuffing, with a default policy that refuses any agent not identified as a browser. That policy works. It does exactly what it was bought to do.

The problem is that a new category of robot has appeared in the meantime, one whose visit carries positive commercial value. The rule has not moved. It treats GPTBot/1.1 the way it treats a scraper: a non-human client, therefore suspect.

Nobody is alerted, because bot protection that blocks a robot does not report an incident — it reports a success. The block shows up in dashboards as a neutralised threat.

The user agents that matter

Before you go looking, know what to look for. The strings that count today:

• GPTBot — OpenAI indexing for training and search
• OAI-SearchBot — the ChatGPT search surface
• ChatGPT-User — a visit triggered live by a user
• ClaudeBot, Claude-User, Claude-SearchBot — Anthropic
• PerplexityBot, Perplexity-User — Perplexity
• Google-Extended — Google’s control token for Gemini and AI Overviews
• Applebot-Extended, meta-externalagent, Bytespider, cohere-ai

The distinction between an indexing crawler and a user-triggered visit matters commercially: ChatGPT-User and Perplexity-User correspond to someone who, at that moment, is trying to buy. Blocking them costs revenue now, not long-term visibility.

Step 1 — Count the refusals in your logs

If your access logs use the Apache or Nginx combined format, one command gives you thirty days of truth:

zgrep -hiE "GPTBot|ClaudeBot|PerplexityBot|Google-Extended|OAI-SearchBot" access.log*.gz \
  | awk '{print $9}' \
  | sort | uniq -c | sort -rn

You get the distribution of status codes returned to these agents. Reading it is immediate:

• mostly 200: you are reachable, go to step 3;
• 403 or 401: application or WAF block;
• 429: rate limiting, often more insidious — access exists but stays too slow to be usable;
• no lines at all: the most common case, and the most misread.

Step 2 — No lines is not good news

If the search returns nothing, there are two hypotheses. Either no AI crawler ever came — implausible for an indexed retail site. Or, and this is almost always the case, the block happens upstream of your application server.

Datadome, Cloudflare Bot Management and Akamai Bot Manager decide at the edge. The request never reaches your origin, so it never appears in your Apache or Nginx logs. You have to look elsewhere:

• Datadome console, Traffic section filtered by user agent;
• Cloudflare, Security Events, filter User Agent contains GPTBot;
• CDN logs, not origin logs.

This is the step where most teams discover tens of thousands of monthly refusals.

Step 3 — Check what the crawler actually receives

A 200 is not enough. The response still has to contain something usable. Compare the size and content served to a browser and to a crawler:

curl -s -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/128.0 Safari/537.36" \
  https://your-site.com/category/running-shoes | wc -c

curl -s -A "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" \
  https://your-site.com/category/running-shoes | wc -c

Two gaps are revealing. If the second response is markedly shorter, you are serving a degraded variant — sometimes an anti-bot challenge page returned with a 200, which escapes every status-code count. If both are identical but short, the problem is not blocking: it is client-side rendering, a separate issue covered below.

Finally, check the text actually present, outside markup:

curl -s -A "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" \
  https://your-site.com/product/ref-1234 \
  | sed -e 's/<script[^>]*>.*<\/script>//g' -e 's/<[^>]*>/ /g' \
  | tr -s ' ' | wc -c

Below 1,500 characters on a product page, assume the agent has nothing to work with: no description, no price, no availability.

Step 4 — Re-read robots.txt, line by line

The block may be entirely deliberate and documented, inherited from a decision taken hastily in 2023 when the press was explaining that you had to “protect yourself from AI”:

User-agent: GPTBot
Disallow: /

That directive still exists on a great many sites. It was added out of fear of content scraping, at a time when no commercial traffic came from these surfaces. That is no longer true.

Watch for implicit rules too: a User-agent: * followed by Disallow: / applies to every crawler that has no group of its own.

What to do with the finding

The reflex to “open everything” is as bad as blanket blocking. The defensible position is a trade-off:

• open the templates that sell — home, categories, product pages, brand pages;
• keep closed account areas, checkout funnels, faceted internal search, and anything generating infinite URLs;
• distinguish indexing crawlers from user-triggered visits, and be more permissive with the latter;
• measure afterwards, or the decision will never be revisited.

An allow rule ships in hours. What takes time is rebuilding what agents stopped seeing during the months when nobody was looking.

What this method does not show

It establishes a fact — you are blocked, or you are not. It does not tell you what that costs, which templates suffer most, or how to fix it without reopening the door to price scraping. Those are three separate questions, and they are answered by looking at your whole delivery chain, not one log line.