Fail the deploy when AI crawlers see an empty page

A new framework version, a CDN rule or a client-side rewrite can leave OAI-SearchBot, GPTBot, ClaudeBot and PerplexityBot with a blocked or empty page, and nobody notices for weeks. This check requests your site with each bot's user-agent string and fails when the answer is not HTTP 200 with readable text. It runs on your machine or in your CI. It calls no AI model and sends nothing to CitedBy.

1. Save as ai-crawler-check.sh

#!/usr/bin/env bash
# Fails when AI crawlers get an error or an empty page from $1 (default: your deployed URL).
# Usage: ./ai-crawler-check.sh https://yourshop.com
set -u
URL="${1:?usage: ai-crawler-check.sh https://your-site}"
MIN_CHARS="${MIN_CHARS:-300}"
fail=0
check() {
  name="$1"; ua="$2"
  body="$(mktemp)"
  code="$(curl -sL -m 20 -A "$ua" -o "$body" -w '%{http_code}' "$URL")"
  chars="$(sed -e 's/<script[^>]*>.*<\/script>//g' -e 's/<[^>]*>/ /g' "$body" | tr -s '[:space:]' ' ' | wc -c | tr -d ' ')"
  rm -f "$body"
  if [ "$code" != "200" ]; then echo "FAIL $name: HTTP $code"; fail=1
  elif [ "$chars" -lt "$MIN_CHARS" ]; then echo "FAIL $name: only $chars readable characters (needs server-rendered HTML)"; fail=1
  else echo "ok   $name: HTTP 200, $chars readable characters"; fi
}
check OAI-SearchBot "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot"
check GPTBot "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot"
check ClaudeBot "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
check PerplexityBot "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
exit $fail

2. GitHub Action (.github/workflows/ai-crawlers.yml)

name: AI crawler check
on:
  deployment_status:   # runs after Vercel, Netlify or Render reports a deploy
  workflow_dispatch:
    inputs: {url: {description: Site URL, required: true}}
jobs:
  ai-crawlers:
    if: github.event_name == 'workflow_dispatch' || github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: bash ai-crawler-check.sh "${{ github.event.inputs.url || github.event.deployment_status.target_url }}"

Vercel, Netlify or any host

Run bash ai-crawler-check.sh https://your-site as a post-deploy step, or keep the workflow above: it fires on every successful deployment status.

What it cannot see

It sends the bots' user-agent strings from your runner. A firewall that verifies the vendors' IP ranges may treat the real bots differently, and it does not run JavaScript, which is also how the bots read your page.

Audit the whole site for AI search, from €9