Generate custom robots.txt rules to block AI crawlers and training scrapers like GPTBot, CCBot and Google-Extended, while keeping citation bots active so your content still earns traffic in AI search.
—
Paste your current file to check which AI crawlers are currently allowed, blocked, or undefined.
Never use a blunt wildcard block. Segment your directives to keep your content in AI search citations while stopping unlicensed LLM training.
GPTBot, ClaudeBot, and Google-Extended) while allowing live citation bots (like OAI-SearchBot and PerplexityBot).User-agent: * Disallow: / — doing so de-indexes your site from Google and Bing search results.Google-Extended is a policy token, not an active crawler; disabling it prevents Gemini training without hurting traditional search rankings.Protect your intellectual property from bulk LLM scraping while ensuring your brand remains discoverable in AI search results.
Select our recommended preset to block bulk training crawlers while keeping citation bots active.
Toggle individual crawlers like ClaudeBot, Bytespider, or Applebot-Extended based on your policy.
Grab your clean, formatted robots.txt with one click or download the ready-to-upload text file.
Add the rules into Rank Math SEO or upload directly to your web root to take effect immediately.
When publishers choose to block AI crawlers, many make the mistake of disallowing all automated bots. This cuts off your visibility in real-time generative search platforms. AI user-agents operate under two very different purposes:
Key distinction: Training crawlers (like CCBot and Bytespider) ingest content to train foundational AI models without sending traffic. Citation bots (like OAI-SearchBot and PerplexityBot) fetch live pages to link sources directly in AI search answers.
According to Cloudflare's 2025 Year in Review, AI training crawling ran up to eight times the volume of AI search crawling in 2025 — yet user-triggered citation crawling grew more than 15×. Explore our guide on AEO vs SEO to understand how answer engine optimization works alongside traditional search.
A common misunderstanding is that Google-Extended is an active crawler that hits your server. As detailed in Google's official documentation, it is a standalone policy token.
Standard Googlebot reads your robots.txt. If it finds a disallow under Google-Extended, Google excludes your content from training pipelines for Gemini and Vertex AI. Your normal organic Google Search ranking, crawling cadence, and rich snippet eligibility remain completely intact.
A complete breakdown of major AI user-agents, who operates them, and whether you should allow or block them.
| User-Agent Token | Operator | Purpose | Default Policy | Impact on Your Site |
|---|---|---|---|---|
GPTBot | OpenAI | LLM Training Data | Block | Collects training datasets. Blocking does not break live ChatGPT search. |
OAI-SearchBot | OpenAI | Search & Attribution | Allow | Powers search results and source links in SearchGPT and ChatGPT. |
ChatGPT-User | OpenAI | User Live Browsing | Allow | Triggered when a ChatGPT subscriber asks the assistant to read your link. |
Google-Extended | Gemini & Vertex AI | Block | Excludes data from model training without altering Google Search ranking. | |
ClaudeBot | Anthropic | Claude LLM Training | Block | Scrapes text corpora for Anthropic foundation models. |
Claude-SearchBot | Anthropic | Real-Time Search | Allow | Fetches live citations and source references in Claude conversations. |
PerplexityBot | Perplexity AI | Search & Answers | Allow | Indexes web content to generate sourced answers in Perplexity. |
Bytespider | ByteDance | TikTok / AI Training | Block | High-frequency crawler known for aggressive server bandwidth usage. |
Applebot-Extended | Apple | Apple Intelligence | Block | Controls training ingestion for Apple Intelligence without disabling Siri search. |
CCBot | Common Crawl | Open Dataset Scraping | Block | Bulk scraper widely ingested by third-party AI companies without attribution. |
Meta-ExternalAgent | Meta | Llama Model Training | Block | Extracts text across the web for Meta AI training sets. |
The most dangerous error is adding a blanket wildcard Disallow: / under User-agent: *. Doing so will completely wipe your website from organic Google and Bing search results.
Another common mistake is confusing OpenAI's user agents. Blocking OAI-SearchBot will stop your site from appearing as a cited source in ChatGPT search. Review OpenAI's official bot documentation for technical details.
Pairing your robots.txt file with structured data helps search engines interpret your content accurately. Use our free schema markup generator to build verified JSON-LD. If high-frequency scrapers are impacting page speed, run our website speed test to identify server latency issues.
Robots.txt controls access. To actually get cited in AI answers, your site needs clean technical structure and authoritative content.
Once you allow citation bots like OAI-SearchBot and PerplexityBot, make sure your content is formatted for retrieval-augmented generation (RAG) engines. Learn the exact playbook in our guide on how to get cited by ChatGPT.
Companion standard: While robots.txt tells bots where they can go, the companion file generated by the llms.txt generator by H&U Solutions ↗ provides curated markdown context directly to language models.
You can verify whether your brand currently appears in generative search overviews using an AI overview checker ↗.
Balancing bot protection, crawl budget, and generative visibility requires senior engineering expertise. If you want your crawl budget and technical architecture optimized by experts, partner with our technical SEO agency.
We also manage enterprise tracking and analytics through our analytics & reporting team and build high-ROI campaigns with our paid search marketing team.
The technical concepts behind modern bot management, without the jargon.
Everything you need to know about robots.txt rules, AI scrapers, and search engine visibility.
Protect proprietary journalism and archives from bulk training while ensuring new articles are indexed in AI search engines.
Audit client robots.txt files to prevent accidental de-indexing and configure modern citation bot permissions.
Prevent aggressive AI scrapers from draining server bandwidth without interfering with standard Googlebot indexing.
The definitive guide to earning live citations and source links in OpenAI search.
How answer engine optimization works alongside traditional search engine optimization.
Master campaign tagging and traffic attribution inside Google Analytics 4.
Generate valid JSON-LD to help both search engines and AI models understand your entities.
Need companion files? Create structured context for AI engines with the llms.txt Generator by H&U Solutions ↗
Every tool is free — no signup. Configure your bot access here, then optimize search and performance across the suite.
Generate valid FAQ, Article & Organization JSON-LD to win rich snippets and AI citations.
Open tool →Check Core Web Vitals (LCP, INP, CLS) and diagnose server performance bottlenecks.
Open tool →Calculate return on ad spend, break-even ROAS, CPC and CPA targets accurately.
Open tool →Browse our complete suite of free technical optimization and audit utilities.
Explore tools →Get a comprehensive crawl budget, bot security and ranking audit from our engineers.
Book now →Devlet engineers technical SEO and AI search architectures that safeguard your assets while driving high-intent organic growth.
Get a free technical audit →Enter your details and we’ll send you a complete website audit report.
