Pakistan
USA
Canada
UK
Dubai
Australia
Germany
- Separate search bots from training bots before writing any rules.
- Blocking Google-Extended does not affect Google Search rankings.
- User-triggered fetchers may not follow robots.txt.
- CDN and firewall settings can block bots your robots.txt allows.
What is the difference between search bots and training bots?
AI companies run several crawlers with different jobs. Search crawlers fetch pages so they can be shown and linked in AI search answers. Training crawlers collect content that may be used to train future models. User-triggered fetchers visit a page because someone asked the assistant to read it, such as when a user pastes a link into a chat.
These are controlled separately. OpenAI, for example, says OAI-SearchBot and GPTBot are independent, so you can allow one and block the other. For most service businesses, being found matters more than keeping content out of training data, so the usual choice is to allow search bots and decide on training bots as a business policy. Write that decision down, so the next developer or hosting change does not quietly reverse it.
Which bot does what?
The main user agents as documented by each company.
User agent | Company and job | Effect of blocking |
|---|---|---|
OAI-SearchBot | OpenAI, ChatGPT search results | Not shown in ChatGPT search answers |
GPTBot | OpenAI, model training | Content signalled as not for training |
ChatGPT-User | OpenAI, user-initiated visits | OpenAI says robots.txt may not apply |
PerplexityBot | Perplexity, search results | Not surfaced in Perplexity results |
Perplexity-User | Perplexity, user-requested fetches | Perplexity says it generally ignores robots.txt |
Google-Extended | Google, Gemini training and grounding | No effect on Google Search inclusion or ranking |
What do example robots.txt rules look like?
Each bot gets its own group. To allow ChatGPT search but not training, use User-agent: OAI-SearchBot with Allow: /, then a separate group with User-agent: GPTBot and Disallow: /. Add User-agent: PerplexityBot with Allow: / if you want Perplexity visibility. Rules for a named bot override the general User-agent: * group for that bot, so be careful that a specific group does not accidentally allow less than you intended.
To opt out of Gemini training without touching Search, add User-agent: Google-Extended with Disallow: /. Never write User-agent: * with Disallow: / unless you really want to block every crawler, including Googlebot. Google's robots.txt guide explains the syntax, and a mistake here can remove a site from search. Test any change with Search Console's robots.txt report after publishing.
Which CDN and firewall settings should I check?
robots.txt is only a request; network settings can still block bots.
- Bot management rulesSome CDNs offer one-click AI bot blocking. Check it matches your robots.txt decisions.
- Managed robots.txtSome services can add their own lines to your robots.txt. View the live file in a browser.
- Rate limits and challengesAggressive challenges can stop legitimate crawlers from fetching pages.
- Published IP rangesOpenAI and Perplexity publish IP ranges; allow-list them if your firewall blocks unknown bots.
- Security pluginsWordPress security plugins sometimes block unfamiliar user agents by default.
Should publishers and service businesses decide differently?
Publishers who earn money from original articles may have good reasons to block training bots, since their content is the product. Even then, many choose to keep search bots allowed so their articles can still be cited and linked. That way they keep a source of referral traffic while still signalling that the content should not be used for model training. The decision is commercial as much as technical.
Service businesses such as clinics, law firms, agencies and contractors usually benefit from being described accurately by AI tools. Their website is a sales tool rather than the product. For them, blocking search bots mostly removes a route to new customers. Our technical SEO guide covers how crawling fits into the bigger picture. If you are unsure, allowing search bots and blocking only training bots is a sensible, reversible starting point.
How we handle this for clients
We read the live robots.txt file, then test what each major crawler actually receives, because CDN and security settings often differ from what the file says. We list every AI user agent seen in the server logs where logs are available, along with the status codes they received. This shows whether the bots are reaching the pages that matter or being turned away.
Then we agree a written policy with the client for each bot: search, training and user-triggered. We update robots.txt and network rules to match, and recheck after any hosting, CDN or plugin change. The policy is kept in writing so anyone working on the site later can see what was decided and why. This is part of our technical SEO services.
Anything specific for Pakistan, Gulf and UK sites?
Many sites in Pakistan and the Gulf sit behind a CDN for speed and security, and default bot settings are rarely reviewed. If you serve customers in several countries, a block in one region's configuration can affect all of them. Hosting companies and developers sometimes switch on AI bot blocking during a security clean-up without telling the business owner, so check after any such work.
Make sure Bingbot is allowed too, since Microsoft Copilot draws on Bing. Keep your Google Business Profile and directory listings accurate, because AI tools can still describe you from those even when your own site is hard to reach. For the visibility side, see getting mentioned in ChatGPT, which covers content and mentions as well as access.
How do I check my server logs?
Logs show which bots actually visit and what they receive.
Filter by user agent
Search access logs for OAI-SearchBot, GPTBot, PerplexityBot, ClaudeBot and Googlebot.
Check status codes
Look for 403 or 429 responses, which suggest blocking or rate limiting.
Verify the source
User agents can be faked, so confirm against the company's published IP ranges.
Recheck after changes
Review logs again a few days after any rule change.
Junaid has more than 18 years in SEO, paid media, web development and custom software. He founded SEOnumber1, builds WordPress plugins, CRMs and web applications, has served 500+ clients in 48 countries, and holds Google Ads, Analytics and Google Blue and Green Belt certifications. He is a National Level Award Winner in Digital Marketing. Read his story or ask a question about this article.