Verified crawlers
Genuine search and assistant crawlers should be able to fetch your sites. Self-hosted scrapers that pretend to be those bots should not get the same treatment.
stillorigin tells them apart by source IP, using each provider's published address list. User-Agent strings are not used for this decision, so spoofed bot names do not unlock the crawler limits.
Their own limits
IPs that fall inside a Search engines or AI crawlers range use the cluster's crawler limit (default 2 new connections per second, burst 10) and the same open-connection limit as everyone else (default 50). By default this is lower than the ordinary per-IP limit, and they are not unlimited. You can raise it on the cluster if a legitimate crawler is being clipped. If a verified crawler exceeds that budget it is throttled, logged as Crawler limit, and can be temporarily blocked like any other offender.
Everyone else, including scrapers that only change their user agent, uses the ordinary per-IP limit (default 10 new connections per second, burst 50). These limits count new connections, not HTTP requests, so traffic over a connection that is already open is not limited.
Crawler limits are a per-group setting. Search engines and AI crawlers start with them on; the other Identified traffic groups start with them off and use the ordinary limits. Only stillorigin staff can change which groups get crawler limits.
You can block a group or a single service, across your whole network or on one cluster, on Block identified traffic. Identified traffic is never added to the naughty list automatically.
Providers we import
Ranges are refreshed from each provider's official published list about once a day. They are grouped as Search engines or AI crawlers:
Search engines
- Applebot
- Bingbot
- Discordbot (link previews)
- DuckDuckBot
- Google special crawlers
- Google user-triggered agents
- Google user-triggered fetchers
- Google user-triggered fetchers (Google)
- Googlebot
- TelegramBot (link previews)
AI crawlers
- Amazonbot
- Amzn-SearchBot
- Amzn-User
- ChatGPT-User
- Claude (ClaudeBot, Claude-User and Claude-SearchBot)
- GPTBot
- OAI-SearchBot
- Perplexity-User
- PerplexityBot
Only providers that publish an official IP list are included.
What is not included
CCBot / Common Crawl is deliberately left off the verified list. It does not use the crawler limits and is treated as ordinary traffic.
A bot that is missing from the list is not blocked by default; it simply uses the normal per-IP budget. If you need different limits, change them on the cluster.
Other identified services
The portal's Identified traffic section also names other services we can identify from their official IP lists. These groups start with crawler limits off, so they use the ordinary limits. Being named never lets their traffic skip your limits, the WAF or your deny list, and you can block them on Block identified traffic.
Monitoring & uptime
- 360 Monitoring
- Better Stack
- HetrixTools
- Oh Dear
- Pingdom
- StatusCake
- updown.io
- Uptime.com
- UptimeRobot
Site management
- Jetpack / WordPress.com
- WP Umbrella
Payment providers
- Braintree
- Checkout.com
- GoCardless
- Mollie
- Paddle
- PayPal
- Square
- Stripe
These are the addresses the providers send payment notifications and webhooks from. Captcha Protection never checks them, so orders are confirmed when the customer pays.
Security platforms
- Patchstack
- Security Headers
Internet scanners
- Censys — labelled, never trusted, and you can block it. Captcha Protection checks it like any other visitor
If a service is missing, you can request it.
Traffic breakdown
The portal's Identified traffic section groups allowed and blocked traffic by type for the last 24 hours: Search engines, AI crawlers, Monitoring & uptime, Site management, Payment providers, Security platforms, Internet scanners, Origin servers (previously Origin / host monitors), Unverified, Unknown host and Firewall blocks. Allowed and blocked counts include both HTTP requests and HTTPS connections. See Identified traffic.
It appears:
- on the dashboard — all of your clusters together
- on each cluster — when you expand it
- on each sentry — on its Health view
- on a domain's Blocks tab, under Identified traffic on this domain — traffic on that domain's hostname only. Firewall blocks with no hostname appear only in the cluster totals
See Dashboard and activity, Block identified traffic and Scraper throttling.