Scraper throttling
stillorigin limits scrapers at the network edge using Network protection connection rules. This does not inspect user agents and does not require Website protection to be enabled on a domain. Every HTTP and HTTPS connection through the sentry (ports 80 and 443) is covered.
How it works
Each client IP may open a set number of new connections per second, plus a burst, and may hold a set number of connections open at once.
- Only new connections count. Once a connection is open, everything sent over it (pages, images, downloads) passes without being metered, so a visitor loading a large page is not penalised.
- New connections over the limit are dropped and recorded as blocked. Browsers retry on their own, so a visitor just slightly over the limit usually sees no error.
- If an IP keeps opening connections faster than the limit for about a minute, it is placed on a short temporary ban, and its open connections are cut too. Sampled drops appear on Logs → Blocked IPs. The live list, with time remaining, appears on Logs → Temporary bans until the ban expires or you unban the address. Temporary bans carry on when a sentry applies a settings change.
Search engines and AI crawlers, identified only by their published IP ranges, get their own separate limits by default (lower than ordinary visitors'). Hosts can raise them if a legitimate bot is being clipped. See Verified crawlers.
Cluster settings
These limits are set per cluster: on Clusters, click Edit. Defaults for a new cluster (an existing cluster keeps the values it already has):
| Setting | Default | What it does |
|---|---|---|
| New connections per second per IP | 10 | How many new connections one visitor IP may open each second |
| Burst (new connections per IP) | 50 | Extra new connections allowed at once, for example when a page first loads |
| Open connections per IP | 50 | Most connections one IP may hold open at the same time to ports 80 and 443 |
| Crawler new connections per second per IP | 2 | The same limit for verified crawler ranges |
| Crawler burst (new connections per IP) | 10 | Extra new connections a verified crawler may open at once |
| Temp ban (minutes) | 10 | How long a persistent offender stays temporarily blocked |
Raise crawler limits if a legitimate bot is being clipped. Lower ordinary limits if a cluster is seeing heavy scrape traffic from unverified IPs.
Block reasons
Logs → Blocked IPs shows which limit caught a drop:
| Reason | Meaning |
|---|---|
| blocked: service | You blocked that identified service on Block identified traffic |
| Connection limit | An IP held more connections open than the limit |
| Crawler limit | A verified crawler IP went over its connection limits |
| Deny-list | The IP or range is on your cluster deny list or stillorigin's shared block list |
| Excessive traffic | An ordinary IP opened new connections faster than its limit |
| Temporary ban | The IP is temporarily banned after sustained excess |
An ordinary IP that is caught for excessive traffic is also added to your cluster Deny list for your deny expiry hours (24 hours when it's not set or 0). See Log reasons explained for every reason and what you can do about it.
Temporary bans expire automatically. They do not have to be removed by hand. On Logs → Temporary bans you can still unban an address across every sentry in the cluster, or promote it onto the cluster deny list as a timed or permanent entry.
What you will see
- The dashboard and each sentry's Health view show allowed (mint) and blocked (coral) HTTP requests for the last 24 hours. Cluster limits on the packet filter count new connections, not HTTP requests.
- A per-provider breakdown of verified crawlers versus other traffic appears on the dashboard, each cluster, and each sentry. See Dashboard and activity.