Skip to main content

Scraper throttling

stillorigin limits scrapers at the network edge using Network protection connection rules. This does not inspect user agents and does not require Website protection to be enabled on a domain. Every HTTP and HTTPS connection through the sentry (ports 80 and 443) is covered.

How it works​

Each client IP may open a set number of new connections per second, plus a burst, and may hold a set number of connections open at once.

  • Only new connections count. Once a connection is open, everything sent over it (pages, images, downloads) passes without being metered, so a visitor loading a large page is not penalised.
  • New connections over the limit are dropped and recorded as blocked. Browsers retry on their own, so a visitor just slightly over the limit usually sees no error.
  • If an IP keeps opening connections faster than the limit for about a minute, it is placed on a short temporary ban, and its open connections are cut too. Sampled drops appear on Logs → Blocked IPs. The live list, with time remaining, appears on Logs → Temporary bans until the ban expires or you unban the address. Temporary bans carry on when a sentry applies a settings change.

Search engines and AI crawlers, identified only by their published IP ranges, get their own separate limits by default (lower than ordinary visitors'). Hosts can raise them if a legitimate bot is being clipped. See Verified crawlers.

Cluster settings​

These limits are set per cluster: on Clusters, click Edit. Defaults for a new cluster (an existing cluster keeps the values it already has):

SettingDefaultWhat it does
New connections per second per IP10How many new connections one visitor IP may open each second
Burst (new connections per IP)50Extra new connections allowed at once, for example when a page first loads
Open connections per IP50Most connections one IP may hold open at the same time to ports 80 and 443
Crawler new connections per second per IP2The same limit for verified crawler ranges
Crawler burst (new connections per IP)10Extra new connections a verified crawler may open at once
Temp ban (minutes)10How long a persistent offender stays temporarily blocked

Raise crawler limits if a legitimate bot is being clipped. Lower ordinary limits if a cluster is seeing heavy scrape traffic from unverified IPs.

Block reasons​

Logs → Blocked IPs shows which limit caught a drop:

ReasonMeaning
blocked: serviceYou blocked that identified service on Block identified traffic
Connection limitAn IP held more connections open than the limit
Crawler limitA verified crawler IP went over its connection limits
Deny-listThe IP or range is on your cluster deny list or stillorigin's shared block list
Excessive trafficAn ordinary IP opened new connections faster than its limit
Temporary banThe IP is temporarily banned after sustained excess

An ordinary IP that is caught for excessive traffic is also added to your cluster Deny list for your deny expiry hours (24 hours when it's not set or 0). See Log reasons explained for every reason and what you can do about it.

Temporary bans expire automatically. They do not have to be removed by hand. On Logs → Temporary bans you can still unban an address across every sentry in the cluster, or promote it onto the cluster deny list as a timed or permanent entry.

What you will see​

  • The dashboard and each sentry's Health view show allowed (mint) and blocked (coral) HTTP requests for the last 24 hours. Cluster limits on the packet filter count new connections, not HTTP requests.
  • A per-provider breakdown of verified crawlers versus other traffic appears on the dashboard, each cluster, and each sentry. See Dashboard and activity.