Scraping Sites That Don't Want to Be Scraped: A Distributed Architecture

Monitoring availability across a few hundred retail sites sounds like a cron job and some HTTP requests. It is, for about ten minutes, until the bans start. Real sites fight back, and on more axes than people expect: IP reputation, request rate, TLS and HTTP fingerprints that reveal you are not a real browser, behavioural signals, and outright challenge pages from the CDN. A single server hammering endpoints from one datacenter IP gets fingerprinted and blocked almost immediately, and once an IP is burned, it stays burned.

It helps to understand what you are actually up against, because each defence shapes part of the architecture. They look at where you are coming from (IP reputation and rate), what you look like at the network level (a plain HTTP client has a different TLS handshake than Chrome), how you behave (real users do not request a thousand product pages a minute in perfect order), and whether you can execute JavaScript and solve a challenge. You are not defeating one check, you are staying under several at once, and the moment any of them trips, the rest get more suspicious.

The architecture that survives

The shape that holds up is a fleet of stateless workers, each routed through a health-checked rotating proxy pool, coordinated by a hub that assigns targets, deduplicates work, and schedules so that no single IP develops a bot-like pattern. Stateless workers matter because you want to scale them horizontally and treat any one as disposable. The coordinator matters because uncoordinated workers re-scrape the same targets, step on each other, and hammer sites into banning you faster than a single polite scraper ever would have.

Proxies are where the money goes, and the type matters enormously. Datacenter proxies are cheap and instantly flagged by serious targets, because their IP ranges are known. Residential and mobile proxies look like real users and cost accordingly. You health-check them continuously on latency, success rate, and ban rate, and you evict the dead ones, because a pool full of burned proxies is worse than no pool at all: it gives you false confidence while quietly failing every request.

What the coordinator actually does

It is easy to wave at "a coordinator" without saying what it earns its keep doing, so here is the work. It owns the schedule, deciding which targets are due for a check and spacing those checks so a given site sees a human-like cadence rather than a metronome. It deduplicates, so two workers never redundantly fetch the same target at the same moment, which both wastes proxies and accelerates bans. It holds the per-target state, the cooldowns, the circuit-breaker status, the last-seen results, so the decision about HTTP-versus-headless or skip-versus-fetch is made consistently no matter which worker picks up the job. And it absorbs priority, pushing time-sensitive targets to the front when freshness matters most. Workers stay dumb and disposable on purpose; the intelligence lives in the coordinator, because that is the only place that can see the whole fleet and keep it collectively under the radar.

Two cheap decisions per request

Headless Chrome can solve almost anything. It runs the JavaScript, passes the challenge, and looks like a browser, but it is heavy on memory, CPU, and time per request. So you do not reach for it by default. Every request makes two cheap decisions first: is this target currently on cooldown, so you do not poke a site that just soft-banned you, and does it actually need a real browser, or will a plain HTTP request with the right headers do? Plain HTTP is an order of magnitude cheaper, so you use it wherever you can and reserve a pooled, reused headless fleet for the targets that genuinely require it. Spinning up a fresh browser per request is the rookie move that makes the bill explode.

The circuit breaker is doing real work here. The moment a target starts returning challenge pages, empty results, or suspiciously uniform data, a soft ban, you open the breaker for that site, back off exponentially, and rotate identity before trying again. Hammering through a soft ban is exactly how you turn it into a hard one, so the breaker is as much about protecting tomorrow's access as serving today's request.

Designing for the rewrite

The single most useful mindset shift in this work is to stop treating a broken adapter as a failure and start treating it as the expected weather. Targets change, and an adapter that parsed a page perfectly last month will silently start returning empty results the week the site ships a redesign. So you instrument for it: you alert on a target whose success rate or result shape suddenly shifts, because a quiet drop to zero is indistinguishable from a ban until you look, and the faster you notice the faster you can rewrite. You keep adapters small and isolated so a rewrite is an afternoon, not a project. And you version them, so you can tell when behaviour changed and correlate it with a change on the site. Building for the rewrite, rather than pretending the adapter is permanent, is what keeps the whole fleet maintainable over the years these systems actually run.

It is an arms race, so build for change

Every target is a moving goalpost. They change their markup, add a challenge, tweak detection, or restructure a page the week you stopped watching. So you isolate each site behind its own adapter with its own parsing and its own quirks, and you accept that adapters break and get rewritten, because that is the maintenance reality of this work, not a sign you did it wrong. You also randomize the fingerprints you control, the user-agent, the header order and casing, the TLS profile, so a fleet does not all look identical to the other side. And you cache aggressively, because the cheapest request is the one you do not make: if a page has not plausibly changed, do not fetch it again.

Be a good citizen, and stay on the right side of the line

The same techniques that monitor stock for a legitimate alerting product can obviously be abused, so the framing matters and it is not an afterthought. You do this for sites and data you are authorized to access. You cache and rate-limit yourself so you are not a burden on the targets. You respect the signals you are given rather than treating every block as a personal challenge to defeat. The engineering here is about resilience and cost under adversarial conditions, not about being maximally aggressive, and keeping that distinction clear is what separates a monitoring product from a nuisance.