What Actually Breaks a Scraper in Production

A scraper that works in a demo and a scraper that survives in production are two different animals. The demo runs once, against a fresh page, from your laptop's IP, and you watch it succeed.

Production runs the same job a thousand times against a moving target, and one morning the dashboard shows a column of zeros. Every row says the same thing: nothing came back.

Here's the trap. That uniform zero is four completely different problems wearing the same costume. And the single decision that separates a scraper that heals from one that flatlines is whether it can tell them apart before it reacts.

The "the site changed" myth

Ask anyone why their scraper broke and you'll hear "the site changed." Sometimes that's true. Mostly it's a shrug dressed up as a diagnosis, and even when the site did change, "changed" is hiding at least four distinct failure modes that demand opposite responses.

The industry numbers back up how messy this gets. Unmanaged structural change drives 40-60% of recurring scraper maintenance effort. Meanwhile the environment they run in keeps getting more hostile: per Imperva's 2025 Bad Bot Report, automated traffic crossed 51% of all web traffic in 2024, with bad bots at 37%, which means target sites are tuning their defenses against exactly the kind of automation a scraper performs.

And here's the part that should keep you up at night: most of these failures are silent. A DOM change can break parsing without throwing a single error message. The scraper looks like it's running fine while it quietly collects nothing.

If your system can't tell a silent selector failure from a silent block, it can't recover from either. It can only guess, and guessing at scale is just being wrong faster.

Four failure modes, one symptom

When we ran Trawl's stuck scrapers down to ground truth, the empty result kept resolving into one of four causes:

1. Navigation never landed. The page didn't load at all. There's no content to parse because there's no page: a timeout, a dropped connection, a redirect that went nowhere. This one is genuinely transient more often than not.

2. Selector rot. The page loaded fine. The target element is just gone or moved: renamed class, restructured layout, new pagination. Crucially, there's no block signature here. The site isn't fighting you; your map is just out of date.

3. Anti-bot block. A challenge, a CAPTCHA, a 403. The page is actively refusing the request. The content exists, but you weren't allowed near it.

4. Geo/timezone incoherence. The sneaky one. When the request exits from one region but the browser advertises a locale that contradicts it, that contradiction is itself a suspicion signal. It doesn't always produce a clean block. It quietly raises the odds of one, and it silently poisons every measurement you take of whether things are working.

Four causes. One symptom. Collapse them into a single "empty, retry it" bucket and you will do the wrong thing three times out of four.

Why blind retry is the wrong default

The naive instinct is to treat every empty result as transient and retry harder. That works for exactly one of the four cases. So why is "retry" the default everywhere? Because it's the only response that requires zero diagnosis, and zero diagnosis is exactly the problem.

Selector rot is where blind retry does the most damage. A thrown selector error returns no block signal, so a system that only escalates when it detects a block never escalates at all. It scores the run as a terminal zero and walks away, never retrying the configuration that would have worked. We hit this gap directly and closed it: the system now treats a selector failure as a reason to escalate and flag, not as a dead end.

Geo/timezone incoherence is worse under blind retry, because retrying doesn't fix the contradiction. It just sends the same incoherent signal again, burns budget, and teaches the target site your pattern. The right move is to make the request coherent, not to repeat it louder.

And some failures shouldn't be retried at all. A hard 403 floor, or a target whose defenses are adaptive and flaky (where one success doesn't predict the next) punishes escalation. Retrying harder there isn't persistence, it's self-harm. The correct response is to stop, mark the target unreliable, and move on.

The taxonomy we run on

This is the table that replaced "the site changed" as our mental model. Diagnose the category first; the right response falls out of it.

What re-measuring truth actually surfaced

Here's the result that made the case for all of this. We took 52 scrapers that were sitting "stuck" (the kind you'd assume are broken, the kind a less careful system would have written off without a second look) and re-probed every one against ground truth.

24 of them weren't broken at all. They were being run on a more expensive setting than they needed, and they recovered cleanly once we re-measured what each one actually required and downgraded it to a cheaper configuration that worked.

Read that again: roughly 46% of "stuck" wasn't broken. It was mis-tiered: recoverable jobs hiding behind a measurement the system had never bothered to refresh.

That's a Trawl-internal result on our own fleet, not a benchmark and not a promise about anyone else's system. But it's the cleanest evidence we have for the thesis: the expensive failure isn't the site changing. It's not knowing which kind of "empty" you're looking at.

Diagnose before you heal

Self-healing is the goal. But an over-eager heal is its own failure mode: it's a cure looking for a disease. An auto-correction that fires on a transient empty result can rewrite a script that was actually fine, clobbering a working diagnosis to "fix" a problem that never existed. So two principles guard everything:

Diagnose before you heal. Categorize the failure first; only then choose the response. On adaptive-flaky targets, auto-correction stays off, because the empty result there is noise, not signal.

Every heal is reversible. No auto-correction is a destructive overwrite. Each one ships as a guarded, reversible change with a fallback net, so a wrong diagnosis costs you a rollback, not a corrupted scraper.

And none of it works without the right contract underneath. The reason the system can escalate and recover at all is that a failed extraction returns "empty" cleanly instead of throwing and dying. A thrown error is a dead end; a clean empty result is a decision point.

That single design choice (fail soft, classify, then act) is what turns a column of zeros from a death sentence into a diagnosis.

Key Takeaways

• A zero-row result is a symptom, not a diagnosis: it maps to at least four distinct failure modes that demand opposite responses.
• Selector rot is the silent killer: it throws no block signal, so block-only escalation logic scores it a terminal zero and never retries the setting that would've worked.
• Geo/timezone incoherence is a failure mode almost nobody names: a contradiction between exit region and claimed locale is itself a detection signal, and it quietly poisons every pass-rate measurement you trust.
• Blind retry is wrong for three of the four cases. Backoff fits transient nav failures; everything else needs escalate, fix-coherence, or stop.
• Re-measuring truth beats assuming broken: a re-probe of 52 stuck Trawl scrapers found 24 weren't broken: just mis-tiered, recoverable on a cheaper working setting.
• Diagnose before you heal, and make every heal reversible: an auto-fix that fires on the wrong diagnosis clobbers a working scraper to solve a problem that was never there.

Whether you're running DIY Puppeteer, stitching together traditional scraping APIs, or self-hosting your own fleet, the lesson is the same: the system that survives production isn't the one that retries hardest. It's the one that knows which failure it's looking at before it lifts a finger. Build that diagnosis layer and your fleet stops flatlining and starts healing. If it's the layer you'd rather not build yourself, that's exactly the layer Trawl is.

Related reading: The proxy tier ladder: how a scrape escalates and Synthetic monitoring without a vendor: catching production failures with scheduled scrapers.

A note on responsible use. Trawl is an orchestration layer. You are responsible for complying with applicable law and the terms of service of any site you target, and for scraping only publicly available data. The tooling automates the how; the whether is on you.

Written by Pierre | June 2026