The Proxy-Tier Ladder: How a Scrape Escalates When It Gets Blocked

A Block Is a Verdict, Not a Fluke

Most scrapers answer a block by sending the same request down the same path again. Same proxy, same fingerprint, same everything, until the run burns the budget or earns the site's permanent attention.

That reflex misreads what a block is. The site did not roll dice and lose. It looked at your request path, put it in a category, and rejected the category. Feeding it identical inputs gets you an identical verdict plus one more data point for its profile of you.

Thesis: a block is a signal to change path, not a reason to repeat one. Start on the cheapest path that could plausibly work, climb one rung only when the rung below provably fails, and pay for the expensive rung only when the cheap ones have been ruled out.

That is the ladder. What follows is how it is built, what ninety days of our own production traffic actually did on it, and the number that surprised us most: how rarely a failure looks like a block at all.

Table of Contents

1. The five-rung ladder
2. Where the traffic actually lands
3. Most failures are not blocks
4. Pins go stale, and staleness costs money
5. Why downgrading is the safe move
6. The ceiling is real
7. FAQ

The Five-Rung Ladder

Every scrape maps to one of five rungs, ordered by capability and by cost. Naming them for what they do beats naming them by tier number.

A scrape enters at Direct and moves up only when the current rung fails to produce a result. The billing shape matters as much as the price. The three proxied rungs are metered on bytes moved, so their cost scales with how much you scrape. The managed rung is billed per successful run, so its cost scales with how often you actually need it. Those are two different curves, and treating them as one number is how scraping budgets get away from people.

Where the Traffic Actually Lands

Architecture posts usually stop at the diagram. Here is the same ladder measured against ninety days of our own production runs. Each row is the share of runs that executed on that rung, and the share of those attempts that came back with data.

Two things in that table are worth sitting with.

The free rung carries the majority. Just over half of all runs execute on Direct, and it produces 60.2% of every successful run we serve. That is the entire economic argument for cheapest-path-first in one number. Route everything through a premium path by default and you would be paying for the hard minority on every single request.

The success rates do not descend cleanly, and that is expected. Premium residential returns data on 48.8% of attempts, well below Direct. That is not the rung underperforming. It is selection: the only requests that reach the upper rungs are the ones the cheaper rungs already failed, so the higher you climb the more adversarial the sample gets. A rung's success rate measures the difficulty of what arrives there, not its quality. Anyone publishing a flat "99% success rate" is either measuring an easy sample or measuring nothing.

The last column is the one that eventually cost us money. A quarter of our domains sit pinned to the most expensive rung. Some of them belong there. Most, it turned out, did not.

Skip the proxy plumbing. Trawl runs this ladder for you: cheapest path first, automatic climb on failure, top rung only when a site genuinely needs it. Try it free and point it at your hardest target.

Most Failures Are Not Blocks

"Fail on a rung, climb to the next" is easy to say. The hard part is defining what counts as a failure, and we got that wrong for longer than I would like.

The obvious trigger is an explicit block: a challenge page, a 403, a recognizable anti-bot signature. We caught those from day one. The one we missed was quieter. The page loads, returns a clean 200, and the data is simply not there. Nothing screams blocked. The run sits on its rung, technically successful and completely useless, and a naive retry loop will spin on it forever.

So we classified every failed run over the same ninety-day window. The breakdown is the most useful thing in this post.

An explicit block accounts for roughly one failure in five. The single largest category is the quiet one: page loaded, status fine, nothing extracted. If your escalation logic keys on 403s and challenge pages, you are reacting to about 20% of the ways a scrape actually dies and silently eating the rest.

Escalation now fires on any failure to produce a result, whatever the surface symptom. That is a one-line change in principle and a substantial one in outcome, because the failures it added are precisely the ones that used to fail silently.

The same measuring discipline saved us elsewhere, less flatteringly. We once spent weeks building our own capability against a hard wall before routing that wall through the managed rung we were already paying for. It passed on the first attempt. Obvious in hindsight, and the reason we now test the paid path before writing a line of replacement for it.

Pins Go Stale, and Staleness Costs Money

A pin gets assigned in a moment of fear. A site blocks you once, you pin it to a high rung, and it stays there. Nobody revisits it. The site's defenses change, the original block turns out to have been a one-off, and the expensive assignment quietly outlives its reason.

So we re-probed 52 of our hardest pinned targets from a lower rung to see whether the expensive assignment was still true. 24 came back as verified downgrades. One site pinned to the very top passed two rungs lower. Each downgrade was a confirmed pass on the cheaper path before we touched anything, not a hunch.

Across a handful of domains that is a rounding error. Across thousands it is a standing tax on every run, paid to a threat model that expired months ago.

Why Downgrading Is the Safe Move

Moving a site to a cheaper rung sounds like the risky direction. It is the opposite, and the reason is structural: escalation is already the backstop.

A downgrade is not a bet that the cheaper rung will always work. It is a corrected record of where the site sits today, applied as a reversible, guarded update. If a downgraded site tightens up later, the next run fails on the cheap rung and climbs again automatically, exactly like any other run. Nobody re-pins anything by hand.

The measured cost of being wrong is small. Of the runs that attempted a downgraded path, 62.1% returned data on the cheaper rung. The remaining 38% paid for one cheap failed attempt and then climbed. That is the entire downside, and it buys a permanently correct pin on the majority.

Truth restoration cuts both ways. A residual class genuinely belongs at the top, and a handful of targets whose anti-bot behavior is too erratic to trust are excluded from downgrade outright. Recording those honestly is the same discipline as clawing back the rest.

The Ceiling Is Real

The managed top rung beats the great majority of anti-bot categories. A couple of them reliably resist it anyway. Some sites are genuinely, permanently hard, and no amount of climbing changes that.

Pretending otherwise would make for a better pitch and a worse product. The honest version is that a ladder buys you the right rung for each site and a truthful record of which sites are out of reach, which is worth considerably more than a fictional 100%.

Build it yourself and you own every proxy decision and retry loop by hand. Buy a flat-rate API and you pay the hard-target premium on trivial targets too. The ladder makes that tradeoff automatic, and the measurement keeps it honest: cheap rungs carry the easy majority, the expensive rung stays reserved for the minority that earns it, and stale pins get walked back down instead of quietly compounding.

For the walls these rungs are climbing, see scraping without getting blocked: proxies, stealth and CAPTCHAs. For why we treat scraping cost as an architecture decision at all, see when to use managed scraping vs build your own.

FAQ

What counts as a failure that triggers escalation?

Any run that fails to produce a result, not just an explicitly blocked one. In our production data an explicit block is only about 20% of classified failures. The largest single category is a page that returns a clean 200 with nothing extractable, followed by stale selectors and navigation failures. Keying escalation on 403s alone leaves most real failures unhandled.

Why not send everything straight to the top rung?

Because just over half of production runs succeed on the free direct path, and that path produces 60% of all successful runs. Defaulting to a premium path would pay hard-target prices on an easy majority. Climbing one rung at a time keeps cost proportional to actual difficulty.

Why do upper rungs show lower success rates than cheap ones?

Selection effect. Only requests that already failed on cheaper rungs ever reach the expensive ones, so the sample gets harder as you climb. A per-rung success rate describes the difficulty of what arrives there, not the rung's capability. Comparing rungs on raw success rate without accounting for that is misleading.

Is downgrading a target's tier risky?

Not meaningfully, because escalation backstops it. A downgrade is a reversible, guarded correction rather than a permanent bet. In practice 62% of downgraded attempts return data on the cheaper rung; the rest cost one cheap failed attempt before climbing again automatically. A re-probe of 52 hard pinned targets produced 24 verified downgrades, so the upside dwarfs that cost.

Disclaimer: You are responsible for complying with all applicable laws and the terms of service of any site you scrape. Trawl is an orchestration layer for accessing publicly available data; it does not authorize access to data you do not have the right to collect. Use it for public data only, and own the legal and ethical responsibility for how you use it.

Written by Pierre | July 2026