Beating Anti-Bot Walls

We Were Wrong About Our Own Walls, Four Times

Every guide to beating anti-bot systems is written from the outside: here are the detection layers, here is the countermeasure for each, good luck. We have written one of those too, and it is useful.

This is the other kind. This is what we found when we stopped reading about walls and audited what our own fleet believed about them. Our tooling was confidently wrong in four separate ways, and each error made the wall look like something it was not.

Before any of the hard engineering, the honest problem is that your success metric may be lying to you. You cannot beat a wall you are measuring incorrectly.

Here is the number that frames the rest. When we classified every failed run over ninety days of production traffic, this is what "blocked" actually turned out to be:

An explicit block is one failure in five. If your escalation logic keys on 403s and challenge pages, you are reacting to a fifth of the ways a scrape actually dies and silently eating the rest. Everything below is how we found that out.

What You'll Learn

• The detector that cried wolf, with the code
• The evidence we were deleting
• The failure class that did not exist
• The wall that alternates, and how many runs prove a fix
• Five checks you can run on your own fleet
• Key Takeaways
• FAQ

The Detector That Cried Wolf

Our block detector looked for a vendor's client-side script in the response. The reasoning felt airtight: that script belongs to a well-known anti-bot product, so if it is present and the extraction came back thin, the page was probably a challenge.

The flaw is that the script ships on every page that vendor serves. It is on the challenge page. It is also on the perfectly ordinary product page you just scraped successfully, sitting there passively, doing nothing.

Roughly, the check was this:

// BEFORE: fires on any page the vendor serves, challenge or not.
const isBlocked = html.includes('challenge-platform');

So it fired on both. Every correctly served page behind that vendor was liable to be classified as blocked, and the block rate for that vendor approached 100% by construction.

The fix is to require evidence that a challenge ran, not that a vendor exists:

// AFTER: require a signal that only a real challenge produces.
const vendorPresent = html.includes('challenge-platform');
const challengeRan =
  status === 403 ||
  /\bcType\s*:\s*["'](managed|interactive|non-interactive)["']/.test(html) ||
  html.includes('cf_chl_opt');            // challenge bootstrap payload

const isBlocked = vendorPresent && challengeRan;

Same idea generalises to any vendor: Cloudflare, DataDome, Akamai and PerimeterX all ship a script on protected pages whether or not they decided to stop you. A detector keyed on the presence of a defense measures which vendor the site bought, not whether you were blocked.

The uncomfortable part is how long it survived. A metric that says "this target is very hard" is rarely challenged, because it agrees with what everyone already believes about anti-bot systems. Wrong numbers that confirm your priors are the ones that live longest.

The Evidence We Were Deleting

The second error is worse, because it made the first one hard to catch.

When a run failed, our tooling read one field: the error snapshot attached to the failure record. Reasonable. Except the real page, the actual DOM the wall served, was not in that field. It lived in a separate collection of run snapshots, and a retention setting was quietly evicting older ones to save storage.

So the workflow for investigating a block was: open the failure, read a field that did not contain the wall, form a theory, move on. The actual evidence had either never been consulted or had already been deleted. We spent real time theorising about pages we had never looked at, and two conclusions from that period were later shown to be wrong.

Before you theorise about a wall, open the page the wall actually served you. If you cannot, that is your first bug and it outranks the wall. Concretely, every failed run should retain:

{
  status,                    // the HTTP status, not just ok/not-ok
  bodyBytes,                 // size, for anomaly detection
  bodySnapshot,              // the actual HTML. this is the one people skip
  finalUrl,                  // after redirects
  exitRegion,                // where the request left from
  claimedLocale,             // what the browser said it was
  extractedRowCount,         // 0 is a result, not an absence
  failureKind,               // your classifier's verdict
  config,                    // proxy tier, fingerprint profile, headers
}

Then check what your retention does to it. The runs you most need to inspect are, by definition, old enough to have been cleaned up.

The Failure Class That Did Not Exist

Here is a diagnostic worth stealing, because it generalises well past scraping.

We noticed that one vendor's challenge had never once appeared in our failure statistics. Not rare. Zero. For a widely deployed product, across a fleet touching a lot of the web, zero is not a plausible number.

Two explanations exist for a category that never appears: it genuinely never happens, or something upstream consumes it before it can be counted. It is almost always the second.

It was. That challenge is served with HTTP 200 and a body that looks, to a naive check, like an ordinary page. Our classifier had an earlier, broader rule that matched first and filed the run under a generic failure. The specific detector downstream was unreachable code. Its category stayed permanently empty, and the runs it should have owned were attributed to a different cause, one that implied a different fix.

The audit is cheap. Count every failure kind your classifier can emit, over a window, and compare against the list of kinds it is able to emit:

// Every failure kind actually observed, last 30 days.
db.runs.aggregate([
  { $match: { finishedAt: { $gte: since }, ok: false } },
  { $group: { _id: '$failureKind', n: { $sum: 1 } } },
  { $sort: { n: -1 } },
]);

// Then, the part that matters:
const observed = new Set(rows.map(r => r._id));
const emittable = classifier.kinds();          // every branch that can fire
const never = emittable.filter(k => !observed.has(k));
// `never` is a list of hypotheses about your classifier, not facts about the web.

Any failure class with a count of zero is a claim that needs testing. In an ordered classifier, an earlier broad rule silently starves every narrower rule behind it, and nothing in your logs will tell you.

Note the shape of the trap too: HTTP 200. Anything that treats a 200 as success will score a challenge page as a win with zero extracted rows, which is the quietest possible failure. Recall the table at the top: an empty result on a clean 200 was our single largest failure category, at 25.8%.

The Wall That Alternates

The last one is why we are careful with the word "solved".

We built detection for a specific challenge type, deployed it, and watched a request sail through. Solved, surely. Then the same target, same configuration, returned a hard denial. Then it passed again.

The wall alternates. Sometimes it serves an interactive challenge, sometimes it refuses outright, and which one you get is not a clean function of anything we controlled.

This breaks the most natural way to validate scraping work: change something, run it once, it works, call it fixed. Against a target that passes half the time, one run has a 50% chance of telling you the wrong thing. The arithmetic is unforgiving:

// If a wall passes at rate p by chance alone, the probability that
// n consecutive successes happen without your fix doing anything:
//   p^n
// For p = 0.5:  1 run = 50%,  3 runs = 12.5%,  5 runs = 3.1%
//
// So: five consecutive passes before you believe a fix on an
// alternating target. One pass is a coin flip you reported as a result.

We got a false positive from exactly this and nearly shipped a conclusion off it. Against an adaptive target the unit of proof is a pass rate over repeated attempts, ideally against a control.

Five Checks You Can Run on Your Own Fleet

Four errors, one shared root: every one was a measurement problem wearing the costume of a scraping problem. These are the checks that would have caught all four.

1. Detect behaviour, not vendor presence. Require evidence that a challenge executed, as in the code above. If your block rate for one vendor is near 100%, suspect the detector before the vendor.

2. Never treat HTTP 200 as success. Validate the shape of what came back:

function looksLikeContent(res, expect) {
  if (res.status !== 200) return false;
  if (res.body.length < expect.minBytes) return false;   // block pages are small
  const rows = extract(res.body);
  if (rows.length < expect.minRows) return false;        // 0 rows is a failure
  return rows.every(r =>
    expect.fields.every(f => r[f] != null) &&
    (r.price == null || (r.price > 0 && r.price < 100000))  // plausible range
  );
}

3. Audit for empty buckets with the aggregate above. Every failure kind that has never fired is a bug hypothesis.

4. Keep the response body, and know your retention. The DOM the wall served is the only primary source. If your investigation tooling reads a summary field instead, it is reading fiction.

5. Re-measure your own routing verdicts. A tier chosen on one bad day stays pinned unless something goes back and checks it. When we audited 52 of our own routing assignments, 24 were running on a more expensive configuration than they needed, and 62% of attempts on the cheaper rung returned data normally. Those pins had been costing money for months on the strength of a verdict nobody revisited. The mechanics are in the proxy-tier ladder, and the diagnostic taxonomy is in what actually breaks a scraper in production.

Want the escalation ladder, the wall detection and the re-measurement without building them? Try Trawl free.

Key Takeaways

1. An explicit block is only 20.8% of classified failures. The largest single category, at 25.8%, is a clean HTTP 200 with nothing extractable.
2. A detector keyed on an anti-bot vendor's script measures which vendor the site bought, not whether you were blocked. Require evidence the challenge ran.
3. Wrong metrics that confirm what you already believe about anti-bot difficulty survive longest, because nobody argues with them.
4. Investigate from the page the wall served you. If retention deletes failure bodies, you are theorising about evidence you destroyed.
5. A failure category that has never once appeared usually means an upstream rule swallows it. Audit for empty buckets.
6. On an alternating target, one passing run is a coin flip. Five consecutive passes before you believe a fix.
7. Routing verdicts go stale. Auditing 52 of our own tier assignments moved 24 onto cheaper configurations.

The pattern across all four is the least glamorous lesson in scraping: most of the work of beating a wall is making sure you can see it correctly. The engineering comes after.

FAQ

Why would a block detector report false positives?

Because vendors like Cloudflare, DataDome, Akamai and PerimeterX serve the same client-side script on every page they protect, challenge or not. A detector that treats the script's presence as evidence of blocking fires on successfully scraped pages too. Key the detection on a signal only a real challenge produces: a 403, a challenge-type marker, or the challenge bootstrap payload.

Can an anti-bot challenge return HTTP 200?

Yes, and several do. The status code is not a reliable signal. Validate the body against the schema you expect: if the fields you need are missing, the row count is zero, or values fall outside plausible ranges, treat the run as failed regardless of status. In our own ninety-day sample, an empty result on a clean 200 was the single largest failure category at 25.8%.

Why does a failure category with zero occurrences matter?

Because zero is rarely true for a widely deployed defense. An empty bucket usually means an earlier, broader rule in your classifier matches first and files those runs elsewhere, leaving the narrower detector as unreachable code. Its runs get attributed to the wrong cause, which points you at the wrong fix.

How many runs prove that a scraping fix works?

On a stable target, a handful. On a target that alternates between challenge and denial, five consecutive passes gets you to roughly 3% odds that the result is chance alone, versus 50% for a single run. Measure a pass rate across repeated attempts spread over time, and compare against a control where you can.

Why re-check a routing tier that was already decided?

Because the tier was recorded on one day under one set of conditions, and nothing revisits it. Conditions change, and a single bad run can pin a job to an expensive configuration indefinitely — which is a cost bug, not a scraping one. We audited 52 of our own tier assignments and 24 ran fine on a cheaper path. Escalation remains the safety net if the cheaper rung stops working.

What should I log for every failed scrape?

Status, body size, the response body or a rendered snapshot, the final URL, exit region, claimed locale, extracted row count, the classifier's verdict, and the configuration used. The body is the one people skip and the one that settles arguments. Check how long retention keeps it: failure evidence tends to be evicted exactly when an investigation needs it.

Disclaimer: Trawl provides scraping infrastructure. The figures in this article come from that infrastructure running jobs its users configure — Trawl does not select the targets, and users remain responsible for ensuring their use complies with applicable laws, robots directives and website terms of service. The techniques described here are diagnostic: they identify when a request was refused, so that it can be recorded and handled correctly rather than repeated blindly. This article is for educational purposes only.

Written by Pierre | August 2026