Trawl Now Catches Stuck Scrapers, Not Just Failed Runs

A scraper that keeps failing now stops itself

One failed run tells you something went wrong that morning. It cannot tell you whether the scraper has stopped working for good. Trawl now tells those two apart.

If a scraper fails three runs in a row, Trawl treats it as stuck. It stops running it on schedule and sends you an email. The card says so too, so you can see at a glance which scrapers had a bad day and which ones need you. Live now as API v2.4.0 and web app v2.10.0.

Three failed runs in a row is the whole rule

There is no score to read and no setting to tune. Three failures in a row, and the scraper counts as stuck. One failure is a bad day. Three is a pattern. Runs you start by hand count too, and a run that is still going is simply not counted yet.

After that, Trawl stops waking the scraper up on its schedule until it works again. Nothing is deleted or switched off. Your schedule is still there, and you can run the scraper by hand at any time, so you can test a fix straight away. Every couple of weeks Trawl also lets one scheduled run through on its own. If the site has gone back to normal, the scraper starts working again by itself, with nothing for you to reset.

An email when it happens, not one every week

A stuck scraper emails its owner, and that does not turn into a weekly nag. Its scheduled runs have already stopped, so the only runs left to fail are the occasional check-ups Trawl does on its own. You get told when it happens, then roughly once a fortnight for as long as it stays broken, and nothing at all once it recovers. Alerts still follow the setting on each scraper, and one run never sends you two emails.

See what your own scrapers' status actually looks like: try Trawl.

A run stopped by site defences now tells you

Not every failure is something Trawl can repair for you. When a site's defences stop a run, or the page never loads at all, there is nothing in your setup to correct, so the automatic repair step stands aside instead of guessing. That is the outcome it reaches most often: 46 of the 141 times it ran over 60 days.

Both of those now send you the normal failure email, so the runs that genuinely need a decision from you are the ones that reach your inbox. The rest stay quiet, because they usually clear on their own. For the fuller picture of why scrapers break, we wrote up what actually breaks a scraper in production.

The icon on the card now says stuck

The card has always shown you how the last run went. It now has one more thing to say, and hovering it gives you the reason in plain words: "Failed its last 3 runs in a row. Trawl skips its scheduled runs until one works again."

It appears as soon as the third failure lands, not at the next scheduled run. A scraper you check weekly is flagged the same day, not a week later. It reads the same whether the scraper failed three times or thirty. What matters is that it needs you. Screen readers announce it too.

If you build on the API

Skip this part if you only use the app. The previous release let you watch a scraper get built before you signed up. This time, the health signal the app uses is available to your own tools as well.

🔗 API · The last few runs come with the list: GET /api/scraps now returns the four most recent runs for each scraper, so your own dashboard can work out which ones are stuck without asking a second time.

🔗 API · Ask for the failing ones only: add ?minConsecutiveFailures=N and you get back only the scrapers that failed their last N runs, each one telling you how many failures in a row, and since when. Enough for a morning check that lists only what needs you.

Three runs deep, using an API key:

$ curl -H "Authorization: Bearer $TRAWL_API_KEY" \
    "https://api.trawl.me/api/scraps?minConsecutiveFailures=3"

{
  "type": "success",
  "message": "scrap list",
  "data": [
    {
      "title": "Competitor room rates",
      "consecutiveFailedRuns": 3,
      "unhealthySince": "2026-08-20T06:00:07.903Z",
      "lastCronOutcome": "skipped_unhealthy",
      "history": [
        { "status": false, "createdAt": "2026-09-03T06:00:09.114Z" },
        { "status": false, "createdAt": "2026-08-27T06:00:11.482Z" },
        { "status": false, "createdAt": "2026-08-20T06:00:07.903Z" }
      ]
    }
  ]
}

The auto-fix switch now means only what you chose

The auto-fix switch on a scraper now means one thing: whether you want Trawl to try repairing it when it breaks. Trawl's own safety limits, like pausing repairs that keep failing, are worked out fresh on each run instead of being stored on that switch.

So the switch stays where you left it, and the "Fix this" button still gives you an attempt even while automatic repairs are paused. When we moved existing scrapers over, all 23 that had been switched off by hand kept their owner's setting.

Also in this round

• ⚙️ Platform · Clearer failure reasons: a failed run now names what actually went wrong. A page that never loaded says so, instead of being reported as an empty result, and a site's challenge page is named as a block. Trawl uses that label to decide how to approach the site next time.
• ⚙️ Platform · Retention: activity records and webhook deliveries are now kept for 365 days. Nothing on the platform was old enough to be removed when this shipped.
• 🌐 Web app · Phones: the home page and the pop-up windows work properly on a phone. Dialogs fill the screen instead of sitting in a small box, and the animated fish step aside on narrow screens.

A scraper that breaks is not the hard part. A scraper that breaks and keeps running is, because it quietly costs you time and money every week while telling you nothing new. Trawl now handles that one itself: it notices, it stops, and it tells you, which leaves you the only part that needed a person.

Written by Pierre | September 2026