Trawl 3.1: Your Scrapers Now Repair Themselves
Websites change. Now an AI agent explores the site, repairs your scraper and only ships the fix if your data comes back complete.
Trawl already repairs your scrapers on its own when a website changes. With 3.1, that repair gets a brain. An AI agent now goes back to the site, explores it the way you would, finds exactly what moved and fixes it. And it only ships the fix when your data comes back complete.
In short, this release brings:
- 🕵️ An agent that explores the site: it clicks through to the right page and tries its guesses before it relies on them.
- 🛡️ Repairs that never cost you data: if a fix loses even one field you had, Trawl throws it away.
- 🌍 Better first runs: Trawl remembers what worked on each site and shows you the local version of the page.
Live now on the web app, the API (v3.1.0) and the CLI (3.12.5). Nothing to turn on: every account has it.
An agent that explores the site for you
Ask Trawl for "the price of every product in this category" and the price often isn't on the page you gave it. It sits one click away, on each product page. The agent now opens them, reads them, and builds your scraper from what it actually saw.
It also checks its own work. Before it relies on a guess, it tests that guess on the page in a fraction of a second. So when your scraper arrives, it has already been tried for real, and Trawl tells you so.
For developers: the tools, the budget and the verified flag
- Tools: the agent chooses its next step each turn. It can test a CSS selector against the page it holds, navigate to another page of the same site, or run a real dry run on a worker.
- Budget: six real dry runs per session (up from four), one deadline for the whole session, and the model's full context window, with older pages compacted as it goes.
- Same agent, two jobs: the wizard that creates a scraper and the automatic repair that fixes one both run on it.
trawl create https://books.toscrape.com \
--prompt "every book title and price on page one" --json{
"success": true,
"historyId": "6ab9e1d4c2a7f3180e5b4d27",
"verified": true,
"iterations": 2,
"scrap": {
"_id": "6ab9e1ccc2a7f3180e5b4d1f",
"title": "Books to Scrape",
"cron": "0 7 * * *"
}
}verified is true only when the code that shipped is the code that passed its own dry run. iterations counts the dry runs it took. A deploy script can branch on verified before it trusts the first rows.
A repair that never costs you data
A scraper that fixes itself is only useful if you can trust the fix. So every repair Trawl ships follows one simple rule: your data must come back at least as complete as the last time it worked.
If yesterday's run gave you names, prices and stock levels, a repair that brings back names and prices but empty stock levels is rejected, even if everything else looks perfect. And a page that only pretends to answer (a "prove you are human" screen, for example) never counts as a success. You keep your data, or you keep looking. Never a silent hole.
For developers: how a repair is judged
- Same bar as production: a candidate must pass the same quality checks as a real scheduled run.
- Field regression guard: a field filled by the last successful run may not come back empty.
- Walls are never green: an anti-bot challenge is never counted as a verified run, whatever HTTP status it carries.
- Out of budget: the agent ships its last dry run that passed, never its last untested idea.
Better first runs, on every site
Some sites let anyone in. Others need more care. Trawl now remembers what worked for each site and starts there next time, so a new scraper on a site you already use gets going faster.
Runs also go through the country of the site when it matters, so a shop that prices by region shows you the local page and the local prices.
For developers: routing changes
- The wizard starts at the proxy tier that last got through on that domain.
- The proxy provider that works for a domain is remembered, and every tier has a backup provider for when the first one refuses a target.
- When a challenge must be solved, the solver uses the same browser identity and exit address as the run itself.
See it for yourself: describe a page in one sentence and let the agent do the rest. Try Trawl free.
Also new
- 🌐 A homepage with proof: trawl.me now shows a dated proof band with a three-month trend, and developers land straight on
trawl create. - 💳 Full usage transparency: your usage shows every attempt a run makes, repair test runs and failure analysis included.
- 🔒 Hardened by default: the CLI stores your login with owner-only file permissions.
Websites will keep changing. Let Trawl handle it: it explores the site, repairs the scraper and checks that nothing was lost, while you do something else. Give it a site that keeps breaking and see what comes back.
Written by Pierre | September 2026