Trawl v3: The Wizard Looks at the Page First

The wizard looks at the page before it writes the scraper

Describe what you want in one sentence and Trawl now opens the target first, reads the page that actually renders, and writes the extraction against what is really there. Live as API v3.0.0, web app v3.0.0 and CLI 3.12.4.

Scraper generation moves from informed guesswork to something much closer to observation, and the app and the CLI both get it, because both call the same wizard.

From one sentence to real rows

The shortest path is a single command. You give a URL and a goal in plain language; the wizard generates the scraper, persists it, runs it once for real, and hands back the result:

trawl create https://books.toscrape.com \
  --prompt "every book title and price on the first page" --json

{
  "success": true,
  "scrap": {
    "_id": "6aa1f0c2d1b8e4470f3c9a12",
    "title": "Books to Scrape",
    "cron": "0 7 * * *",
    "cronTimezone": "UTC"
  },
  "historyId": "6aa1f0c9d1b8e4470f3c9a1e"
}

That scraper is persistent and scheduled from the moment it exists, so a deploy script can create a watcher for a competitor page and read trawl data <id> every morning without a browser ever opening again.

Climbing until the page renders

Before generating anything, the wizard probes the target through the proxy tier ladder, one rung at a time, and stops at the first rung that returns a rendered page. The scraper is written against that document and inherits the tier that worked, so the very first real run starts on a route already proven to get through. Every account gets the climb.

Generate, dry run, observe, regenerate

One page is rarely the whole job. A search box, a click, a results page: goals like that need the wizard to see more than the entry URL, and now it does. Generation is a bounded loop. It writes a candidate, dry runs it on a real worker at the real proxy tier, reads what came back, and writes the next candidate against that observation rather than against a guess. It stops when the data arrives or the budget runs out, and what it learned on the way is what the scraper ships with.

Describe a page in one sentence and see what comes back: try Trawl.

Repair that builds on what it learns

Automatic repair has been rewriting broken scrapers since v2.0. This release gives it a memory inside its own attempt, and a much better view of the page it is reasoning about.

• ⚙️ Platform · The whole page, with the credentials stripped out: repair reasons over up to 220,000 characters of live markup, and it is not a blunt dump. On the way into the prompt, Bearer tokens, API keys, cookie headers, basic-auth URLs and JWTs are redacted, and the redaction keeps the surrounding shape, so apiKey="<REDACTED>" is still a readable attribute. The model gets the page's real structure and never sees a secret. Longer than the budget, and the cut lands on a line boundary, never mid-tag.
• ⚙️ Platform · One page budget for generation and repair: the wizard's probed page and the repair snapshot pass through the same sanitize-and-truncate choke point, so a page too large for one is too large for the other, identically, and a single switch governs both. A rule computed twice drifts; computed once, it cannot.
• ⚙️ Platform · Each candidate inherits what the last one saw: every repair candidate is dry run on a real worker at the scraper's real proxy tier, and the result of that run feeds the next candidate. Round three knows what rounds one and two learned.
• ⚙️ Platform · Dead browser APIs are corrected, not discouraged: a script reaching for a Puppeteer method that no longer exists is rewritten by a deterministic codemod. A rule in a prompt is advice; a codemod is a guarantee.
• ⚙️ Platform · The safety breaker reopens on its own: repeated failed repairs pause automatic fixing on a scraper, and the pause lifts after 30 days, so a scraper gets a fresh chance once the site has moved on.
• ⚙️ Platform · Only verified knowledge is kept: what a repair learns reaches the knowledge base once a re-run has confirmed it, and model-written scripts are tagged, so generation quality is measured rather than assumed.

Where you still steer

Reading the page first retires a whole class of failure, the selector written for a page nobody opened. Two things stay yours to judge: whether a site that serves its list from an authenticated background call is worth an account, and whether a target behind a hard wall is worth the tier it costs. Trawl surfaces both rather than quietly burning your budget on them, and the full map of what breaks a scraper is here.

Also in this round

• 💳 Billing · Pending cancellations are visible: a subscription set to end at the period date now says so on the subscriptions panel, so you can see what you are still paying for and until when.
• 💳 Billing · Plans and extras have their own tabs: the pricing page is built from configuration, each tab carrying its own content, so the catalogue can grow without a page rewrite.
• 🌐 Web app · The usage gauge reads your quota: the gauge and the weekly meter answer one question, how much of your quota is left, so a glance is enough before launching a heavy run.

Look, write, run, look again. That loop is the whole of this release, applied at creation by the wizard and at repair by the fixer, and it is the same instinct that made the last one name a login page instead of blaming a selector. Code written against a page someone actually opened is simply better code, and the machine can do the opening.

Written by Pierre | September 2026