Why We Run Scrape Scheduling on Mongo, Not Kafka

Reaching for Kafka here would have been the mistake. When a scheduled scrape fires at Trawl, exactly one worker has to pick it up, run it, and record the outcome, even when several replicas could each see the same due job at the same instant, even when a pod restarts mid-run. That is a coordination problem, and the architecture-diagram reflex says coordination needs a message broker. Kafka. RabbitMQ. Redis with a queue library bolted on top. We did none of it. Our scheduling coordinates through MongoDB (the database that was already in our stack) with one atomic operation doing all the heavy lifting. This is the story of why, what broke to force the decision, and the honest trade-offs we took to keep our infrastructure boring. ## The instinct that almost cost us The architecture-diagram instinct goes: scheduled work needs a queue, queues need a broker, brokers are a solved problem. It feels rigorous. It feels like what a "real" backend does. And it optimizes for a number we don't have. Brokers earn their keep at high message rates: thousands of messages per second, fan-out across consumer groups, replayable logs measured in terabytes. Trawl runs scrapes on the order of tens a day, occasionally a burst into the low hundreds when many schedules land together. Not thousands per second. Per day. **At that volume, broker-class throughput isn't a feature you buy. It's a capability you pay to keep idle.** And the payment is real. Adding a stateful broker to a **solo-operated, single-server setup** means a new thing that can fail at 3 a.m. A new thing to monitor, patch, back up, and reason about mid-incident. Dan McKinley's [_Choose Boring Technology_](https://mcfunley.com/choose-boring-technology) framed it precisely: every team has a finite number of "innovation tokens," and the true cost of a choice is its lifetime operational maintenance minus the velocity it buys you. A broker we didn't need would have spent a token on negative velocity. Pure ops tax, zero product gain. So the real question was never "Kafka or RabbitMQ or Redis?" It was: **does our actual coordination need justify a new stateful dependency at all?** It didn't. ## The incident that forced the decision We didn't arrive here from first principles alone. We arrived here because something broke. On 2026-04-18, eighteen scrapes were all scheduled "daily", and "daily" meant the same cron expression, `0 0 * * *`, midnight UTC. So all eighteen fired within roughly 700 milliseconds of each other on a single worker pod. The browser layer choked under that many near-simultaneous page loads, and several runs died with navigation timeouts. The symptom was the classic thundering herd, with one twist: the herd wasn't volume piling up over time. It was a fan-out at a single instant, every job sharing the same trigger. **The post-mortem made the requirement embarrassingly clear: we didn't need more throughput, we needed the opposite.** Spread a cluster of identical triggers across a window. And make sure that even if two code paths both think a scrape is due, only one actually runs it. The problem was never "move messages faster." It was "stop starting browsers all at the same second." ## What we actually built The design is deliberately unremarkable, and it has two parts. **First, de-cluster the triggers.** We locked scrape schedules to a small enum of presets (every 30 minutes, hourly, every 6/12 hours, daily, weekly, monthly) instead of arbitrary cron expressions. Each preset carries a per-preset jitter window, so jobs sharing a preset spread across a safe slice of their interval instead of all firing on the same tick. The same-second fan-out that caused the incident is now mathematically bounded. **Second, make "exactly one runner" a property of the database.** A scheduled scrape can plausibly be picked up by more than one place: the cron tick, an API-triggered run, a replica that just restarted. To guarantee a given scrape only runs once at a time across all of them, each run path must first acquire a per-scrap **run-lock**. The lock is one atomic `findOneAndUpdate`: ```js // Acquire a run-lock on a scrap. Multi-replica safe: only one caller // across all pods gets `true` until the lock is released or its TTL lapses. const result = await Scrap.findOneAndUpdate( { _id: scrapId, $or: [ { runLockedAt: null }, // never locked { runLockedAt: { $exists: false } }, // legacy doc, no field { runLockedAt: { $lt: staleBefore } }, // stale → TTL recovery ], }, { $set: { runLockedAt: now } }, // claim it, stamp the lease { new: false, projection: { _id: 1 } }, ); return Boolean(result); // truthy = this caller won the race ``` That is the one piece of real engineering, and it's a few lines long. The match condition and the `$set` happen as a single indivisible operation, so two callers can never both walk away holding the lock. The database arbitrates the race for us, no lock service required. This is exactly the [well-trodden MongoDB primitive](https://oneuptime.com/blog/post/2026-03-31-mongodb-job-queue/view): atomic compare-and-set, a lease timestamp, reclaim on expiry. We didn't invent it. We reused a primitive our codebase **already trusted elsewhere**, which means zero new operational surface and zero new failure modes to learn. That single atomic lock buys us more than mutual exclusion: - **Crash-safety.** The lock carries a lease (a visibility timeout). If the worker dies mid-run (pod restart, OOM, whatever), the lease goes stale and the next tick can reclaim it. The lock is released in a `finally` block so a clean crash frees it immediately; the TTL is the backstop for an unclean one. Nothing stays pinned forever. - **A quota gate in front.** Before any run consumes compute, a billing-quota check runs, so an exhausted account is denied *before* a browser starts, not after. - **A path to scale.** The same Mongo lock that protects one replica protects N. The day volume justifies more workers, the lock already makes that safe. ## The decision, on the axes that mattered We scored the choice on the dimensions that actually applied to a solo single-server operation, not the dimensions a broker is built to win. | Axis | Broker (Kafka / Rabbit / Redis) | Mongo-backed coordination | |---|---|---| | **Ops burden** | New stateful service to deploy, monitor, patch, back up, and debug on-call | Reuse the database already in the stack, no new surface | | **Throughput needed** | Tuned for thousands of msgs/sec | We run tens of scrapes/day. Broker throughput sits idle | | **Durability** | Replicated commit log, replayable | Durable Mongo document + lease/reclaim on crash | | **Dependency cost** | A permanent operational tax with no product gain at our volume | Marginal: one field, one atomic op | | **Verdict** | Wins a race we're not running | **Wins on every axis that applied** | Mongo doesn't win because it's the better queue in the abstract. At scale, a broker is purpose-built and would pull ahead, no contest. **It wins because it matches our volume and our ops budget, and the best architecture is the one that fits the problem you actually have.** ## What we actually validated After the fix, we confirmed the two things that mattered. The preset-plus-jitter change de-clusters identical schedules: a batch that previously fired in one ~700 ms spike now spreads across its jitter window, so the browser layer never sees the simultaneous load that caused the navigation timeouts. And the run-lock holds across replicas: a scrape that two paths both consider due is acquired by exactly one, the other gets a falsy result and stands down. Those are the claims the incident demanded, and those are the ones we checked. Not a job-queue cycle we never built. ## The honest residuals A credible build post names what it didn't solve. - **At-least-once, not exactly-once.** On a true pod crash after a run starts but before it completes, the lease-reclaim can re-run a scrape. This is rare and bounded, and our run paths are built to tolerate it: we avoid double-charging on a retry, for instance. But we don't pretend it's exactly-once. It isn't. - **The lock is per-scrap, not a global concurrency cap.** Today's lock guarantees a single scrape can't run twice at once; it does not, on its own, cap the *total* number of browsers across all scrapes. The jitter window is what keeps simultaneous starts in check. A true global admission cap is work we've scoped but not shipped, and we'd rather say so than imply a ceiling that isn't there yet. ## FAQ **Can MongoDB really be used to coordinate jobs?** Yes, for the right scale. An atomic `findOneAndUpdate` with a lease timestamp gives you compare-and-set, mutual exclusion, and crash recovery using storage you already run. It's a well-documented pattern, not a hack. It is not a replacement for a broker's throughput. It's the right tool when your volume is low and your ops budget is tight. **When should you actually reach for Kafka?** When throughput, fan-out across many consumer groups, or a replayable event log are genuine product requirements, typically thousands of messages per second and up. If you're processing tens or low hundreds of jobs a day on a single server, a broker is capacity you pay to keep idle. **Is the atomic-claim pattern safe with multiple workers?** That's the whole point. Because the find-and-set is one indivisible Mongo operation, only one caller across all replicas can win the lock for a given record at a time. The others get a falsy result and skip. The lease/TTL then recovers the lock if the winner crashes. **What is a lease (visibility timeout)?** A lease is a timestamp stamped when a worker takes the lock, after which the lock is considered abandoned. If the worker finishes, it releases the lock; if it dies, the lease eventually lapses and another worker may reclaim the record. It's how you get crash-safety without a separate lock service. ## Key Takeaways - **Match the tool to your real volume, not the diagram.** Tens of scrapes a day is not a broker problem; treating it like one buys idle capacity at a permanent ops cost. - **A new stateful dependency is a recurring tax, not a one-time purchase**, especially on a solo, single-server setup where every service is yours to babysit. - **An atomic `findOneAndUpdate` lock turns an existing database into a crash-safe coordinator:** mutual exclusion and a lease, arbitrated by storage you already run. - **De-cluster shared triggers with jitter.** Our incident wasn't volume. It was many jobs sharing one cron tick. A preset enum plus a jitter window fixed the root cause directly. - **Name your residuals.** At-least-once delivery and a per-scrap (not yet global) lock are honest costs we chose with eyes open. The lesson isn't "never use Kafka." It's that the most rigorous-feeling choice is often the most expensive one in disguise. Pick the boring tool that fits the load you actually have, and spend your innovation tokens where they move the product, not on infrastructure nobody asked for. If you want more on how Trawl thinks about infrastructure and reliability, see [BYOI: bring your own infrastructure for scraping](https://blog.trawl.me/byoi-explained-bring-your-own-infrastructure-for-scraping/) and [self-healing scrapers: why AI fixes beat retry logic](https://blog.trawl.me/self-healing-scrapers-why-ai-fixes-beat-retry-logic/). > **Disclaimer:** You are responsible for complying with all applicable laws and the terms of service of any site you scrape. Trawl is an orchestration layer for collecting **publicly available data**; how you use it (including which targets you point it at and whether you have the right to access that data) is your responsibility, not ours. Scrape public data only, and respect target sites' ToS. *Written by Pierre | June 2026*