developers

Building a Web Data Pipeline for Claude MCP

Step-by-step guide to architecting a web data pipeline that feeds structured, normalized data into Claude via the Model Context Protocol. Covers scraping, normalization, and MCP server design.

Pierre

• 10 min read
Share
Trawl banner: mcp-ai-agents grainy gradient with filet mesh

Your AI Agent Is Only as Good as Its Data

Claude can reason, plan, and synthesize. What it cannot do on its own is fetch live prices, read today's job listings, or pull the current content of any given URL. Without a data pipeline feeding it fresh context, the most capable AI agent in the world is reasoning from stale training data. B2B sales teams use this same architecture to build sales intelligence pipelines from public web data

A web data pipeline is not a scraping project. It is an infrastructure decision: you are building the nervous system that keeps an AI agent grounded in reality.

What You'll Learn

Why Web Data Pipelines Matter for AI Agents

Large language models have a knowledge cutoff. Every AI agent built on top of them inherits that limitation unless you deliberately bridge the gap. The Model Context Protocol (MCP) is the bridge: an open JSON-RPC 2.0 standard that lets Claude call external tools, read resources, and receive structured context through a unified interface.

The protocol has reached significant adoption in a short time. According to the official MCP blog, the SDK surpassed 97 million monthly downloads by early 2026, with adoption across every major AI vendor. That growth signals something real: developers are standardizing on MCP as the layer that connects AI reasoning to live data.

The missing piece for most teams is not the MCP server itself. It is the upstream pipeline that keeps the data fresh. This guide covers that pipeline: how to scrape, normalize, and serve web data in a form that Claude can actually use.

The Three-Layer Architecture: Scrape, Normalize, Serve

Production web-data pipelines for AI agents follow a consistent three-layer pattern, regardless of what tools you choose for each layer:

  • Layer 1 (Scrape): Fetch raw page content from target URLs on a schedule or on demand. Handle JavaScript rendering, pagination, and anti-bot measures at this layer.
  • Layer 2 (Normalize): Extract structured fields from raw HTML or Markdown. Apply schema validation and deduplication. Store clean records in a queryable cache.
  • Layer 3 (Serve): Expose normalized data as MCP resources or tool responses so Claude can retrieve exactly the context it needs for a given task.

Each layer has a distinct failure mode. Scraping fails on page-structure drift, bot detection, or rate limits. Normalization fails on schema mismatch, encoding issues, or missing fields. Serving fails on stale cache, schema version conflicts, or token-budget overruns in the AI context window. Isolating the layers means you can debug each failure independently.

Layer 1: Scraping Structured Data at the Source

The first decision is whether you need a full browser or a plain HTTP client. Many pages serve structured data in JSON-LD or embedded JSON blobs that are available without JavaScript execution. A lightweight HTTP fetch with CSS selector extraction runs 10 to 50 times faster than a headless browser and produces less noise in your pipeline.

When to use a headless browser

Use a headless browser when the target page loads its content via client-side JavaScript: single-page applications, infinite scroll lists, or pages that require interaction before rendering the data you need. The tradeoff is latency and fingerprinting risk. Modern anti-bot systems score browser sessions on dozens of signals; a headless Chrome instance that looks like a headless Chrome instance will get blocked.

A minimal Playwright scraper that handles a dynamic page looks like this:

const { chromium } = require('playwright');

async function fetchStructuredData(url) {
  const browser = await chromium.launch({ headless: true });
  const page = await browser.newPage();
  await page.goto(url, { waitUntil: 'networkidle' });

  // Extract JSON-LD if present
  const jsonLd = await page.evaluate(() => {
    const el = document.querySelector('script[type="application/ld+json"]');
    return el ? JSON.parse(el.textContent) : null;
  });

  // Fallback: structured field extraction
  const fields = await page.evaluate(() => ({
    title: document.querySelector('h1')?.textContent?.trim(),
    price: document.querySelector('[data-price]')?.dataset?.price,
    updated: document.querySelector('time')?.getAttribute('datetime'),
  }));

  await browser.close();
  return jsonLd || fields;
}

Respecting rate limits

Production scrapers should enforce a per-domain request budget: a max requests-per-minute, a minimum delay between requests to the same host, and a circuit breaker that pauses on repeated 429 or 503 responses. Crawling responsibly keeps your pipeline running long-term.

Layer 2: Normalization — Turning Raw HTML into MCP-Ready JSON

Raw scraper output is noisy. Pages include nav bars, footers, cookie banners, and tracking scripts that have nothing to do with the data you need. The normalization layer strips noise and shapes clean records that fit a stable schema.

Define a schema first

Start with a JSON schema or a typed interface that describes exactly what fields your AI agent needs. If Claude is answering questions about product availability, it needs: name, price, currency, availability, URL, and a timestamp. It does not need the site's full DOM. Keeping the schema tight reduces token consumption in the AI context window.

interface ProductRecord {
  url: string;
  name: string;
  price: number;
  currency: string;
  availability: 'in_stock' | 'out_of_stock' | 'unknown';
  scraped_at: string; // ISO 8601
  source_domain: string;
}

Validation and deduplication

Validate every record against the schema before writing it to the cache. Reject and log records that fail validation rather than silently storing malformed data. Deduplicate on a composite key (URL plus a time bucket) so Claude does not receive ten identical records for the same product scraped across a day of runs.

Choosing a cache store

For most pipelines, a simple key-value store with TTL is sufficient. Redis works well for sub-second latency lookups. SQLite with a lastModified index works well for lightweight single-server deployments. Avoid over-engineering the cache until you have evidence that query patterns require something more complex.

Want to skip the normalization boilerplate? Try Trawl free — scrape, schedule, and get structured JSON output ready for MCP in minutes.

Layer 3: Serving Data via an MCP Server

MCP servers expose three primitives: tools (callable functions the model invokes), resources (read-only data fetched by URI), and prompts (reusable instruction templates). For a web data pipeline, you will primarily use resources for cached datasets and tools for on-demand fetches.

Resources versus tools: when to use each

Use a resource when the data is pre-fetched and stored: a cache of product records refreshed every hour, a snapshot of job listings updated each morning, a normalized dataset that changes on a known schedule. The client fetches the resource by URI without triggering a live scrape.

Use a tool when the agent needs to trigger a fresh fetch at query time: "get the current price of this URL right now." Tools are synchronous calls that return a result; they are appropriate for on-demand lookups where freshness trumps latency.

A minimal MCP server in Python

The MCP specification supports two transport modes: stdio for local subprocess communication, and Streamable HTTP with optional SSE for remote servers. Here is a minimal stdio MCP server that exposes a normalized product cache as a resource:

import json
import sys
from datetime import datetime

def handle_request(req):
    method = req.get('method')

    if method == 'resources/list':
        return {
            'resources': [{
                'uri': 'products://cache/latest',
                'name': 'Latest product records',
                'description': 'Normalized product data, refreshed hourly',
                'mimeType': 'application/json',
            }]
        }

    if method == 'resources/read':
        uri = req['params']['uri']
        if uri == 'products://cache/latest':
            # Load from your cache store
            records = load_from_cache()
            return {
                'contents': [{
                    'uri': uri,
                    'mimeType': 'application/json',
                    'text': json.dumps(records),
                }]
            }

    return {'error': {'code': -32601, 'message': 'Method not found'}}

def load_from_cache():
    # Replace with your actual cache lookup
    return {'records': [], 'updated_at': datetime.utcnow().isoformat()}

for line in sys.stdin:
    req = json.loads(line.strip())
    resp = {'jsonrpc': '2.0', 'id': req.get('id'), 'result': handle_request(req)}
    print(json.dumps(resp), flush=True)

This is a skeleton. A production server adds input validation, error handling, authentication, and a proper cache client. The point is that the interface is simple: read from stdin, write to stdout, speak JSON-RPC 2.0.

Wiring the Pipeline End-to-End

The three layers connect through a shared cache. The scraper writes normalized records to the cache on a schedule. The MCP server reads from the same cache on demand. Claude never waits for a live scrape unless you explicitly expose a tool for on-demand fetching.

For a product-monitoring agent: a cron job triggers the scraper hourly; the scraper validates and writes records to Redis with a 2-hour TTL; the MCP server stays running alongside your AI host; when Claude needs product context it calls the resource URI and reads from Redis. Claude never blocks on a live HTTP request. The scraper and normalizer run asynchronously in the background.

If you want to learn more about connecting Claude to live web data via MCP at a higher level, Ground Claude with Live Web Data via MCP covers the full conceptual model and tool selection guide that underpins this pipeline architecture.

Scheduling and Freshness: Keeping the Cache Current

Cache freshness is the most common failure mode in production. The scraper runs, the cache fills, then the target site changes its page structure and silent normalization failures mean Claude is reasoning from weeks-old data. Three signals to monitor: schema validation failure rate (a spike means the site changed structure), field completeness per required field (gradual decay signals a partial redesign), and TTL expiry rate (how often queries hit stale entries).

Not all data has the same freshness requirement. Prices on high-velocity pages may need hourly scrapes. Job listings may be fresh enough on a daily run. Build per-data-type scheduling policies rather than running everything on the same cadence. This reduces server load and keeps your pipeline under request budget limits.

Error Handling and Observability

Scraping pipelines fail silently by default. The scraper returns empty, the normalizer writes zero records, the cache TTL expires, and Claude starts producing confident-sounding wrong answers. The fix: fail loudly at every layer. The scraper should throw on unexpected empty responses. The normalizer should reject and log validation failures, not coerce them. The MCP server should return a structured error when the cache is empty rather than an empty dataset that looks valid:

# MCP server: explicit empty-cache error
if not records:
    return {
        'error': {
            'code': -32000,
            'message': 'Cache empty',
            'data': {
                'last_successful_scrape': get_last_scrape_timestamp(),
                'hint': 'Run the scraper manually or check scraper logs'
            }
        }
    }

Log a structured event for every scrape run: URL, status code, record count, validation failure count, and duration. These logs become your debugging surface and freshness dashboard in one.

Tools and Resources

The three-layer architecture described here is framework-agnostic. These are the categories of tools you will need for each layer, along with representative examples:

  • Trawl — a managed scraping platform that handles scheduling, retries, structured output, and MCP-ready JSON normalization. Useful when you want to skip building and operating the scraping and normalization layers yourself.
  • Playwright / Puppeteer — open-source headless browser libraries for Layer 1 when JavaScript rendering is required. Battle-tested and well-documented.
  • FastMCP (Python) — a lightweight library for building MCP servers faster, with tool and resource decorators that reduce boilerplate for Layer 3.
  • Redis — low-latency key-value cache for the normalization layer. Native TTL support and simple pub/sub for triggering downstream refresh logic.
  • Zod (TypeScript) / Pydantic (Python) — schema validation libraries for Layer 2. Enforce your data contract at write time, not at query time.

Key Takeaways

  1. A web data pipeline for Claude MCP has three distinct layers: scrape, normalize, serve. Keep them separated so each can be debugged and scaled independently.
  2. MCP resources suit pre-fetched cached datasets. MCP tools suit on-demand live fetches. Choose based on the freshness requirement of the data, not the convenience of the implementation.
  3. Define your normalized schema before writing any scraper code. The schema is the contract between your pipeline and your AI agent.
  4. Scraping pipelines fail silently. Instrument each layer with explicit error logging, schema validation failure rates, and cache freshness metrics.
  5. Adaptive scheduling (per data type, not uniform cadence) reduces load and keeps freshness where it matters without over-scraping low-velocity sources.
  6. The MCP server should return structured errors when the cache is empty rather than letting Claude reason over missing data.
  7. The most common production failure is stale cache caused by silent normalization failures after a target site redesign. Monitor field completeness, not just scraper uptime.

If you want to skip building the scrape and normalization layers from scratch, Trawl handles both and returns MCP-ready structured JSON on a schedule you define.

FAQ

What is a web data pipeline for Claude MCP?

A web data pipeline for Claude MCP is a three-layer system: a scraper that fetches raw page content on a schedule, a normalizer that extracts and validates structured fields, and an MCP server that exposes the cleaned data as resources or tools for Claude to consume. The pipeline keeps Claude grounded in live web data without requiring it to make real-time HTTP requests.

What is the difference between an MCP resource and an MCP tool?

An MCP resource is a read-only data source fetched by URI, typically backed by a pre-populated cache. An MCP tool is a callable function the model invokes to perform an action or retrieve on-demand data. For web data pipelines, use resources for scheduled batch data (prices refreshed hourly, job listings updated daily) and tools for on-demand fetches where the freshness window is minutes, not hours.

Do I need a headless browser for every scraping use case?

No. Many pages expose structured data in JSON-LD script tags or embedded JSON blobs that are accessible with a plain HTTP GET. Use a headless browser only when the target page requires JavaScript execution to render the content you need. Plain HTTP fetches are 10 to 50 times faster and generate less noise in your pipeline logs.

How do I handle target site structure changes in production?

Monitor your schema validation failure rate per data source. A sudden spike in validation failures almost always means the target site changed its HTML structure or removed a field. Log the raw failed records, alert on the anomaly, and update your extractor selectors. The key is catching the failure quickly rather than letting Claude reason from stale data for days.

What transport should I use for my MCP server?

Use stdio transport if your MCP server runs as a local subprocess alongside Claude Desktop or a local AI host. Use the Streamable HTTP transport (with optional SSE) if your server needs to handle multiple clients, run as a standalone service, or scale horizontally. The MCP spec describes both in detail at modelcontextprotocol.io.

How should I size my cache TTL for different data types?

Match TTL to the real-world change velocity of the data. Prices on high-velocity retail pages change frequently and benefit from hourly scrapes with a 90-minute TTL. Job listings change on a daily or weekly basis and are fresh enough with a 24-hour TTL. Using a single TTL for all data types is the most common over-scraping mistake in early-stage pipelines.

How do I keep MCP context windows from getting too large?

Design your normalized schema to include only the fields Claude needs for its task. If the agent answers availability questions, it needs name, availability status, and URL. It does not need every product attribute on the page. Tight schemas reduce token consumption in the AI context window and also make validation failures easier to spot.

Is this architecture specific to Claude or does it work with other AI hosts?

The pipeline architecture is host-agnostic. MCP is an open standard supported by Anthropic, OpenAI, Google, Microsoft, and AWS as of early 2026. Any AI host that implements the MCP client spec can consume your server's resources and tools. Building your pipeline against the MCP interface means you can swap AI hosts without rewriting the data layer.

Disclaimer: Trawl provides scraping infrastructure. Users are responsible for ensuring their use complies with applicable laws and website terms of service. This article is for educational purposes only.

Written by Pierre | August 2026