developers

Ground Claude with Live Web Data via MCP

Claude is powerful but frozen in time. The Model Context Protocol (MCP) lets you wire live scraped web data directly into Claude agents, eliminating staleness and grounding every answer in current reality.

Pierre

• 12 min read
Share
Trawl banner: mcp-ai-agents grainy gradient with filet mesh

Your AI Agent Is Flying Blind

Claude knows a lot. It knows history, code patterns, reasoning frameworks, and language at a scale no human could match. But it knows nothing about what happened this morning. It cannot tell you the current price of a product, whether a job posting went live an hour ago, or what a competitor added to their website overnight. Every answer it gives without live data is a snapshot of the past. Live price data — including flight and rental price monitoring — is one of the most common grounding use cases.

An AI agent without live data is a brilliant analyst working from last year's files.

What You'll Learn

What the Model Context Protocol Actually Is

The Model Context Protocol (MCP) is an open standard introduced by Anthropic in November 2024 that gives large language models a universal, JSON-RPC 2.0-based interface for connecting to external tools, databases, and services. Think of it as a USB-C port for AI: one standard connector, any peripheral.

Before MCP, every integration between an LLM and an external data source required custom code, custom prompting, and custom maintenance. MCP replaces that ad-hoc plumbing with a defined handshake: the server advertises its capabilities, the client negotiates what it supports, and the model calls tools through a consistent interface without any bespoke glue code.

By early 2026, public MCP registries counted over 17,000 servers across databases, file systems, CRMs, catalogs, and developer environments. The protocol is now supported by Anthropic, OpenAI, and Google DeepMind. In December 2025, Anthropic donated MCP governance to the Linux Foundation's Agentic AI Foundation (AAIF), making it vendor-neutral infrastructure.

Why this matters for web data

The web is the largest real-time data source on the planet. Prices, job listings, product catalogs, travel availability, public filings: all of it changes continuously. MCP gives Claude a structured way to reach out and pull any of that data on demand, mid-conversation, mid-task, without leaving the reasoning loop. Practical use cases range from e-commerce buybox and MAP monitoring to real-time competitor tracking.

Why Grounding Matters: The Hallucination-Staleness Gap

Two distinct failure modes hurt AI agents in production. The first is hallucination: the model states a falsehood with confidence. The second is staleness: the model gives an answer that was true at training time but is no longer accurate. Both look identical to the end user.

According to a 2026 analysis of AI hallucination benchmarks, properly implemented retrieval-augmented generation (RAG) reduces hallucination rates by up to 71 percent in production systems. But RAG over a static document corpus does not solve staleness. For live web data, you need a pipeline that fetches at query time, not at index time.

The pattern that solves both problems is grounding: giving the model real, current, structured data before it generates its answer. MCP is the infrastructure layer that makes grounding composable and maintainable.

MCP Architecture: Hosts, Clients, and Servers

Understanding MCP means understanding three roles. According to the official MCP specification:

  • Host: the AI application that embeds an MCP client. Claude Desktop, Claude Code, and most agentic frameworks are hosts.
  • Client: a connection manager inside the host that maintains exactly one persistent session with one MCP server.
  • Server: the process that exposes capabilities (tools, resources, prompts) over either stdio (local) or HTTP with Server-Sent Events (remote).

The flow is: host spawns or connects to a server, client and server negotiate capabilities, model receives the tool catalog as part of its context, and then calls tools by name when it needs data. The server executes the action (scrape a URL, query a database, call an API) and returns structured results. The model never sees raw protocol traffic: it just sees tool results that arrive in context.

Local vs. remote servers

Local MCP servers communicate over stdio: the host spawns the server as a child process. This is the simplest setup for development. Remote servers communicate over HTTP + SSE: any client on the network can connect, and the server can be deployed to a cloud function or a container alongside your scraping infrastructure. For production web data pipelines, remote HTTP servers are the standard pattern.

The Three MCP Primitives: Tools, Resources, and Prompts

Every MCP server exposes up to three types of primitives, each with a standardized list-and-get/call interface.

  • Tools: executable actions with parameters. A scraping tool accepts a URL and returns page data. A search tool accepts a query and returns results. The model calls tools proactively when it needs them.
  • Resources: read-only data sources the host can inject into context. A resource might be a live database view, a file, or a pre-fetched dataset. Resources are pulled by the host, not called by the model.
  • Prompts: reusable prompt templates that encode how to use a server's capabilities correctly. An MCP server for price data might expose a prompt that instructs the model on how to interpret extracted price fields.

For a web data pipeline, you will primarily use tools: the model calls them on demand, mid-task, to fetch the specific page or dataset it needs right now. Resources are useful for injecting a pre-crawled corpus. Prompts are useful when you want to encode extraction logic in a reusable template.

Building a Web Data Pipeline for Claude via MCP

For a focused guide on this topic, see Building a Web Data Pipeline for Claude MCP.

A production web data pipeline for Claude has four layers. Each layer is independent and can be upgraded without touching the others.

Layer 1: The scraping engine

This is the process that actually fetches pages. For static content, an HTTP client with proper headers is sufficient. For JavaScript-rendered pages, you need a headless browser. The scraping engine does not need to know anything about MCP: it just needs to return data on request.

Layer 2: The MCP server

The MCP server wraps the scraping engine and exposes it as a tool. A minimal server in Python (using the official MCP SDK) looks like this:

from mcp.server.fastmcp import FastMCP

mcp = FastMCP("web-data-server")

@mcp.tool()
def fetch_page(url: str, selector: str = "body") -> dict:
    """Fetch a URL and return text content from a CSS selector."""
    import httpx
    from bs4 import BeautifulSoup
    r = httpx.get(url, headers={"User-Agent": "Mozilla/5.0"}, follow_redirects=True)
    soup = BeautifulSoup(r.text, "html.parser")
    el = soup.select_one(selector)
    return {
        "url": url,
        "status": r.status_code,
        "content": el.get_text(strip=True) if el else "",
        "fetched_at": __import__("datetime").datetime.utcnow().isoformat() + "Z"
    }

if __name__ == "__main__":
    mcp.run(transport="stdio")

The server exposes a single tool, fetch_page, that Claude can call with a URL and an optional CSS selector to target the relevant section of the page.

Layer 3: The Claude host configuration

Register the server in your Claude Desktop or Claude Code config so it is available in every session:

{
  "mcpServers": {
    "web-data": {
      "command": "python",
      "args": ["/path/to/server.py"]
    }
  }
}

Once registered, Claude sees fetch_page in its tool catalog automatically. No extra prompting is needed to make it available.

Layer 4: The prompt design

With live data available, prompts shift from static queries to task instructions. Instead of "what is the price of X?", you write "use fetch_page to get the current price of X from this URL, then summarize the change since yesterday using this baseline value." The model plans and executes the steps.

Step-by-Step: Wire a Scraping MCP Server to Claude

Here is a concrete walkthrough for setting up a price-monitoring agent that uses Claude to track product pages and surface changes.

Step 1: Install the MCP Python SDK

pip install mcp httpx beautifulsoup4

Step 2: Define your tools

Add a second tool for structured price extraction, building on the basic fetch_page above:

@mcp.tool()
def extract_price(url: str) -> dict:
    """Extract a numeric price from a product page."""
    import re, httpx
    from bs4 import BeautifulSoup
    r = httpx.get(url, headers={"User-Agent": "Mozilla/5.0"}, follow_redirects=True)
    soup = BeautifulSoup(r.text, "html.parser")
    for sel in ["[itemprop='price']", ".price", "#price", ".a-price-whole"]:
        el = soup.select_one(sel)
        if el:
            raw = el.get_text(strip=True)
            match = re.search(r"[\d,]+\.?\d*", raw.replace(",", ""))
            if match:
                return {"url": url, "price": float(match.group()), "raw": raw,
                        "fetched_at": __import__("datetime").datetime.utcnow().isoformat() + "Z"}
    return {"url": url, "price": None, "error": "no price found"}

Step 3: Register and test

Run the server locally, add it to your Claude config, and open a new Claude session. Type: "Use extract_price to get the current price from [URL]." Claude calls the tool, receives the structured result, and incorporates it into its response. No hallucination about the price: it read it live, two seconds ago.

Step 4: Deploy as a remote server for shared access

For team use or agent workflows, switch the transport from stdio to HTTP. The FastMCP library supports this with one line change:

mcp.run(transport="http", host="0.0.0.0", port=8080)

Now any Claude host on the network can connect to http://your-server:8080. Add authentication headers in your Claude config to secure it.

Data Quality: Structured Output Over Raw HTML

The most common mistake in early MCP web pipelines is passing raw HTML to Claude and asking it to extract what it needs. This wastes context window, introduces noise, and makes the model do parsing work that a deterministic scraper does better and faster.

The right pattern: the MCP server is responsible for extraction. It returns a clean JSON object with named fields. Claude receives structured data and reasons about it, not markup.

Schema design principles

  • Return typed fields, not strings. Prices as numbers, dates as ISO 8601 strings, availability as booleans. Let Claude do semantic reasoning, not type coercion.
  • Include a fetched_at timestamp on every response so the model can reason about data freshness explicitly.
  • Include the source URL so the model can cite it or re-fetch if needed.
  • Return an error field rather than raising exceptions. The model handles errors gracefully in its reasoning if they are structured.
  • Keep payloads under 4,000 tokens. If a page has more content than that, extract only the relevant section with a CSS selector or a targeted XPath query.

Production Patterns: Scheduling, Caching, and Error Handling

A development MCP server and a production MCP server differ primarily in reliability engineering, not in the core protocol logic.

Caching with TTL

Most web data does not change every minute. Fetching the same product page 40 times during a single Claude session is wasteful and risks triggering rate limits. Add an in-process cache with a short TTL (5-15 minutes for prices, longer for structural data) keyed on URL plus selector:

import time
_cache = {}

def cached_fetch(url, selector, ttl=600):
    key = f"{url}::{selector}"
    hit = _cache.get(key)
    if hit and time.time() - hit["ts"] < ttl:
        return {**hit["data"], "cache_hit": True}
    data = fetch_page(url, selector)
    _cache[key] = {"ts": time.time(), "data": data}
    return data

Retry and backoff

Network failures happen. Wrap every outbound request in an exponential backoff loop with a maximum of 3 retries. The tenacity library handles this cleanly for Python:

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(min=1, max=8))
def resilient_fetch(url):
    import httpx
    return httpx.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=10)

Scheduled pre-fetching

For agents that run on a schedule (daily briefings, change alerts, monitoring dashboards), pre-fetch and cache the pages before the agent runs rather than fetching at query time. This removes network latency from the critical path and ensures the agent always has data, even if the target site is temporarily unavailable.

A managed scraping platform that handles scheduling and delivery into your MCP server removes the operational burden of running your own fetch infrastructure. You define the URLs and schedule; the platform handles rotating proxies, browser rendering, and retry logic. Trawl is built for exactly this pattern: scheduled scraping jobs that deliver structured results your MCP server can serve to Claude on demand.

Ready to stop managing fetch infrastructure manually? Try Trawl free and wire live web data to your Claude agents in minutes.

Observability

Log every tool call with its input, output size, latency, and status code. When an agent produces a wrong answer, the first question is always: did the tool return bad data, or did the model reason badly about good data? Structured logs per tool invocation answer that question instantly.

Tools and Resources

Several tools make it easier to build MCP-connected web data pipelines for Claude:

  • MCP Python SDK: the official SDK for building MCP servers in Python. Handles protocol negotiation, transport, and tool registration. Start here.
  • Trawl: managed scraping infrastructure with a scheduling layer. Run scraping jobs on a cron and serve their output via a simple MCP server wrapper. Handles proxy rotation, browser rendering, and retry logic so your MCP server does not have to.
  • MCP specification (modelcontextprotocol.io): the canonical reference for protocol semantics, capability negotiation, and transport options.
  • Playwright / Puppeteer: headless browser libraries for JavaScript-rendered pages. Use as the engine inside your MCP server for dynamic sites that do not expose clean HTML to plain HTTP requests.
  • FastAPI or FastMCP: HTTP server frameworks for deploying MCP servers as remote HTTP endpoints. FastMCP builds the MCP transport layer on top; FastAPI is useful when you need more control over the HTTP interface.

Key Takeaways

  1. MCP is a JSON-RPC 2.0 standard that gives Claude (and any MCP-compatible host) a composable, maintainable interface for calling external tools, including scrapers.
  2. Grounding Claude with live web data eliminates both hallucination (fabricated facts) and staleness (outdated facts), the two failure modes that undermine AI agent reliability in production.
  3. The right server design returns structured JSON, not raw HTML: typed fields, timestamps, source URLs, and explicit error objects.
  4. Production MCP servers need caching with TTL, retry with backoff, and per-call observability. Build these in from the start, not as afterthoughts.
  5. Remote HTTP transport (instead of stdio) is the production pattern: it decouples the scraping infrastructure from the Claude host and enables team-wide or multi-agent access.
  6. Pre-fetching on a schedule and serving from cache removes network latency from the critical agent path and makes the system resilient to target site availability.
  7. MCP is now governed by the Linux Foundation and supported across major AI providers, making it a safe long-term bet for integration architecture.

If you want to add live web data to your Claude agents without managing fetch infrastructure yourself, Trawl handles the scraping layer so your MCP server stays focused on protocol, not plumbing.

FAQ

What is the Model Context Protocol (MCP)?

MCP is an open JSON-RPC 2.0 standard introduced by Anthropic in November 2024 that defines how large language models connect to external tools, databases, and services. It gives AI hosts like Claude a universal interface for calling scraping tools, querying databases, and accessing any data source through a consistent protocol without custom integration code per data source.

How does MCP differ from RAG?

RAG (Retrieval-Augmented Generation) retrieves from a pre-indexed static corpus at query time. MCP calls live tools that can take actions, fetch current data, and interact with external systems in real time. For current web data that changes continuously, MCP (or MCP plus a pre-fetching layer) is the right pattern. RAG is better suited for large document corpora where freshness is not the primary concern.

Do I need to know the MCP protocol internals to build a server?

No. The official MCP Python and TypeScript SDKs handle all protocol negotiation, transport, and serialization. You write your tool as a plain function with a docstring, decorate it with @mcp.tool(), and the SDK handles the rest. Protocol knowledge is useful for debugging edge cases but is not required to ship a working server.

Can Claude call MCP tools autonomously, or does a human need to trigger each call?

Claude calls MCP tools autonomously based on the task context. When you give Claude a task that requires current data, it identifies the relevant tool from its catalog, constructs the parameters, calls the tool, and incorporates the result into its reasoning, all without human intervention per call. You control which tools are available; Claude decides when and how to use them.

What is the difference between an MCP tool and an MCP resource?

Tools are callable actions with parameters, initiated by the model at reasoning time. Resources are read-only data that the host injects into context before the model starts reasoning. For live web data fetched on demand, use tools. For a pre-loaded dataset or cached corpus you want in context for every session, use resources.

How do I handle JavaScript-rendered pages in an MCP scraping server?

Replace the HTTP client in your tool implementation with a headless browser: Playwright or Puppeteer. The MCP server interface stays identical. The only change is that the tool calls browser.new_page() and page.goto(url) instead of httpx.get(url). The model calls the same tool name with the same parameters and receives the same structured output.

How do I secure a remote MCP server?

For remote HTTP transport, use bearer token authentication: add an Authorization header to every client request and validate it on the server. Do not expose an MCP server that can hit arbitrary URLs to the public internet without authentication: a malicious prompt could instruct Claude to fetch attacker-controlled URLs. Allowlist the domains your tools are permitted to fetch from.

What happens when a web scraping tool fails inside an MCP server?

The server should return a structured error object rather than raising an unhandled exception. Return {"url": url, "error": "fetch_failed", "detail": str(e)} and let Claude handle it gracefully in its reasoning. The model will typically acknowledge the failure and either retry, explain the limitation to the user, or proceed with partial data. Unhandled exceptions that crash the server break the entire session.

Is there a registry of existing MCP servers I can reuse?

Yes. The community maintains a public registry at mcpservers.org, and the official Anthropic repository lists reference servers. As of early 2026, independent census counts identified over 17,000 public MCP servers covering databases, file systems, and developer tools. Before building a custom server, check whether a maintained community server already covers your data source.

Can I use MCP with AI models other than Claude?

Yes. MCP is now supported by OpenAI (via the OpenAI Agents SDK), Google DeepMind, and a growing number of third-party agent frameworks including LangChain and LlamaIndex. A server you build for Claude will work with any MCP-compatible client. The protocol is vendor-neutral, governed by the Linux Foundation's Agentic AI Foundation since December 2025.

Disclaimer: Trawl provides scraping infrastructure. Users are responsible for ensuring their use complies with applicable laws and website terms of service. This article is for educational purposes only.

Written by Pierre | August 2026