Webfetch Mcp

What is webfetch-mcp?

webfetch-mcp is a local Python MCP server that replaces an AI assistant’s built-in WebFetch tool with a fully configurable HTTP client. It supports domain-scoped headers, retries, proxies, timeouts, output formats, bot-block detection, and prompt-injection sanitization.

How to use webfetch-mcp?

Clone the repository, create a Python 3.10+ virtual environment, install dependencies from requirements.txt, copy and edit the YAML configuration file (webfetch.yaml.example → webfetch.yaml), then register the server’s command, args, and env in your AI assistant’s MCP server settings (e.g., ~/.claude/settings.json). Restart the assistant; the tool is exposed as mcp__webfetch__fetch.

Key features of webfetch-mcp

• Domain-scoped custom headers and auth tokens.
• Configurable YAML config file.
• Retry with exponential backoff on failures.
• Bot-block detection with optional retry.
• Prompt-injection sanitization (flag or strip).
• JavaScript rendering via Playwright (optional).

Use cases of webfetch-mcp

• Fetching authenticated API content with per-domain tokens.
• Scraping bot-protected websites (Cloudflare, Akamai).
• Extracting main article content with trafilatura format.
• Running CI/CD smoke tests using response assertions.
• Rendering JavaScript-heavy SPAs before content extraction.

FAQ from webfetch-mcp

How is webfetch-mcp different from a built-in WebFetch tool?

The built-in tool sends requests without custom headers, so it is blocked by bot-protection systems and cannot authenticate against APIs. webfetch-mcp allows per-domain headers, retries, proxies, and output formatting, all configured automatically.

What are the runtime dependencies of webfetch-mcp?

Python 3.10+ and the packages in requirements.txt: mcp[cli], httpx, python-dotenv, markdownify, trafilatura, pyyaml, and beautifulsoup4. JS rendering requires Playwright (playwright install chromium).

Where does the fetched data live?

The server fetches data directly and returns it to the AI assistant in the response. No data is stored or persisted by the server itself. Response length can be capped with max_bytes.

What transports or authentication does webfetch-mcp support?

It communicates with MCP-compatible assistants over stdio (the command-line invocation). Authentication is handled by domain-scoped headers set in the YAML config file.

Are there any known limits of webfetch-mcp?

The README does not document explicit functional limits. It supports response truncation via max_bytes to avoid filling the assistant’s context window.