What is webfetch-mcp?
webfetch-mcp is a local Python MCP server that replaces an AI assistant’s built-in WebFetch tool with a fully configurable HTTP client. It supports domain-scoped headers, retries, proxies, timeouts, output formats, bot-block detection, and prompt-injection sanitization.
How to use webfetch-mcp?
Clone the repository, create a Python 3.10+ virtual environment, install dependencies from requirements.txt, copy and edit the YAML configuration file (webfetch.yaml.example → webfetch.yaml), then register the server’s command, args, and env in your AI assistant’s MCP server settings (e.g., ~/.claude/settings.json). Restart the assistant; the tool is exposed as mcp__webfetch__fetch.
Key features of webfetch-mcp
- Domain-scoped custom headers and auth tokens.
- Configurable YAML config file.
- Retry with exponential backoff on failures.
- Bot-block detection with optional retry.
- Prompt-injection sanitization (flag or strip).
- JavaScript rendering via Playwright (optional).
Use cases of webfetch-mcp
- Fetching authenticated API content with per-domain tokens.
- Scraping bot-protected websites (Cloudflare, Akamai).
- Extracting main article content with
trafilatura format.
- Running CI/CD smoke tests using response assertions.
- Rendering JavaScript-heavy SPAs before content extraction.
コメント