Overview
What is mcp-server-webcrawl?
mcp-server-webcrawl bridges the gap between your web crawl and AI language models using the Model Context Protocol (MCP). It allows an AI client to filter and analyze web content under your direction or autonomously, and works with a variety of web crawlers (WARC, wget, InterroBot, Katana, SiteOne).
How to use mcp-server-webcrawl?
Install via pip install mcp-server-webcrawl (requires Python ≥3.10 and Claude Desktop). Configure the MCP server in Claude Desktop’s settings file by adding an entry under mcpServers with the command mcp-server-webcrawl and arguments specifying the crawler type and data source path. On macOS, use the absolute path to the executable.
Key features of mcp-server-webcrawl
- Claude Desktop ready
- Full-text search with boolean support
- Filter resources by type, HTTP status, and more
- Compatible with multiple web crawlers
- Quick MCP configuration
- ChatGPT support coming soon
Use cases of mcp-server-webcrawl
- Search and analyze crawled web archives with natural language queries
- Filter web content by file type or HTTP status for targeted analysis
- Enable AI assistants to browse and summarize local crawl data
- Integrate with various crawler outputs (WARC, wget, etc.) in one tool
FAQ from mcp-server-webcrawl
What crawlers are supported by mcp-server-webcrawl?
mcp-server-webcrawl works with WARC, wget, InterroBot, Katana, and SiteOne.
How do I install mcp-server-webcrawl?
Install via pip install mcp-server-webcrawl. It requires Python 3.10 or higher and Claude Desktop.
How do I configure mcp-server-webcrawl for macOS?
On macOS, you must use the absolute path to the mcp-server-webcrawl executable in the command field (e.g., /Users/yourusername/.local/bin/mcp-server-webcrawl). Run which mcp-server-webcrawl to find the correct path.
What is the data source argument for mcp-server-webcrawl?
The --datasrc argument specifies the directory or database path for the crawled content. Its value depends on the crawler: for wget it’s the parent directory of mirrors; for WARC it’s the parent of WARC files; for InterroBot it’s the direct database path; for Katana and SiteOne it’s the parent directory of archives.
What transport does mcp-server-webcrawl use?
mcp-server-webcrawl uses stdio transport as configured in Claude Desktop’s MCP settings via the command and args fields.
