What crawlers are supported by mcp-server-webcrawl?
mcp-server-webcrawl works with WARC, wget, InterroBot, Katana, and SiteOne.
MCP Server Webcrawl
@MCP-Mirror
Basic information
About MCP Server Webcrawl
Mirror of
MCP Server Webcrawl config
No standard config provided
This server doesn't expose a parseable MCP config block in its README. See the repository for install instructions.
RepositoryMCP Server Webcrawl tools
Tool list not available yet
This listing does not currently have a tool list to display. This does not mean the server has no tools; check its documentation for supported capabilities.
We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.
Overview
What is mcp-server-webcrawl?
mcp-server-webcrawl bridges the gap between your web crawl and AI language models using the Model Context Protocol (MCP). It allows an AI client to filter and analyze web content under your direction or autonomously, and works with a variety of web crawlers (WARC, wget, InterroBot, Katana, SiteOne).
How to use mcp-server-webcrawl?
Install via pip install mcp-server-webcrawl (requires Python ≥3.10 and Claude Desktop). Configure the MCP server in Claude Desktop’s settings file by adding an entry under mcpServers with the command mcp-server-webcrawl and arguments specifying the crawler type and data source path. On macOS, use the absolute path to the executable.
Key features of mcp-server-webcrawl
- Claude Desktop ready
- Full-text search with boolean support
- Filter resources by type, HTTP status, and more
- Compatible with multiple web crawlers
- Quick MCP configuration
- ChatGPT support coming soon
Use cases of mcp-server-webcrawl
- Search and analyze crawled web archives with natural language queries
- Filter web content by file type or HTTP status for targeted analysis
- Enable AI assistants to browse and summarize local crawl data
- Integrate with various crawler outputs (WARC, wget, etc.) in one tool
Frequently asked questions
How do I install mcp-server-webcrawl?
Install via `pip install mcp-server-webcrawl`. It requires Python 3.10 or higher and Claude Desktop.
How do I configure mcp-server-webcrawl for macOS?
On macOS, you must use the absolute path to the `mcp-server-webcrawl` executable in the `command` field (e.g., `/Users/yourusername/.local/bin/mcp-server-webcrawl`). Run `which mcp-server-webcrawl` to find the correct path.
What is the data source argument for mcp-server-webcrawl?
The `--datasrc` argument specifies the directory or database path for the crawled content. Its value depends on the crawler: for wget it’s the parent directory of mirrors; for WARC it’s the parent of WARC files; for InterroBot it’s the direct database path; for Katana and SiteOne it’s the parent directory of archives.
What transport does mcp-server-webcrawl use?
mcp-server-webcrawl uses stdio transport as configured in Claude Desktop’s MCP settings via the `command` and `args` fields.
More Other MCP servers
EverArt
modelcontextprotocolModel Context Protocol Servers
88.1K
Hindsight
vectorize-ioHindsight: Agent Memory That Learns
50.4K
Figs, Relationship Expert
Figs O'Sullivan, LMFTCouples therapist Figs O'Sullivan (LMFT, 17 years, endorsed by Sue Johnson, creator of EFT) inside your AI. Ask relationship questions answered from his clinical writing, name the pattern in a fight, get a three-step rep
Klarix Intelligence Engine
KlarixB2B competitive intelligence, live sales battlecards, 5-axis ICP prospect fit scoring, and tech stack teardowns over MCP and REST with cited public-web evidence.
N
Next Generation Movers: free moving quotes (remote MCP)
Next Generation MoversFree moving quotes for AI agents from Next Generation Movers, a licensed interstate mover (USDOT 3764697). US long-distance nationwide and local DC/MD/VA. Indicative ballpark price and booking options; optional $0.01 x40
Comments