What crawlers are supported by mcp-server-webcrawl?
mcp-server-webcrawl works with WARC, wget, InterroBot, Katana, and SiteOne.
MCP Server Webcrawl
@MCP-Mirror
Basic information
About MCP Server Webcrawl
Mirror of
MCP Server Webcrawl config
No standard config provided
This server doesn't expose a parseable MCP config block in its README. See the repository for install instructions.
RepositoryMCP Server Webcrawl tools
Tool list not available yet
This listing does not currently have a tool list to display. This does not mean the server has no tools; check its documentation for supported capabilities.
We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.
Overview
What is mcp-server-webcrawl?
mcp-server-webcrawl bridges the gap between your web crawl and AI language models using the Model Context Protocol (MCP). It allows an AI client to filter and analyze web content under your direction or autonomously, and works with a variety of web crawlers (WARC, wget, InterroBot, Katana, SiteOne).
How to use mcp-server-webcrawl?
Install via pip install mcp-server-webcrawl (requires Python ≥3.10 and Claude Desktop). Configure the MCP server in Claude Desktop’s settings file by adding an entry under mcpServers with the command mcp-server-webcrawl and arguments specifying the crawler type and data source path. On macOS, use the absolute path to the executable.
Key features of mcp-server-webcrawl
- Claude Desktop ready
- Full-text search with boolean support
- Filter resources by type, HTTP status, and more
- Compatible with multiple web crawlers
- Quick MCP configuration
- ChatGPT support coming soon
Use cases of mcp-server-webcrawl
- Search and analyze crawled web archives with natural language queries
- Filter web content by file type or HTTP status for targeted analysis
- Enable AI assistants to browse and summarize local crawl data
- Integrate with various crawler outputs (WARC, wget, etc.) in one tool
Frequently asked questions
How do I install mcp-server-webcrawl?
Install via `pip install mcp-server-webcrawl`. It requires Python 3.10 or higher and Claude Desktop.
How do I configure mcp-server-webcrawl for macOS?
On macOS, you must use the absolute path to the `mcp-server-webcrawl` executable in the `command` field (e.g., `/Users/yourusername/.local/bin/mcp-server-webcrawl`). Run `which mcp-server-webcrawl` to find the correct path.
What is the data source argument for mcp-server-webcrawl?
The `--datasrc` argument specifies the directory or database path for the crawled content. Its value depends on the crawler: for wget it’s the parent directory of mirrors; for WARC it’s the parent of WARC files; for InterroBot it’s the direct database path; for Katana and SiteOne it’s the parent directory of archives.
What transport does mcp-server-webcrawl use?
mcp-server-webcrawl uses stdio transport as configured in Claude Desktop’s MCP settings via the `command` and `args` fields.
More Other MCP servers
Blender
ahujasidOpen-source MCP to use Blender with any LLM
23.6K
Reactive Resume
amruthpillaiA one-of-a-kind resume builder that keeps your privacy in mind. Completely secure, customizable, portable, open-source and free forever. Try it out today!
39.5K
Activepieces
activepiecesAI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
23.2K
Yuppsy
Yuppsy MCP: build and publish real estate websites from your AI assistant. Yuppsy lets your AI assistant create live websites for real estate agents and individual property listings. You don't need a designer or any code
P
PickFu MCP
PickFu MCP is a Model Context Protocol (MCP) server that connects AI assistants, developers, marketers, product managers, and AI agents to PickFu's consumer research platform. It enables users to create surveys, conduct
Comments