mcp-server-webcrawl

What is mCP Server WebCrawl?

mcp-server-webcrawl provides advanced search and retrieval for web crawler data. It allows AI clients to filter and analyze web content via a fulltext search with boolean support and resource filtering by type, HTTP status, and more. It is designed for users of Claude Desktop who work with data from crawlers like ArchiveBox, HTTrack, or wget.

How to use mCP Server WebCrawl?

mcp-server-webcrawl requires Python (>=3.10) and is installed via pip install mcp-server-webcrawl. After installation, it is configured as an MCP server for Claude Desktop. It can also run in interactive mode as a terminal search tool for web archives without an AI client.

Key features of mCP Server WebCrawl

• Claude Desktop ready
• Multi-crawler compatible (ArchiveBox, wget, and more)
• Filter resources by type, HTTP status, and more
• Boolean search with field-specific and fulltext queries
• Support for Markdown, snippets, regex, and XPath extras

Use cases of mCP Server WebCrawl

• SEO audits of crawled website content
• Broken link detection and pattern analysis
• Website performance and optimization analysis
• File organization and asset composition review
• Searchable web archive in a terminal via interactive mode

FAQ from mCP Server WebCrawl

What web crawlers does mCP Server WebCrawl work with?

It works with ArchiveBox, HTTrack, InterroBot, Katana, SiteOne, WARC format, and wget.

What are the system requirements for mCP Server WebCrawl?

It requires Python version 3.10 or higher and the Claude Desktop application.

Is mCP Server WebCrawl free and open source?

Yes, it is free and open source.

What search syntax does mCP Server WebCrawl support?

It supports boolean operators (AND, OR, NOT), exact phrase matching, wildcards, field-specific searches (e.g., type: html, status: 200), and complex grouped expressions.