Advertise
MCP.so
Sign In

MCP Server Webcrawl

@pragmar

RepositoryHomepage

Basic information

Category

Other

License

NOASSERTION

Transports

stdio

Author

pragmar

Submitted by

pragmar

About MCP Server Webcrawl

MCP server tailored to connecting web crawler data and archives

#web-crawler#web-scraping

MCP Server Webcrawl config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "webcrawl": {
      "command": "mcp-server-webcrawl",
      "args": [
        "--crawler",
        "wget",
        "--datasrc",
        "/path/to/wget/archives/"
      ]
    }
  }
}

MCP Server Webcrawl tools

Tool list not available yet

This listing does not currently have a tool list to display. This does not mean the server has no tools; check its documentation for supported capabilities.

We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.

Overview

What is mcp-server-webcrawl?

mcp-server-webcrawl provides advanced search and retrieval for web crawler data, enabling an AI client to filter and analyze web content. It includes a fulltext search interface with boolean support and resource filtering by type, HTTP status, and more. It works with various web crawlers such as ArchiveBox, HTTrack, InterroBot, Katana, SiteOne, WARC, and wget.

How to use mcp-server-webcrawl?

Install via pip: pip install mcp-server-webcrawl. Requires Python (>=3.10) and Claude Desktop. For step-by-step MCP server setup, refer to the Setup Guides. The server can also be used in interactive terminal mode for classic boolean search without AI.

Key features of mcp-server-webcrawl

  • Multi-crawler compatible (ArchiveBox, HTTrack, InterroBot, Katana, SiteOne, WARC, wget)
  • Boolean search support with field-specific queries
  • Filter by type, status, and more
  • Extras: markdown, snippets, regex, xpath, thumbnails
  • Prompt routines provided (SEO audit, 404 audit, Gopher interface, etc.)
  • Interactive mode for terminal-based searching of web archives

Use cases of mcp-server-webcrawl

  • Analyze crawled website content with fulltext and boolean queries
  • Perform technical SEO audits using provided prompt routines
  • Detect broken links and suggest fixes with 404 audit
  • Build a roll-your-own website knowledgebase from crawled data
  • Search web archives remotely via SSH interactive mode

Frequently asked questions

1

What crawlers are supported?

mcp-server-webcrawl works with ArchiveBox, HTTrack, InterroBot, Katana, SiteOne, WARC, and wget. Each has a dedicated setup guide.

2

What are the runtime requirements?

The server requires Python >=3.10 and Claude Desktop. It is installed via `pip install mcp-server-webcrawl`.

3

Can I use it without an AI?

Yes, mcp-server-webcrawl includes an interactive mode that runs classic Boolean search on your web archives in a terminal, no AI required.

4

What extras help reduce token usage?

Markdown, snippets, regex, xpath, and thumbnails can transform HTTP data to be more token-efficient. For example, markdown produces about one-third the bytes of source HTML.

5

How does boolean search work?

The query engine supports field-specific searches (e.g., `type: html`, `status: 200`) and boolean operators (AND, OR, NOT). Fulltext can combine url, content, and headers fields. Wildcards and exact phrases are also supported.

Comments

More Other MCP servers

N

Next Generation Movers: free moving quotes (remote MCP)

Next Generation Movers

Free moving quotes for AI agents from Next Generation Movers, a licensed interstate mover (USDOT 3764697). US long-distance nationwide and local DC/MD/VA. Indicative ballpark price and booking options; optional $0.01 x40

DeepMark

DeepMark helps teachers deliver rapid, consistent marking with meaningful feedback for every student — in a fraction of the time. What once took a week, now takes one free period.

ICSS

chokcoco

不止于 CSS

20.7K

🚀 Model Context Protocol (MCP) Curriculum for Beginners

microsoft

This open-source curriculum introduces the fundamentals of Model Context Protocol (MCP) through real-world, cross-language examples in .NET, Java, TypeScript, JavaScript, Rust and Python. Designed for developers, it focuses on practical techniques for building modular, scalable,

16.7K

Blender

ahujasid

Open-source MCP to use Blender with any LLM

23.6K
Advertise