广告合作
MCP.so
登录

MCP Server Webcrawl

@pragmar

代码仓库主页

基本信息

分类

其他

许可证

NOASSERTION

传输方式

stdio

作者

pragmar

提交者

pragmar

关于 MCP Server Webcrawl

MCP server tailored to connecting web crawler data and archives

#web-crawler#web-scraping

MCP Server Webcrawl 配置

使用下面的配置,将此服务器添加到你的 MCP 客户端。

{
  "mcpServers": {
    "webcrawl": {
      "command": "mcp-server-webcrawl",
      "args": [
        "--crawler",
        "wget",
        "--datasrc",
        "/path/to/wget/archives/"
      ]
    }
  }
}

MCP Server Webcrawl 工具

工具列表暂未提供

当前收录暂时没有可展示的工具列表，不代表服务没有工具。请查阅项目文档了解支持的能力。

工具是从 README 中自动提取的。维护者可以在 ## Tools 标题下列出工具,即可填充这部分内容。

概览

What is mcp-server-webcrawl?

mcp-server-webcrawl provides advanced search and retrieval for web crawler data, enabling an AI client to filter and analyze web content. It includes a fulltext search interface with boolean support and resource filtering by type, HTTP status, and more. It works with various web crawlers such as ArchiveBox, HTTrack, InterroBot, Katana, SiteOne, WARC, and wget.

How to use mcp-server-webcrawl?

Install via pip: pip install mcp-server-webcrawl. Requires Python (>=3.10) and Claude Desktop. For step-by-step MCP server setup, refer to the Setup Guides. The server can also be used in interactive terminal mode for classic boolean search without AI.

Key features of mcp-server-webcrawl

  • Multi-crawler compatible (ArchiveBox, HTTrack, InterroBot, Katana, SiteOne, WARC, wget)
  • Boolean search support with field-specific queries
  • Filter by type, status, and more
  • Extras: markdown, snippets, regex, xpath, thumbnails
  • Prompt routines provided (SEO audit, 404 audit, Gopher interface, etc.)
  • Interactive mode for terminal-based searching of web archives

Use cases of mcp-server-webcrawl

  • Analyze crawled website content with fulltext and boolean queries
  • Perform technical SEO audits using provided prompt routines
  • Detect broken links and suggest fixes with 404 audit
  • Build a roll-your-own website knowledgebase from crawled data
  • Search web archives remotely via SSH interactive mode

常见问题

1

What crawlers are supported?

mcp-server-webcrawl works with ArchiveBox, HTTrack, InterroBot, Katana, SiteOne, WARC, and wget. Each has a dedicated setup guide.

2

What are the runtime requirements?

The server requires Python >=3.10 and Claude Desktop. It is installed via `pip install mcp-server-webcrawl`.

3

Can I use it without an AI?

Yes, mcp-server-webcrawl includes an interactive mode that runs classic Boolean search on your web archives in a terminal, no AI required.

4

What extras help reduce token usage?

Markdown, snippets, regex, xpath, and thumbnails can transform HTTP data to be more token-efficient. For example, markdown produces about one-third the bytes of source HTML.

5

How does boolean search work?

The query engine supports field-specific searches (e.g., `type: html`, `status: 200`) and boolean operators (AND, OR, NOT). Fulltext can combine url, content, and headers fields. Wildcards and exact phrases are also supported.

评论

其他 分类下的更多 MCP 服务器

Glasswarp

Glasswarp

See and control a real Windows PC you own — from any MCP client, locally or remotely. Observe (UIA + screenshots), click/type/drag/scroll, launch apps, owner Live View. BYOH: your machine, your key.

Hindsight

vectorize-io

Hindsight: Agent Memory That Learns

50.4K

Handl

Handl speaks the Model Context Protocol. Your whole billing operation — projects, invoices, clients, reports, approvals — as tools any AI assistant can call, inside your autonomy rules.

Codelf

unbug

A search tool helps dev to solve the naming things problem.

14.2K

The Living Bread

The Living Bread

A free remote MCP server for The Living Bread. Bible passages read word for word from a stored public domain text with an evidence label, keyword search and cross references, real churches, gatherings and prayer groups w

广告合作