広告掲載
MCP.so
ログイン

MCP Server Webcrawl

@pragmar

リポジトリホームページ

基本情報

カテゴリ

その他

ライセンス

NOASSERTION

トランスポート

stdio

作者

pragmar

投稿者

pragmar

MCP Server Webcrawl について

MCP server tailored to connecting web crawler data and archives

#web-crawler#web-scraping

MCP Server Webcrawl の設定

以下の設定を使って、このサーバーを MCP 対応クライアントに追加してください。

{
  "mcpServers": {
    "webcrawl": {
      "command": "mcp-server-webcrawl",
      "args": [
        "--crawler",
        "wget",
        "--datasrc",
        "/path/to/wget/archives/"
      ]
    }
  }
}

MCP Server Webcrawl のツール

ツール一覧はまだありません

この掲載には表示できるツール一覧がまだありません。サーバーにツールがないという意味ではありません。対応機能はプロジェクトのドキュメントをご確認ください。

ツールは README から自動的に抽出されます。メンテナーは ## Tools という見出しの下に記載することで、このタブに反映できます。

概要

What is mcp-server-webcrawl?

mcp-server-webcrawl provides advanced search and retrieval for web crawler data, enabling an AI client to filter and analyze web content. It includes a fulltext search interface with boolean support and resource filtering by type, HTTP status, and more. It works with various web crawlers such as ArchiveBox, HTTrack, InterroBot, Katana, SiteOne, WARC, and wget.

How to use mcp-server-webcrawl?

Install via pip: pip install mcp-server-webcrawl. Requires Python (>=3.10) and Claude Desktop. For step-by-step MCP server setup, refer to the Setup Guides. The server can also be used in interactive terminal mode for classic boolean search without AI.

Key features of mcp-server-webcrawl

  • Multi-crawler compatible (ArchiveBox, HTTrack, InterroBot, Katana, SiteOne, WARC, wget)
  • Boolean search support with field-specific queries
  • Filter by type, status, and more
  • Extras: markdown, snippets, regex, xpath, thumbnails
  • Prompt routines provided (SEO audit, 404 audit, Gopher interface, etc.)
  • Interactive mode for terminal-based searching of web archives

Use cases of mcp-server-webcrawl

  • Analyze crawled website content with fulltext and boolean queries
  • Perform technical SEO audits using provided prompt routines
  • Detect broken links and suggest fixes with 404 audit
  • Build a roll-your-own website knowledgebase from crawled data
  • Search web archives remotely via SSH interactive mode

よくある質問

1

What crawlers are supported?

mcp-server-webcrawl works with ArchiveBox, HTTrack, InterroBot, Katana, SiteOne, WARC, and wget. Each has a dedicated setup guide.

2

What are the runtime requirements?

The server requires Python >=3.10 and Claude Desktop. It is installed via `pip install mcp-server-webcrawl`.

3

Can I use it without an AI?

Yes, mcp-server-webcrawl includes an interactive mode that runs classic Boolean search on your web archives in a terminal, no AI required.

4

What extras help reduce token usage?

Markdown, snippets, regex, xpath, and thumbnails can transform HTTP data to be more token-efficient. For example, markdown produces about one-third the bytes of source HTML.

5

How does boolean search work?

The query engine supports field-specific searches (e.g., `type: html`, `status: 200`) and boolean operators (AND, OR, NOT). Fulltext can combine url, content, and headers fields. Wildcards and exact phrases are also supported.

コメント

「その他」の他のコンテンツ

Blender

ahujasid

Open-source MCP to use Blender with any LLM

23.6K

DeepMark

DeepMark helps teachers deliver rapid, consistent marking with meaningful feedback for every student — in a fraction of the time. What once took a week, now takes one free period.

The Living Bread

The Living Bread

A free remote MCP server for The Living Bread. Bible passages read word for word from a stored public domain text with an evidence label, keyword search and cross references, real churches, gatherings and prayer groups w

Senso MCP for Joomla

sensomedia

Senso MCP turns a Joomla 5 or 6 site into a remote MCP server. It runs inside Joomla itself (no extra server, no Node.js bridge) and is secured with OAuth 2.1 and Joomla's own ACL. The free pack lets an AI assistant crea

Figs, Relationship Expert

Figs O'Sullivan, LMFT

Couples therapist Figs O'Sullivan (LMFT, 17 years, endorsed by Sue Johnson, creator of EFT) inside your AI. Ask relationship questions answered from his clinical writing, name the pattern in a fight, get a three-step rep

広告掲載