広告掲載
MCP.so
ログイン

mcp-server-spider: A spider MCP server

@GeorgeLS

リポジトリ

基本情報

カテゴリ

その他

ライセンス

MIT

トランスポート

stdio

作者

GeorgeLS

mcp-server-spider: A spider MCP server について

概要はまだありません

mcp-server-spider: A spider MCP server の設定

以下の設定を使って、このサーバーを MCP 対応クライアントに追加してください。

{
  "mcpServers": {
    "mcp-server-spider": {
      "command": "uvx",
      "args": [
        "mcp-server-spider"
      ]
    }
  }
}

mcp-server-spider: A spider MCP server のツール

4

`url`: The url to crawl

List of URLs found

Same as `crawl`

A list of JSON objects (as a string) that contain the url, links and content of each page discovered

概要

What is mcp-server-spider?

mcp-server-spider is a Model Context Protocol (MCP) server that provides tools for Spider crawler interaction and automation. It allows crawling and scraping web pages via MCP-compatible clients. The server is currently in early development.

How to use mcp-server-spider?

Install and run using uvx (recommended) or install via pip (pip install mcp-server-spider) and run with python -m mcp_server_spider. The server exposes two tools: crawl and scrape, each accepting parameters such as url, headers, user_agent, depth, blacklist, whitelist, respect_robots_txt, and accept_invalid_certs.

Key features of mcp-server-spider

  • Crawl a URL and return discovered links
  • Scrape a URL and return structured JSON with content
  • Configurable crawl depth and user agent
  • Support for additional HTTP headers
  • Whitelist/blacklist URLs via regular expressions
  • Option to respect or bypass robots.txt
  • Option to accept invalid TLS certificates

Use cases of mcp-server-spider

  • Automate web crawling for link discovery
  • Extract structured content from web pages
  • Integrate web scraping into MCP-enabled workflows
  • Build custom crawlers with depth constraints and filtering

よくある質問

1

What tools does mcp-server-spider provide?

Two tools: `crawl` (returns list of URLs found) and `scrape` (returns a JSON string with URL, links, and content of each page).

2

What input parameters are available for crawl and scrape?

Both tools accept: `url` (required), `headers`, `user_agent`, `depth`, `blacklist`, `whitelist`, `respect_robots_txt`, and `accept_invalid_certs`.

3

How do I install and run mcp-server-spider?

Recommended: use `uvx` with no separate installation. Alternatively, install via `pip install mcp-server-spider` and run with `python -m mcp_server_spider`.

4

Is mcp-server-spider stable?

The README notes it is in early development; there may be bugs and features added in the future.

コメント

「その他」の他のコンテンツ

Health by Dr Brad

Dr Brad Stanfield

Health by Dr Brad works out a health plan from your own results: which screening is due at your age, and evidence-based suggestions, each with the reason and the published guideline or study behind it. Your record is one

MaxKB

1Panel-dev

🔥 MaxKB is an open-source platform for building enterprise-grade agents. 强大易用的开源企业级智能体平台。

22.0K

Sequential Thinking

modelcontextprotocol

Model Context Protocol Servers

88.2K

Activepieces

activepieces

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

23.2K

🚀 Model Context Protocol (MCP) Curriculum for Beginners

microsoft

This open-source curriculum introduces the fundamentals of Model Context Protocol (MCP) through real-world, cross-language examples in .NET, Java, TypeScript, JavaScript, Rust and Python. Designed for developers, it focuses on practical techniques for building modular, scalable,

16.7K
広告掲載