mcp-server-spider: A spider MCP server

Name: mcp-server-spider: A spider MCP server
Author: GeorgeLS

@GeorgeLS

リポジトリ

mcp-server-spider: A spider MCP server について

概要はまだありません

基本情報

カテゴリ

その他

ライセンス

MIT

ランタイム

python

トランスポート

stdio

公開者

GeorgeLS

設定

以下の設定を使って、このサーバーを MCP 対応クライアントに追加してください。

{
  "mcpServers": {
    "mcp-server-spider": {
      "command": "uvx",
      "args": [
        "mcp-server-spider"
      ]
    }
  }
}

ツール

`url`: The url to crawl

List of URLs found

Same as `crawl`

A list of JSON objects (as a string) that contain the url, links and content of each page discovered

概要

What is mcp-server-spider?

mcp-server-spider is a Model Context Protocol (MCP) server that provides tools for Spider crawler interaction and automation. It allows crawling and scraping web pages via MCP-compatible clients. The server is currently in early development.

How to use mcp-server-spider?

Install and run using uvx (recommended) or install via pip (pip install mcp-server-spider) and run with python -m mcp_server_spider. The server exposes two tools: crawl and scrape, each accepting parameters such as url, headers, user_agent, depth, blacklist, whitelist, respect_robots_txt, and accept_invalid_certs.

Key features of mcp-server-spider

Crawl a URL and return discovered links
Scrape a URL and return structured JSON with content
Configurable crawl depth and user agent
Support for additional HTTP headers
Whitelist/blacklist URLs via regular expressions
Option to respect or bypass robots.txt
Option to accept invalid TLS certificates

Use cases of mcp-server-spider

Automate web crawling for link discovery
Extract structured content from web pages
Integrate web scraping into MCP-enabled workflows
Build custom crawlers with depth constraints and filtering

FAQ from mcp-server-spider

What tools does mcp-server-spider provide?

Two tools: crawl (returns list of URLs found) and scrape (returns a JSON string with URL, links, and content of each page).

What input parameters are available for crawl and scrape?

Both tools accept: url (required), headers, user_agent, depth, blacklist, whitelist, respect_robots_txt, and accept_invalid_certs.

How do I install and run mcp-server-spider?

Recommended: use uvx with no separate installation. Alternatively, install via pip install mcp-server-spider and run with python -m mcp_server_spider.