广告合作
MCP.so
登录

mcp-server-spider: A spider MCP server

@GeorgeLS

代码仓库

基本信息

分类

其他

许可证

MIT

传输方式

stdio

作者

GeorgeLS

关于 mcp-server-spider: A spider MCP server

暂无概览

mcp-server-spider: A spider MCP server 配置

使用下面的配置,将此服务器添加到你的 MCP 客户端。

{
  "mcpServers": {
    "mcp-server-spider": {
      "command": "uvx",
      "args": [
        "mcp-server-spider"
      ]
    }
  }
}

mcp-server-spider: A spider MCP server 工具

4

`url`: The url to crawl

List of URLs found

Same as `crawl`

A list of JSON objects (as a string) that contain the url, links and content of each page discovered

概览

What is mcp-server-spider?

mcp-server-spider is a Model Context Protocol (MCP) server that provides tools for Spider crawler interaction and automation. It allows crawling and scraping web pages via MCP-compatible clients. The server is currently in early development.

How to use mcp-server-spider?

Install and run using uvx (recommended) or install via pip (pip install mcp-server-spider) and run with python -m mcp_server_spider. The server exposes two tools: crawl and scrape, each accepting parameters such as url, headers, user_agent, depth, blacklist, whitelist, respect_robots_txt, and accept_invalid_certs.

Key features of mcp-server-spider

  • Crawl a URL and return discovered links
  • Scrape a URL and return structured JSON with content
  • Configurable crawl depth and user agent
  • Support for additional HTTP headers
  • Whitelist/blacklist URLs via regular expressions
  • Option to respect or bypass robots.txt
  • Option to accept invalid TLS certificates

Use cases of mcp-server-spider

  • Automate web crawling for link discovery
  • Extract structured content from web pages
  • Integrate web scraping into MCP-enabled workflows
  • Build custom crawlers with depth constraints and filtering

常见问题

1

What tools does mcp-server-spider provide?

Two tools: `crawl` (returns list of URLs found) and `scrape` (returns a JSON string with URL, links, and content of each page).

2

What input parameters are available for crawl and scrape?

Both tools accept: `url` (required), `headers`, `user_agent`, `depth`, `blacklist`, `whitelist`, `respect_robots_txt`, and `accept_invalid_certs`.

3

How do I install and run mcp-server-spider?

Recommended: use `uvx` with no separate installation. Alternatively, install via `pip install mcp-server-spider` and run with `python -m mcp_server_spider`.

4

Is mcp-server-spider stable?

The README notes it is in early development; there may be bugs and features added in the future.

评论

其他 分类下的更多 MCP 服务器

MaxKB

1Panel-dev

🔥 MaxKB is an open-source platform for building enterprise-grade agents. 强大易用的开源企业级智能体平台。

22.0K

Hindsight

vectorize-io

Hindsight: Agent Memory That Learns

50.4K

The Living Bread

The Living Bread

A free remote MCP server for The Living Bread. Bible passages read word for word from a stored public domain text with an evidence label, keyword search and cross references, real churches, gatherings and prayer groups w

Glasswarp

Glasswarp

See and control a real Windows PC you own — from any MCP client, locally or remotely. Observe (UIA + screenshots), click/type/drag/scroll, launch apps, owner Live View. BYOH: your machine, your key.

P

PickFu MCP

PickFu MCP is a Model Context Protocol (MCP) server that connects AI assistants, developers, marketers, product managers, and AI agents to PickFu's consumer research platform. It enables users to create surveys, conduct

广告合作