mcp-server-spider: A spider MCP server
@GeorgeLS
关于 mcp-server-spider: A spider MCP server
暂无概览
基本信息
配置
使用下面的配置,将此服务器添加到你的 MCP 客户端。
{
"mcpServers": {
"mcp-server-spider": {
"command": "uvx",
"args": [
"mcp-server-spider"
]
}
}
}工具
4`url`: The url to crawl
List of URLs found
Same as `crawl`
A list of JSON objects (as a string) that contain the url, links and content of each page discovered
概览
What is mcp-server-spider?
mcp-server-spider is a Model Context Protocol (MCP) server that provides tools for Spider crawler interaction and automation. It allows crawling and scraping web pages via MCP-compatible clients. The server is currently in early development.
How to use mcp-server-spider?
Install and run using uvx (recommended) or install via pip (pip install mcp-server-spider) and run with python -m mcp_server_spider. The server exposes two tools: crawl and scrape, each accepting parameters such as url, headers, user_agent, depth, blacklist, whitelist, respect_robots_txt, and accept_invalid_certs.
Key features of mcp-server-spider
- Crawl a URL and return discovered links
- Scrape a URL and return structured JSON with content
- Configurable crawl depth and user agent
- Support for additional HTTP headers
- Whitelist/blacklist URLs via regular expressions
- Option to respect or bypass
robots.txt - Option to accept invalid TLS certificates
Use cases of mcp-server-spider
- Automate web crawling for link discovery
- Extract structured content from web pages
- Integrate web scraping into MCP-enabled workflows
- Build custom crawlers with depth constraints and filtering
FAQ from mcp-server-spider
What tools does mcp-server-spider provide?
Two tools: crawl (returns list of URLs found) and scrape (returns a JSON string with URL, links, and content of each page).
What input parameters are available for crawl and scrape?
Both tools accept: url (required), headers, user_agent, depth, blacklist, whitelist, respect_robots_txt, and accept_invalid_certs.
How do I install and run mcp-server-spider?
Recommended: use uvx with no separate installation. Alternatively, install via pip install mcp-server-spider and run with python -m mcp_server_spider.
Is mcp-server-spider stable?
The README notes it is in early development; there may be bugs and features added in the future.
其他 分类下的更多 MCP 服务器
Blender
ahujasidOpen-source MCP to use Blender with any LLM
EverArt
modelcontextprotocolModel Context Protocol Servers
Awesome-MCP-ZH
yzflyMCP 资源精选， MCP指南，Claude MCP，MCP Servers, MCP Clients
Activepieces
activepiecesAI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
Mcp
browsermcpBrowser MCP is a Model Context Provider (MCP) server that allows AI applications to control your browser
评论