mcp-server-spider: A spider MCP server
@GeorgeLS
About mcp-server-spider: A spider MCP server
No overview available yet
Basic information
Config
Add this server to your MCP-compatible client using the configuration below.
{
"mcpServers": {
"mcp-server-spider": {
"command": "uvx",
"args": [
"mcp-server-spider"
]
}
}
}Tools
4`url`: The url to crawl
List of URLs found
Same as `crawl`
A list of JSON objects (as a string) that contain the url, links and content of each page discovered
Overview
What is mcp-server-spider?
mcp-server-spider is a Model Context Protocol (MCP) server that provides tools for Spider crawler interaction and automation. It allows crawling and scraping web pages via MCP-compatible clients. The server is currently in early development.
How to use mcp-server-spider?
Install and run using uvx (recommended) or install via pip (pip install mcp-server-spider) and run with python -m mcp_server_spider. The server exposes two tools: crawl and scrape, each accepting parameters such as url, headers, user_agent, depth, blacklist, whitelist, respect_robots_txt, and accept_invalid_certs.
Key features of mcp-server-spider
- Crawl a URL and return discovered links
- Scrape a URL and return structured JSON with content
- Configurable crawl depth and user agent
- Support for additional HTTP headers
- Whitelist/blacklist URLs via regular expressions
- Option to respect or bypass
robots.txt - Option to accept invalid TLS certificates
Use cases of mcp-server-spider
- Automate web crawling for link discovery
- Extract structured content from web pages
- Integrate web scraping into MCP-enabled workflows
- Build custom crawlers with depth constraints and filtering
FAQ from mcp-server-spider
What tools does mcp-server-spider provide?
Two tools: crawl (returns list of URLs found) and scrape (returns a JSON string with URL, links, and content of each page).
What input parameters are available for crawl and scrape?
Both tools accept: url (required), headers, user_agent, depth, blacklist, whitelist, respect_robots_txt, and accept_invalid_certs.
How do I install and run mcp-server-spider?
Recommended: use uvx with no separate installation. Alternatively, install via pip install mcp-server-spider and run with python -m mcp_server_spider.
Is mcp-server-spider stable?
The README notes it is in early development; there may be bugs and features added in the future.
More Other MCP servers
Inbox Zero AI MCP
elie222The world's best AI personal assistant for email. Open source app to help you reach inbox zero fast.
MCP Go 🚀
mark3labsA Go implementation of the Model Context Protocol (MCP), enabling seamless integration between LLM applications and external data sources and tools.
Reactive Resume
amruthpillaiA one-of-a-kind resume builder that keeps your privacy in mind. Completely secure, customizable, portable, open-source and free forever. Try it out today!
Unity MCP ✨
justinpbarnettUnity MCP acts as a bridge between AI assistants and your Unity Editor. Give your LLM tools to manage assets, control scenes, edit scripts, and automate tasks within Unity.
Servers
modelcontextprotocolModel Context Protocol Servers
Comments