Documentation Crawler & MCP Server

@alizdavoodi

代码仓库

基本信息

分类

记忆与知识

许可证

MIT

传输方式

stdio

作者

alizdavoodi

关于 Documentation Crawler & MCP Server

This project provides a toolset to crawl websites wikis, tool/library documentions and generate Markdown documentation, and make that documentation searchable via a Model Context Protocol (MCP) server, designed for integration with tools like Cursor.

#crawler #mcp #mcp-server

Documentation Crawler & MCP Server 配置

使用下面的配置,将此服务器添加到你的 MCP 客户端。

{
  "mcpServers": {
    "MCPDocSearch": {
      "command": "uv",
      "args": [
        "sync"
      ]
    }
  }
}

Documentation Crawler & MCP Server 工具

工具列表暂未提供

当前收录暂时没有可展示的工具列表，不代表服务没有工具。请查阅项目文档了解支持的能力。

工具是从 README 中自动提取的。维护者可以在 ## Tools 标题下列出工具,即可填充这部分内容。

概览

What is Documentation Crawler & MCP Server?

A toolset that crawls websites, generates Markdown documentation, and makes the content searchable via a Model Context Protocol (MCP) server for integration with tools like Cursor.

How to use Documentation Crawler & MCP Server?

Crawl a website using uv run python crawl.py <start-url> with optional filters. Then configure your MCP client (e.g., Cursor) to launch the server with uv run python -m mcp_server.main. The server automatically loads, chunks, and embeds documents from the ./storage/ directory, caching results for faster subsequent startups.

Key features of Documentation Crawler & MCP Server

Web crawler with configurable depth, URL patterns, and content filters.
Generates a single consolidated Markdown file from crawled content.
MCP server provides semantic search via vector embeddings.
Automatic caching of processed chunks and embeddings for fast restarts.
Exposes MCP tools: list_documents, get_document_headings, search_documentation.
Designed for use with stdio transport and Cursor integration.

Use cases of Documentation Crawler & MCP Server

Crawl and search any public documentation site (e.g., Apache Pulsar) from within Cursor.
Maintain an offline, searchable knowledge base of your own project's documentation.
Enable an AI coding assistant to query specific sections of local documentation.

常见问题

What are the main dependencies and runtime requirements?

Python 3 with `uv`, and key libraries: `crawl4ai`, `fastmcp`, `sentence-transformers`, `torch`. A compatible GPU (CUDA or Apple Silicon/MPS) speeds up embedding generation, but CPU works.

Where does the crawled data and cache reside?

All Markdown files are stored in the `./storage/` directory. The server's processed chunks and embeddings are cached in `storage/document_chunks_cache.pkl`. The cache is invalidated when any `.md` file changes.

What transport and authentication does the MCP server use?

The server runs via `stdio` transport (no HTTP). No authentication is mentioned in the README; it relies on the local execution environment.

Are there known limits on crawl depth or embedding time?

Crawl depth must be between 1 and 5. Embedding generation time varies significantly based on hardware (GPU vs CPU) and total document size; first startup after crawling may take several minutes.

What tools does the MCP server expose?

`list_documents` (list available docs), `get_document_headings` (retrieve heading structure), and `search_documentation` (semantic search over document chunks).

记忆与知识分类下的更多 MCP 服务器

LoreKeeper RPG (lorekeeper.com)

LoreKeeper Ltd

Mess in, magic out: pour in your D&D and TTRPG notes and LoreKeeper keeps them as a private store for your world. Ask Claude or ChatGPT about it, or build a sourcebook.