Documentation Crawler & MCP Server

What is Documentation Crawler & MCP Server?

A toolset that crawls websites, generates Markdown documentation, and makes the content searchable via a Model Context Protocol (MCP) server for integration with tools like Cursor.

How to use Documentation Crawler & MCP Server?

Crawl a website using uv run python crawl.py <start-url> with optional filters. Then configure your MCP client (e.g., Cursor) to launch the server with uv run python -m mcp_server.main. The server automatically loads, chunks, and embeds documents from the ./storage/ directory, caching results for faster subsequent startups.

Key features of Documentation Crawler & MCP Server

• Web crawler with configurable depth, URL patterns, and content filters.
• Generates a single consolidated Markdown file from crawled content.
• MCP server provides semantic search via vector embeddings.
• Automatic caching of processed chunks and embeddings for fast restarts.
• Exposes MCP tools: list_documents, get_document_headings, search_documentation.
• Designed for use with stdio transport and Cursor integration.

Use cases of Documentation Crawler & MCP Server

• Crawl and search any public documentation site (e.g., Apache Pulsar) from within Cursor.
• Maintain an offline, searchable knowledge base of your own project's documentation.
• Enable an AI coding assistant to query specific sections of local documentation.

FAQ from Documentation Crawler & MCP Server

What are the main dependencies and runtime requirements?

Python 3 with uv, and key libraries: crawl4ai, fastmcp, sentence-transformers, torch. A compatible GPU (CUDA or Apple Silicon/MPS) speeds up embedding generation, but CPU works.

Where does the crawled data and cache reside?

All Markdown files are stored in the ./storage/ directory. The server's processed chunks and embeddings are cached in storage/document_chunks_cache.pkl. The cache is invalidated when any .md file changes.

What transport and authentication does the MCP server use?

The server runs via stdio transport (no HTTP). No authentication is mentioned in the README; it relies on the local execution environment.

Are there known limits on crawl depth or embedding time?

Crawl depth must be between 1 and 5. Embedding generation time varies significantly based on hardware (GPU vs CPU) and total document size; first startup after crawling may take several minutes.

What tools does the MCP server expose?

list_documents (list available docs), get_document_headings (retrieve heading structure), and search_documentation (semantic search over document chunks).