Documentation Crawler & MCP Server

@alizdavoodi

Basic information

About Documentation Crawler & MCP Server

This project provides a toolset to crawl websites wikis, tool/library documentions and generate Markdown documentation, and make that documentation searchable via a Model Context Protocol (MCP) server, designed for integration with tools like Cursor.

#crawler #mcp #mcp-server

Documentation Crawler & MCP Server config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "MCPDocSearch": {
      "command": "uv",
      "args": [
        "sync"
      ]
    }
  }
}

Documentation Crawler & MCP Server tools

Tool list not available yet

This listing does not currently have a tool list to display. This does not mean the server has no tools; check its documentation for supported capabilities.

We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.

Overview

What is Documentation Crawler & MCP Server?

A toolset that crawls websites, generates Markdown documentation, and makes the content searchable via a Model Context Protocol (MCP) server for integration with tools like Cursor.

How to use Documentation Crawler & MCP Server?

Crawl a website using uv run python crawl.py <start-url> with optional filters. Then configure your MCP client (e.g., Cursor) to launch the server with uv run python -m mcp_server.main. The server automatically loads, chunks, and embeds documents from the ./storage/ directory, caching results for faster subsequent startups.

Key features of Documentation Crawler & MCP Server

Web crawler with configurable depth, URL patterns, and content filters.
Generates a single consolidated Markdown file from crawled content.
MCP server provides semantic search via vector embeddings.
Automatic caching of processed chunks and embeddings for fast restarts.
Exposes MCP tools: list_documents, get_document_headings, search_documentation.
Designed for use with stdio transport and Cursor integration.

Use cases of Documentation Crawler & MCP Server

Crawl and search any public documentation site (e.g., Apache Pulsar) from within Cursor.
Maintain an offline, searchable knowledge base of your own project's documentation.
Enable an AI coding assistant to query specific sections of local documentation.

Frequently asked questions

What are the main dependencies and runtime requirements?

Python 3 with `uv`, and key libraries: `crawl4ai`, `fastmcp`, `sentence-transformers`, `torch`. A compatible GPU (CUDA or Apple Silicon/MPS) speeds up embedding generation, but CPU works.

Where does the crawled data and cache reside?

All Markdown files are stored in the `./storage/` directory. The server's processed chunks and embeddings are cached in `storage/document_chunks_cache.pkl`. The cache is invalidated when any `.md` file changes.

What transport and authentication does the MCP server use?

The server runs via `stdio` transport (no HTTP). No authentication is mentioned in the README; it relies on the local execution environment.

Are there known limits on crawl depth or embedding time?

Crawl depth must be between 1 and 5. Embedding generation time varies significantly based on hardware (GPU vs CPU) and total document size; first startup after crawling may take several minutes.

What tools does the MCP server expose?

`list_documents` (list available docs), `get_document_headings` (retrieve heading structure), and `search_documentation` (semantic search over document chunks).

Comments

More Memory & Knowledge MCP servers

PLUR

plur-ai

AI agents start every session with amnesia — you re-explain the project, repeat your preferences, and correct the same mistakes over and over. PLUR gives them a memory that persists. Your agent's corrections, preference

226

AOCI-CODE

aoci-spec

Local MCP server and CLI that keeps a Git-versioned, model-written index of a codebase and its database schema, so coding agents start each task with the whole system in view.

1.7K

Agimem

Persistent memory for AI agents. Hosted MCP server with capsule-scoped key/value storage, tags, TTL expiration, bulk ops, and RBAC.

ctxfile

Local-first MCP server that snapshots your project's working state into one context object. Open-core, privacy-first.

agentleFS

One permissioned folder your team and your AI agents both log into.