広告掲載
MCP.so
ログイン

Documentation Crawler & MCP Server

@alizdavoodi

リポジトリ

基本情報

カテゴリ

メモリとナレッジ

ライセンス

MIT

トランスポート

stdio

作者

alizdavoodi

Documentation Crawler & MCP Server について

This project provides a toolset to crawl websites wikis, tool/library documentions and generate Markdown documentation, and make that documentation searchable via a Model Context Protocol (MCP) server, designed for integration with tools like Cursor.

#crawler#mcp#mcp-server

Documentation Crawler & MCP Server の設定

以下の設定を使って、このサーバーを MCP 対応クライアントに追加してください。

{
  "mcpServers": {
    "MCPDocSearch": {
      "command": "uv",
      "args": [
        "sync"
      ]
    }
  }
}

Documentation Crawler & MCP Server のツール

ツール一覧はまだありません

この掲載には表示できるツール一覧がまだありません。サーバーにツールがないという意味ではありません。対応機能はプロジェクトのドキュメントをご確認ください。

ツールは README から自動的に抽出されます。メンテナーは ## Tools という見出しの下に記載することで、このタブに反映できます。

概要

What is Documentation Crawler & MCP Server?

A toolset that crawls websites, generates Markdown documentation, and makes the content searchable via a Model Context Protocol (MCP) server for integration with tools like Cursor.

How to use Documentation Crawler & MCP Server?

Crawl a website using uv run python crawl.py <start-url> with optional filters. Then configure your MCP client (e.g., Cursor) to launch the server with uv run python -m mcp_server.main. The server automatically loads, chunks, and embeds documents from the ./storage/ directory, caching results for faster subsequent startups.

Key features of Documentation Crawler & MCP Server

  • Web crawler with configurable depth, URL patterns, and content filters.
  • Generates a single consolidated Markdown file from crawled content.
  • MCP server provides semantic search via vector embeddings.
  • Automatic caching of processed chunks and embeddings for fast restarts.
  • Exposes MCP tools: list_documents, get_document_headings, search_documentation.
  • Designed for use with stdio transport and Cursor integration.

Use cases of Documentation Crawler & MCP Server

  • Crawl and search any public documentation site (e.g., Apache Pulsar) from within Cursor.
  • Maintain an offline, searchable knowledge base of your own project's documentation.
  • Enable an AI coding assistant to query specific sections of local documentation.

よくある質問

1

What are the main dependencies and runtime requirements?

Python 3 with `uv`, and key libraries: `crawl4ai`, `fastmcp`, `sentence-transformers`, `torch`. A compatible GPU (CUDA or Apple Silicon/MPS) speeds up embedding generation, but CPU works.

2

Where does the crawled data and cache reside?

All Markdown files are stored in the `./storage/` directory. The server's processed chunks and embeddings are cached in `storage/document_chunks_cache.pkl`. The cache is invalidated when any `.md` file changes.

3

What transport and authentication does the MCP server use?

The server runs via `stdio` transport (no HTTP). No authentication is mentioned in the README; it relies on the local execution environment.

4

Are there known limits on crawl depth or embedding time?

Crawl depth must be between 1 and 5. Embedding generation time varies significantly based on hardware (GPU vs CPU) and total document size; first startup after crawling may take several minutes.

5

What tools does the MCP server expose?

`list_documents` (list available docs), `get_document_headings` (retrieve heading structure), and `search_documentation` (semantic search over document chunks).

コメント

「メモリとナレッジ」の他のコンテンツ

server-noonien - shared memory MCP

sequico

Shared, serverless, conflict-free memory for AI agents: a multi-machine drop-in for the MCP memory knowledge graph.

priostack

ideaswave

Open-source Python SDK & MCP interface for Priostack Agent Context Network (ACN). Zero-setup, autonomous memory orchestration for AI agents.

M1K3

Round-Tower

Local AI agent for your Mac with a built-in MCP server: 18 tools for your notes and PDFs, a temporal memory graph, an on-device LLM and a neural voice. No account, no telemetry. No data centers required.

7

Context7 MCP - Up-to-date Docs For Any Cursor Prompt

upstash

Context7 Platform -- Up-to-date code documentation for LLMs and AI code editors

58.7K

Scento Perfume API (MCP server): free fragrance API, no key

Scento

Scento Perfume API is a free fragrance API and perfume notes database for developers and AI agents. Search 1,500+ perfumes and 400+ ingredients, compare scents, find similar perfumes, map scent families and build occasio

広告掲載