mcp-server-for-rag

What is mcp-server-for-rag?

mcp-server-for-rag is a microservice-based MCP server that crawls web pages, indexes them in Elasticsearch, and enables retrieval-augmented generation (RAG) search from Claude Desktop, Cline, or a built-in browser chat UI. It runs entirely in Docker, so no local environment setup is needed.

How to use mcp-server-for-rag?

Run ./run.sh to start Elasticsearch, the MCP API, chat UI, and optionally the embedding API. Then run ./run-crawler.sh to crawl and index documents. For MCP hosts like Claude Desktop, add a docker exec command to claude_desktop_config.json. Optionally, set an OpenAI API key in openai_token.txt for semantic search; otherwise keyword search is used.

Key features of mcp-server-for-rag

• Crawls web pages and indexes them in Elasticsearch.
• Provides MCP tools: search, get_document_by_id, list_elasticsearch_indices.
• Supports both keyword and semantic (vector) search.
• Includes a browser-based chat UI (localhost:15173).
• Offers a dedicated rag_mode prompt for Claude Desktop.
• Runs entirely in Docker with no local dependencies.

Use cases of mcp-server-for-rag

• Add RAG search to Claude Desktop or Cline for private documentation.
• Build a searchable knowledge base from crawled internal or public web pages.
• Test and prototype RAG without installing Python or Elasticsearch locally.
• Create a custom chat interface that retrieves information from indexed documents.

FAQ from mcp-server-for-rag

How does semantic search differ from keyword search?

Semantic search requires an OpenAI API key set in openai_token.txt. If the file is missing or empty, the system falls back to keyword-only search.

What are the runtime requirements?

Docker and Docker Compose are required. For semantic search, an OpenAI API key is needed. All services run inside Docker containers.

Where is the indexed data stored?

Data is stored locally in Elasticsearch, which runs inside a Docker container managed by the run.sh script.

What transport protocols does the MCP server support?

The server supports Streamable HTTP (default) and SSE (Server-Sent Events). The communication mode is selected via environment files (.env.streamable_http or .env.sse).

How do I configure the crawler?

Crawler behavior is defined in YAML files inside crawler_config/. You can set start URLs, allowed domains, URL patterns, crawl depth, delay, user-agent, Elasticsearch index name, and optional embedding settings.