Crawl4AI RAG MCP Server

What is Crawl4AI RAG MCP Server?

Crawl4AI RAG MCP Server is an MCP implementation integrated with Crawl4AI and Supabase, providing AI agents and coding assistants with advanced web crawling and RAG capabilities. It enables scraping web content and storing it in a vector database for semantic search and retrieval-augmented generation.

How to use Crawl4AI RAG MCP Server?

Install via Docker (recommended) or directly with Python/uv. Configure a .env file with OPENAI_API_KEY, SUPABASE_URL, and SUPABASE_SERVICE_KEY. Run the server using docker run or uv run src/crawl4ai_mcp.py. Connect MCP clients via SSE (default) or stdio by adding the appropriate configuration to the client’s mcpServers block. The server provides four tools: crawl_single_page, smart_crawl_url, get_available_sources, and perform_rag_query.

Key features of Crawl4AI RAG MCP Server

• Smart URL detection (webpages, sitemaps, text files).
• Recursive crawling of internal links.
• Parallel processing for efficient crawling.
• Intelligent content chunking by headers and size.
• Vector search with optional source filtering.
• Retrieval of available data sources for guided RAG.

Use cases of Crawl4AI RAG MCP Server

• Crawl a single web page and store its content for later retrieval.
• Full website crawling via sitemap or recursive link following.
• Semantic search over crawled content with source-specific filters.
• Build a RAG knowledge base from arbitrary web content.

FAQ from Crawl4AI RAG MCP Server

What is Crawl4AI RAG MCP Server?

It is an MCP server that integrates Crawl4AI for web crawling and Supabase as a vector database, enabling AI agents to scrape content and use it for retrieval-augmented generation (RAG).

What are the prerequisites for installation?

You need Docker Desktop (for containerized deployment) or Python 3.12+ (for direct execution), a Supabase instance for the vector database, and an OpenAI API key for generating embeddings.

What tools does the server provide?

The server provides four tools: crawl_single_page (crawl one page), smart_crawl_url (intelligently crawl a full site), get_available_sources (list domains in the database), and perform_rag_query (semantic search with optional source filtering).

How do I connect the server to my MCP client?

Use SSE transport by configuring the client with "transport": "sse" and the server’s URL (e.g., http://localhost:8051/sse). For stdio transport, use the command uv run src/crawl4ai_mcp.py or a Docker command, and pass environment variables as shown in the README.

Where is crawled data stored?

All crawled content is stored in a Supabase database with the pgvector extension enabled. The server uses SUPABASE_URL and SUPABASE_SERVICE_KEY from the .env file to connect.