Crawl4AI RAG MCP Server

@Anshumaan031

Basic information

About Crawl4AI RAG MCP Server

A distributed Model Context Protocol (MCP) server implementing an intelligent web ingestion and retrieval-augmented generation pipeline for AI agents and coding assistants.

Crawl4AI RAG MCP Server config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "Crawl4AI-RAG-MCP-Server": {
      "command": "docker",
      "args": [
        "build",
        "-t",
        "mcp/crawl4ai-rag",
        "--build-arg",
        "PORT=8051",
        "."
      ]
    }
  }
}

Crawl4AI RAG MCP Server tools

Tool list not available yet

This listing does not currently have a tool list to display. This does not mean the server has no tools; check its documentation for supported capabilities.

We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.

Overview

What is Crawl4AI RAG MCP Server?

Crawl4AI RAG MCP Server is an MCP implementation integrated with Crawl4AI and Supabase, providing AI agents and coding assistants with advanced web crawling and RAG capabilities. It enables scraping web content and storing it in a vector database for semantic search and retrieval-augmented generation.

How to use Crawl4AI RAG MCP Server?

Install via Docker (recommended) or directly with Python/uv. Configure a .env file with OPENAI_API_KEY, SUPABASE_URL, and SUPABASE_SERVICE_KEY. Run the server using docker run or uv run src/crawl4ai_mcp.py. Connect MCP clients via SSE (default) or stdio by adding the appropriate configuration to the client’s mcpServers block. The server provides four tools: crawl_single_page, smart_crawl_url, get_available_sources, and perform_rag_query.

Key features of Crawl4AI RAG MCP Server

Smart URL detection (webpages, sitemaps, text files).
Recursive crawling of internal links.
Parallel processing for efficient crawling.
Intelligent content chunking by headers and size.
Vector search with optional source filtering.
Retrieval of available data sources for guided RAG.

Use cases of Crawl4AI RAG MCP Server

Crawl a single web page and store its content for later retrieval.
Full website crawling via sitemap or recursive link following.
Semantic search over crawled content with source-specific filters.
Build a RAG knowledge base from arbitrary web content.

Frequently asked questions

What is Crawl4AI RAG MCP Server?

It is an MCP server that integrates Crawl4AI for web crawling and Supabase as a vector database, enabling AI agents to scrape content and use it for retrieval-augmented generation (RAG).

What are the prerequisites for installation?

You need Docker Desktop (for containerized deployment) or Python 3.12+ (for direct execution), a Supabase instance for the vector database, and an OpenAI API key for generating embeddings.

What tools does the server provide?

The server provides four tools: `crawl_single_page` (crawl one page), `smart_crawl_url` (intelligently crawl a full site), `get_available_sources` (list domains in the database), and `perform_rag_query` (semantic search with optional source filtering).

How do I connect the server to my MCP client?

Use SSE transport by configuring the client with `"transport": "sse"` and the server’s URL (e.g., `http://localhost:8051/sse`). For stdio transport, use the command `uv run src/crawl4ai_mcp.py` or a Docker command, and pass environment variables as shown in the README.

Where is crawled data stored?

All crawled content is stored in a Supabase database with the pgvector extension enabled. The server uses `SUPABASE_URL` and `SUPABASE_SERVICE_KEY` from the `.env` file to connect.

Comments

More Memory & Knowledge MCP servers

LoreKeeper RPG (lorekeeper.com)

LoreKeeper Ltd

Mess in, magic out: pour in your D&D and TTRPG notes and LoreKeeper keeps them as a private store for your world. Ask Claude or ChatGPT about it, or build a sourcebook.