Advertise
MCP.so
Sign In
C

Crawl4ai Rag MCP Server

@coleam00

Repository

Basic information

Category

Memory & Knowledge

Transports

stdio

Author

coleam00

Submitted by

Tariq Forde

About Crawl4ai Rag MCP Server

No overview available yet

#crawl4ai#web-crawling#ai-coding-assistant

Crawl4ai Rag MCP Server config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "crawl4ai-rag": {
      "command": "python",
      "args": [
        "path/to/crawl4ai-mcp/src/crawl4ai_mcp.py"
      ],
      "env": {
        "TRANSPORT": "stdio",
        "OPENAI_API_KEY": "your_openai_api_key",
        "SUPABASE_URL": "your_supabase_url",
        "SUPABASE_SERVICE_KEY": "your_supabase_service_key",
        "USE_KNOWLEDGE_GRAPH": "false",
        "NEO4J_URI": "bolt://localhost:7687",
        "NEO4J_USER": "neo4j",
        "NEO4J_PASSWORD": "your_neo4j_password"
      }
    }
  }
}

Crawl4ai Rag MCP Server tools

Tool list not available yet

This listing does not currently have a tool list to display. This does not mean the server has no tools; check its documentation for supported capabilities.

We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.

Overview

What is Crawl4ai Rag?

A Model Context Protocol (MCP) server that integrates Crawl4AI web crawling with Supabase vector storage to provide AI agents and coding assistants with web crawling and RAG (Retrieval-Augmented Generation) capabilities.

How to use Crawl4ai Rag?

Install via Docker or uv (Python 3.12+), configure a .env file with Supabase and OpenAI API key, set up a Supabase database with pgvector, and optionally set up Neo4j for knowledge graph features. The server exposes MCP tools for crawling, searching, and querying crawled content.

Key features of Crawl4ai Rag

  • Smart URL detection (regular pages, sitemaps, text files)
  • Recursive crawling following internal links
  • Parallel processing for efficient multi-page crawling
  • Content chunking by headers and size
  • Vector search with optional source filtering
  • Advanced RAG strategies (contextual embeddings, hybrid search, agentic RAG, reranking, knowledge graph)

Use cases of Crawl4ai Rag

  • AI coding assistants scraping and indexing documentation for code-aware RAG
  • Building a knowledge engine for AI agents to crawl and query websites
  • Performing semantic search over previously crawled content with source filtering
  • Detecting AI hallucinations by validating code against a repository knowledge graph

Frequently asked questions

1

What dependencies does Crawl4ai Rag require?

Requires Docker or Python 3.12+, a Supabase project (for RAG), and an OpenAI API key. Neo4j is optional for knowledge graph functionality.

2

How are RAG strategies configured?

Each strategy (contextual embeddings, hybrid search, agentic RAG, reranking, knowledge graph) is enabled via a `USE_*` environment variable set to `"true"` in the `.env` file. All default to `"false"`.

3

Where is crawled content stored?

Content is stored in a Supabase database with the pgvector extension. A separate table is created for agentic RAG code examples if enabled. Knowledge graph data is stored in a Neo4j graph database.

4

What transport does the server use?

The server uses SSE (Server-Sent Events) transport by default, configurable via the `TRANSPORT` environment variable.

5

Are there known limitations or planned improvements?

The repository is a testbed for integration into Archon; known areas for improvement include supporting multiple embedding models, local Ollama execution, faster crawling/indexing, and enhanced chunking strategies.

Comments

More Memory & Knowledge MCP servers

HandbookHub MCP

veryayskiy

Search saved employee handbooks from your AI assistant. Four read-only tools list authorized handbooks, browse policies, search content, and read source pages with citation links. Uses scoped bearer tokens and Streamable

PLUR

plur-ai

AI agents start every session with amnesia — you re-explain the project, repeat your preferences, and correct the same mistakes over and over. PLUR gives them a memory that persists. Your agent's corrections, preference

226

server-noonien - shared memory MCP

sequico

Shared, serverless, conflict-free memory for AI agents: a multi-machine drop-in for the MCP memory knowledge graph.

ctxfile

ctxfile

Local-first MCP server that snapshots your project's working state into one context object. Open-core, privacy-first.

2

AOCI-CODE

aoci-spec

Local MCP server and CLI that keeps a Git-versioned, model-written index of a codebase and its database schema, so coding agents start each task with the whole system in view.

1.7K
Advertise