Crawl4ai Rag

Name: Crawl4ai Rag
Author: coleam00

@coleam00

代码仓库

关于 Crawl4ai Rag

暂无概览

#crawl4ai #web-crawling #ai-coding-assistant

基本信息

分类

记忆与知识

传输方式

stdio

发布者

coleam00

提交者

Tariq Forde

配置

使用下面的配置,将此服务器添加到你的 MCP 客户端。

{
  "mcpServers": {
    "crawl4ai-rag": {
      "command": "python",
      "args": [
        "path/to/crawl4ai-mcp/src/crawl4ai_mcp.py"
      ],
      "env": {
        "TRANSPORT": "stdio",
        "OPENAI_API_KEY": "your_openai_api_key",
        "SUPABASE_URL": "your_supabase_url",
        "SUPABASE_SERVICE_KEY": "your_supabase_service_key",
        "USE_KNOWLEDGE_GRAPH": "false",
        "NEO4J_URI": "bolt://localhost:7687",
        "NEO4J_USER": "neo4j",
        "NEO4J_PASSWORD": "your_neo4j_password"
      }
    }
  }
}

工具

未检测到工具

工具是从 README 中自动提取的。维护者可以在 ## Tools 标题下列出工具,即可填充这部分内容。

概览

What is Crawl4ai Rag?

A Model Context Protocol (MCP) server that integrates Crawl4AI web crawling with Supabase vector storage to provide AI agents and coding assistants with web crawling and RAG (Retrieval-Augmented Generation) capabilities.

How to use Crawl4ai Rag?

Install via Docker or uv (Python 3.12+), configure a .env file with Supabase and OpenAI API key, set up a Supabase database with pgvector, and optionally set up Neo4j for knowledge graph features. The server exposes MCP tools for crawling, searching, and querying crawled content.

Key features of Crawl4ai Rag

Smart URL detection (regular pages, sitemaps, text files)
Recursive crawling following internal links
Parallel processing for efficient multi-page crawling
Content chunking by headers and size
Vector search with optional source filtering
Advanced RAG strategies (contextual embeddings, hybrid search, agentic RAG, reranking, knowledge graph)

Use cases of Crawl4ai Rag

AI coding assistants scraping and indexing documentation for code-aware RAG
Building a knowledge engine for AI agents to crawl and query websites
Performing semantic search over previously crawled content with source filtering
Detecting AI hallucinations by validating code against a repository knowledge graph

FAQ from Crawl4ai Rag

What dependencies does Crawl4ai Rag require?

Requires Docker or Python 3.12+, a Supabase project (for RAG), and an OpenAI API key. Neo4j is optional for knowledge graph functionality.

How are RAG strategies configured?

Each strategy (contextual embeddings, hybrid search, agentic RAG, reranking, knowledge graph) is enabled via a USE_* environment variable set to "true" in the .env file. All default to "false".

Where is crawled content stored?

Content is stored in a Supabase database with the pgvector extension. A separate table is created for agentic RAG code examples if enabled. Knowledge graph data is stored in a Neo4j graph database.

What transport does the server use?

The server uses SSE (Server-Sent Events) transport by default, configurable via the TRANSPORT environment variable.

Are there known limitations or planned improvements?

The repository is a testbed for integration into Archon; known areas for improvement include supporting multiple embedding models, local Ollama execution, faster crawling/indexing, and enhanced chunking strategies.

记忆与知识分类下的更多 MCP 服务器

Anytype MCP Server

anyproto

An MCP server enabling AI assistants to interact with Anytype - your encrypted, local and collaborative wiki - to organize objects, lists, and more through natural language.

467