Crawl4ai Rag

Name: Crawl4ai Rag
Author: coleam00

@coleam00

リポジトリ

Crawl4ai Rag について

概要はまだありません

#crawl4ai #web-crawling #ai-coding-assistant

基本情報

カテゴリ

メモリとナレッジ

トランスポート

stdio

公開者

coleam00

投稿者

Tariq Forde

設定

以下の設定を使って、このサーバーを MCP 対応クライアントに追加してください。

{
  "mcpServers": {
    "crawl4ai-rag": {
      "command": "python",
      "args": [
        "path/to/crawl4ai-mcp/src/crawl4ai_mcp.py"
      ],
      "env": {
        "TRANSPORT": "stdio",
        "OPENAI_API_KEY": "your_openai_api_key",
        "SUPABASE_URL": "your_supabase_url",
        "SUPABASE_SERVICE_KEY": "your_supabase_service_key",
        "USE_KNOWLEDGE_GRAPH": "false",
        "NEO4J_URI": "bolt://localhost:7687",
        "NEO4J_USER": "neo4j",
        "NEO4J_PASSWORD": "your_neo4j_password"
      }
    }
  }
}

ツール

ツールは検出されませんでした

ツールは README から自動的に抽出されます。メンテナーは ## Tools という見出しの下に記載することで、このタブに反映できます。

概要

What is Crawl4ai Rag?

A Model Context Protocol (MCP) server that integrates Crawl4AI web crawling with Supabase vector storage to provide AI agents and coding assistants with web crawling and RAG (Retrieval-Augmented Generation) capabilities.

How to use Crawl4ai Rag?

Install via Docker or uv (Python 3.12+), configure a .env file with Supabase and OpenAI API key, set up a Supabase database with pgvector, and optionally set up Neo4j for knowledge graph features. The server exposes MCP tools for crawling, searching, and querying crawled content.

Key features of Crawl4ai Rag

Smart URL detection (regular pages, sitemaps, text files)
Recursive crawling following internal links
Parallel processing for efficient multi-page crawling
Content chunking by headers and size
Vector search with optional source filtering
Advanced RAG strategies (contextual embeddings, hybrid search, agentic RAG, reranking, knowledge graph)

Use cases of Crawl4ai Rag

AI coding assistants scraping and indexing documentation for code-aware RAG
Building a knowledge engine for AI agents to crawl and query websites
Performing semantic search over previously crawled content with source filtering
Detecting AI hallucinations by validating code against a repository knowledge graph

FAQ from Crawl4ai Rag

What dependencies does Crawl4ai Rag require?

Requires Docker or Python 3.12+, a Supabase project (for RAG), and an OpenAI API key. Neo4j is optional for knowledge graph functionality.

How are RAG strategies configured?

Each strategy (contextual embeddings, hybrid search, agentic RAG, reranking, knowledge graph) is enabled via a USE_* environment variable set to "true" in the .env file. All default to "false".

Where is crawled content stored?

Content is stored in a Supabase database with the pgvector extension. A separate table is created for agentic RAG code examples if enabled. Knowledge graph data is stored in a Neo4j graph database.

What transport does the server use?

The server uses SSE (Server-Sent Events) transport by default, configurable via the TRANSPORT environment variable.

Are there known limitations or planned improvements?

The repository is a testbed for integration into Archon; known areas for improvement include supporting multiple embedding models, local Ollama execution, faster crawling/indexing, and enhanced chunking strategies.

「メモリとナレッジ」の他のコンテンツ

🧠 Ultimate MCP Server

Dicklesworthstone

Comprehensive MCP server exposing dozens of capabilities to AI agents: multi-provider LLM delegation, browser automation, document processing, vector ops, and cognitive memory systems

154