広告掲載
MCP.so
ログイン

Crawl4AI RAG MCP Server

@Anshumaan031

リポジトリ

基本情報

カテゴリ

メモリとナレッジ

ライセンス

MIT

トランスポート

stdio

作者

Anshumaan031

Crawl4AI RAG MCP Server について

A distributed Model Context Protocol (MCP) server implementing an intelligent web ingestion and retrieval-augmented generation pipeline for AI agents and coding assistants.

Crawl4AI RAG MCP Server の設定

以下の設定を使って、このサーバーを MCP 対応クライアントに追加してください。

{
  "mcpServers": {
    "Crawl4AI-RAG-MCP-Server": {
      "command": "docker",
      "args": [
        "build",
        "-t",
        "mcp/crawl4ai-rag",
        "--build-arg",
        "PORT=8051",
        "."
      ]
    }
  }
}

Crawl4AI RAG MCP Server のツール

ツール一覧はまだありません

この掲載には表示できるツール一覧がまだありません。サーバーにツールがないという意味ではありません。対応機能はプロジェクトのドキュメントをご確認ください。

ツールは README から自動的に抽出されます。メンテナーは ## Tools という見出しの下に記載することで、このタブに反映できます。

概要

What is Crawl4AI RAG MCP Server?

Crawl4AI RAG MCP Server is an MCP implementation integrated with Crawl4AI and Supabase, providing AI agents and coding assistants with advanced web crawling and RAG capabilities. It enables scraping web content and storing it in a vector database for semantic search and retrieval-augmented generation.

How to use Crawl4AI RAG MCP Server?

Install via Docker (recommended) or directly with Python/uv. Configure a .env file with OPENAI_API_KEY, SUPABASE_URL, and SUPABASE_SERVICE_KEY. Run the server using docker run or uv run src/crawl4ai_mcp.py. Connect MCP clients via SSE (default) or stdio by adding the appropriate configuration to the client’s mcpServers block. The server provides four tools: crawl_single_page, smart_crawl_url, get_available_sources, and perform_rag_query.

Key features of Crawl4AI RAG MCP Server

  • Smart URL detection (webpages, sitemaps, text files).
  • Recursive crawling of internal links.
  • Parallel processing for efficient crawling.
  • Intelligent content chunking by headers and size.
  • Vector search with optional source filtering.
  • Retrieval of available data sources for guided RAG.

Use cases of Crawl4AI RAG MCP Server

  • Crawl a single web page and store its content for later retrieval.
  • Full website crawling via sitemap or recursive link following.
  • Semantic search over crawled content with source-specific filters.
  • Build a RAG knowledge base from arbitrary web content.

よくある質問

1

What is Crawl4AI RAG MCP Server?

It is an MCP server that integrates Crawl4AI for web crawling and Supabase as a vector database, enabling AI agents to scrape content and use it for retrieval-augmented generation (RAG).

2

What are the prerequisites for installation?

You need Docker Desktop (for containerized deployment) or Python 3.12+ (for direct execution), a Supabase instance for the vector database, and an OpenAI API key for generating embeddings.

3

What tools does the server provide?

The server provides four tools: `crawl_single_page` (crawl one page), `smart_crawl_url` (intelligently crawl a full site), `get_available_sources` (list domains in the database), and `perform_rag_query` (semantic search with optional source filtering).

4

How do I connect the server to my MCP client?

Use SSE transport by configuring the client with `"transport": "sse"` and the server’s URL (e.g., `http://localhost:8051/sse`). For stdio transport, use the command `uv run src/crawl4ai_mcp.py` or a Docker command, and pass environment variables as shown in the README.

5

Where is crawled data stored?

All crawled content is stored in a Supabase database with the pgvector extension enabled. The server uses `SUPABASE_URL` and `SUPABASE_SERVICE_KEY` from the `.env` file to connect.

コメント

「メモリとナレッジ」の他のコンテンツ

The Cloud

Teka

The Cloud is the AI-native workspace that gives every AI — Claude, ChatGPT, Gemini, and other Frontier & Free or Local models — shared memory and real agents. Not cloud computing: thecloud.so is one workspace where your

Orcha

westonhancock

Orcha is a unified context layer for AI tools and agents. It stores and indexes and exposes organizational, team, or individual knowledge. Files, structured databases, and connected sources live in one workspace, and age

LoreKeeper RPG (lorekeeper.com)

LoreKeeper Ltd

Mess in, magic out: pour in your D&D and TTRPG notes and LoreKeeper keeps them as a private store for your world. Ask Claude or ChatGPT about it, or build a sourcebook.

Codebase Memory

DeusData

High-performance code intelligence MCP server. Indexes codebases into a persistent knowledge graph — average repo in milliseconds. 162 languages, sub-ms queries, 99% fewer tokens. Single static binary, zero dependencies.

46.7K

Sooveryn

PANDAPROG

Sooveryn gives Claude Code a team of AI personas that remembers your project. Soov, the partner and team lead, keeps your decisions and how you work in a lasting memory, and finds them again by words and by meaning, from

広告掲載