广告合作
MCP.so
登录

Crawl4AI RAG MCP Server

@Anshumaan031

代码仓库

基本信息

分类

记忆与知识

许可证

MIT

传输方式

stdio

作者

Anshumaan031

关于 Crawl4AI RAG MCP Server

A distributed Model Context Protocol (MCP) server implementing an intelligent web ingestion and retrieval-augmented generation pipeline for AI agents and coding assistants.

Crawl4AI RAG MCP Server 配置

使用下面的配置,将此服务器添加到你的 MCP 客户端。

{
  "mcpServers": {
    "Crawl4AI-RAG-MCP-Server": {
      "command": "docker",
      "args": [
        "build",
        "-t",
        "mcp/crawl4ai-rag",
        "--build-arg",
        "PORT=8051",
        "."
      ]
    }
  }
}

Crawl4AI RAG MCP Server 工具

工具列表暂未提供

当前收录暂时没有可展示的工具列表，不代表服务没有工具。请查阅项目文档了解支持的能力。

工具是从 README 中自动提取的。维护者可以在 ## Tools 标题下列出工具,即可填充这部分内容。

概览

What is Crawl4AI RAG MCP Server?

Crawl4AI RAG MCP Server is an MCP implementation integrated with Crawl4AI and Supabase, providing AI agents and coding assistants with advanced web crawling and RAG capabilities. It enables scraping web content and storing it in a vector database for semantic search and retrieval-augmented generation.

How to use Crawl4AI RAG MCP Server?

Install via Docker (recommended) or directly with Python/uv. Configure a .env file with OPENAI_API_KEY, SUPABASE_URL, and SUPABASE_SERVICE_KEY. Run the server using docker run or uv run src/crawl4ai_mcp.py. Connect MCP clients via SSE (default) or stdio by adding the appropriate configuration to the client’s mcpServers block. The server provides four tools: crawl_single_page, smart_crawl_url, get_available_sources, and perform_rag_query.

Key features of Crawl4AI RAG MCP Server

  • Smart URL detection (webpages, sitemaps, text files).
  • Recursive crawling of internal links.
  • Parallel processing for efficient crawling.
  • Intelligent content chunking by headers and size.
  • Vector search with optional source filtering.
  • Retrieval of available data sources for guided RAG.

Use cases of Crawl4AI RAG MCP Server

  • Crawl a single web page and store its content for later retrieval.
  • Full website crawling via sitemap or recursive link following.
  • Semantic search over crawled content with source-specific filters.
  • Build a RAG knowledge base from arbitrary web content.

常见问题

1

What is Crawl4AI RAG MCP Server?

It is an MCP server that integrates Crawl4AI for web crawling and Supabase as a vector database, enabling AI agents to scrape content and use it for retrieval-augmented generation (RAG).

2

What are the prerequisites for installation?

You need Docker Desktop (for containerized deployment) or Python 3.12+ (for direct execution), a Supabase instance for the vector database, and an OpenAI API key for generating embeddings.

3

What tools does the server provide?

The server provides four tools: `crawl_single_page` (crawl one page), `smart_crawl_url` (intelligently crawl a full site), `get_available_sources` (list domains in the database), and `perform_rag_query` (semantic search with optional source filtering).

4

How do I connect the server to my MCP client?

Use SSE transport by configuring the client with `"transport": "sse"` and the server’s URL (e.g., `http://localhost:8051/sse`). For stdio transport, use the command `uv run src/crawl4ai_mcp.py` or a Docker command, and pass environment variables as shown in the README.

5

Where is crawled data stored?

All crawled content is stored in a Supabase database with the pgvector extension enabled. The server uses `SUPABASE_URL` and `SUPABASE_SERVICE_KEY` from the `.env` file to connect.

评论

记忆与知识 分类下的更多 MCP 服务器

Sensefold

SenseFold Labs

Personal context for every AI: search, read, and write back to your private Markdown library of articles, threads, PDFs, notes, and captured ChatGPT/Claude/Gemini/Grok conversations. Remote server, OAuth 2.1 paste-and-au

Memlord

MyrikLD

Memlord is a remote MCP server available at https://app.memlord.com.

AOCI-CODE

aoci-spec

Local MCP server and CLI that keeps a Git-versioned, model-written index of a codebase and its database schema, so coding agents start each task with the whole system in view.

1.7K

M1K3

Round-Tower

Local AI agent for your Mac with a built-in MCP server: 18 tools for your notes and PDFs, a temporal memory graph, an on-device LLM and a neural voice. No account, no telemetry. No data centers required.

7

Takibi Base

Deian Isac

Ask your Takibi Base collections anything; get answers with cited passages and a support score.

广告合作