crawl4-mcp

Name: crawl4-mcp
Author: ShiDuLin

@ShiDuLin

Repository

About crawl4-mcp

本项目是一个crawl4ai 的爬虫MCP SERVER，提供高级网络爬虫。使用此 MCP SERVER，您可以抓取任何内容，将内容保存为本地markdown文件，然后在任何地方将该知识用于 RAG。

#crawl4ai #web-scraper #data-collection

Basic information

Config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "crawl4-mcp": {
      "command": "uv",
      "args": [
        "venv",
        "-p",
        "3.12"
      ]
    }
  }
}

Tools

No tools detected

We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.

Overview

What is crawl4-mcp?

crawl4-mcp is an MCP server built on crawl4ai that provides advanced web crawling capabilities. It allows users to crawl any web content, save it as local markdown files, and use that knowledge anywhere for RAG (Retrieval-Augmented Generation).

How to use crawl4-mcp?

Install by cloning the repository, creating a Python 3.12 virtual environment with uv, running uv sync and crawl4ai-setup, then creating a .env file. Start the server with uv run src/crawl4ai_mcp.py. The server listens on the configured host and port via SSE transport. Integrate with any MCP client using transport: sse and url: http://localhost:8051/sse.

Key features of crawl4-mcp

Crawls any web content with advanced crawl4ai engine
Saves content as local markdown files
Output ready for RAG (Retrieval-Augmented Generation)
SSE-based MCP server integration
Simple Python 3.12+ setup with uv

Use cases of crawl4-mcp

Building a local knowledge base from web pages for RAG pipelines
Archiving website content as markdown for offline reference
Feeding crawled data into AI assistants or document retrieval systems

FAQ from crawl4-mcp

What are the environment requirements?

Python 3.12 or higher and the uv package manager are required.

How do I install crawl4-mcp?

Clone the repository, create a Python 3.12 virtual environment with uv, run uv sync and crawl4ai-setup, then create a .env file.

How do I run the server?

Execute uv run src/crawl4ai_mcp.py in the activated virtual environment.

How do I integrate crawl4-mcp with an MCP client?

Configure the client with transport type sse and the URL http://localhost:8051/sse (adjust host/port as needed).

What happens to the crawled data?

The content is saved as local markdown files, ready to be used for RAG or other applications.

Comments

More Other MCP servers

Sequential Thinking

modelcontextprotocol

Model Context Protocol Servers

88.2K

Activepieces

activepieces

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

23.2K