crawl4ai-mcp

What is crawl4ai-mcp?

crawl4ai-mcp is a Model Context Protocol (MCP) server that provides AI systems with access to the live web. Built on FastMCP v3 and Crawl4AI, it exposes 4 tools, 2 resources, and 3 prompts through a standardized MCP interface, backed by a lifespan-managed headless Chromium browser.

How to use crawl4ai-mcp?

Install via pip install mcp-crawl4ai or uv add mcp-crawl4ai, then run mcp-crawl4ai for stdio transport (default) or mcp-crawl4ai --transport http --port 8000 for HTTP. For Claude Desktop, add to MCP settings with the command mcp-crawl4ai and args ["--transport", "stdio"].

Key features of crawl4ai-mcp

• Full MCP compliance via FastMCP v3 with tool annotations
• 4 focused tools for scrape, crawl, session lifecycle, and artifacts
• 3 prompts for common LLM workflows (summarize, extract schema, compare)
• 2 resources exposing server configuration and version info
• Headless Chromium managed as a lifespan singleton
• Multiple transports: stdio (default) and Streamable HTTP
• LLM-optimized output in markdown, cleaned HTML, raw HTML, or plain text
• Auto browser setup and detection of missing Playwright browsers

Use cases of crawl4ai-mcp

• AI agents that need to scrape or crawl web pages for real-time information
• Summarizing a page with a specified focus using the summarize_page prompt
• Building a CSS extraction schema for structured data from a URL
• Comparing two web pages and producing a structured comparison
• Deep traversal of a website using recursive BFS/DFS from a single seed

FAQ from crawl4ai-mcp

What is the difference between crawl4ai-mcp and using Crawl4AI directly?

crawl4ai-mcp wraps Crawl4AI in an MCP server with only 2 runtime dependencies (fastmcp and crawl4ai), providing a standardized interface for AI tools and LLM clients. It adds session management, resource endpoints, and prompt templates.

What are the runtime dependencies of crawl4ai-mcp?

The server requires Python 3.13+ and has exactly two runtime dependencies: fastmcp and crawl4ai. Playwright browsers are needed for headless Chromium but are installed automatically or via mcp-crawl4ai --setup.

Does crawl4ai-mcp require a browser to be installed?

Yes, it uses headless Chromium. The server auto-detects missing Playwright browsers and attempts to install them on first startup, or you can run mcp-crawl4ai --setup manually. The Docker image includes browsers pre-installed.

What transports does crawl4ai-mcp support?

Stdio (default) for desktop clients like Claude Desktop and Cursor, and HTTP (Streamable) for remote access. HTTP binds to 127.0.0.1:8000 by default; use --host and a reverse proxy for external exposure.

Are there any known limits in crawl4ai-mcp?

The scrape tool accepts up to 20 URLs in a bounded list. The crawl tool supports both list traversal (bounded) and deep traversal (recursive BFS/DFS) from a single seed. There are configurable limits exposed via the config://server resource.