Overview
What is Crawl4AI MCP Server?
Crawl4AI Documentation Crawler is a CLI tool that crawls documentation websites using the Crawl4AI library and converts pages to Markdown files. It supports single‑page and recursive deep crawling with URL prefix filtering, automatic navigation/footer removal, and domain‑based output directory generation.
How to use Crawl4AI MCP Server?
Install dependencies with uv sync. Use the CLI command uv run cli.py crawl <url> [options] to crawl a page or entire site. Specify --recursive for deep crawling, -o to save to a directory, and --prefix to filter URLs. Run uv run cli.py config-list to see available preset configurations.
Key features of Crawl4AI MCP Server
- Converts web pages to clean Markdown files
- Recursive deep crawl with configurable depth and page limits
- URL prefix filter for selective crawling of specific paths
- Automatic removal of navigation and footer content
- Domain‑based directory organization for output files
- Asynchronous crawling using AsyncIO for performance
Use cases of Crawl4AI MCP Server
- Crawl and archive a full product documentation site as Markdown
- Extract specific sections of a large documentation site using prefix filtering
- Create a local offline backup of API references or developer guides
- Convert multi‑page documentation into structured file trees for further processing
FAQ from Crawl4AI MCP Server
What is the difference between single‑page and deep crawl mode?
Single‑page mode crawls only the given URL and outputs Markdown to the console or a single file. Deep crawl mode recursively follows links on the same domain to spider the entire site, saving pages in a directory tree.
What are the runtime and dependency requirements?
The tool requires Python 3.13+ and uses uv for dependency management. It depends on Crawl4AI, Typer, and AsyncIO.
Where does the crawled data get saved?
If no output directory is specified, Markdown appears on the console. With -o, data is saved to a directory named after the domain (e.g., docs_crawl4ai_com/). Deep crawl mode always saves to a directory.
Are there any limits on crawling?
Yes: for deep crawl, you can set --max-pages (default 100) and --max-depth (default 2). Only same‑domain pages are crawled automatically.
Does the tool support authentication or private sites?
The README does not mention any authentication support or transport protocols. It is designed for public documentation sites only.
