NOTICE

What is Crawl4AI MCP Server?

Crawl4AI MCP Server is an open-source implementation of the Model Context Protocol (MCP) that integrates with the Crawl4AI web scraping and crawling library. It is deployed as a remote MCP server on CloudFlare Workers, enabling AI assistants such as Claude to perform web scraping, crawling, and structured data extraction.

How to use Crawl4AI MCP Server?

Install Node.js (v18+), npm, and Wrangler, then clone the repository, run npm install, create a CloudFlare KV namespace, update wrangler.toml, and deploy with npm run deploy. Connect MCP clients (e.g., Claude Desktop) using the assigned CloudFlare Workers URL. Available tools include crawl, getCrawl, listCrawls, search, and extract.

Key features of Crawl4AI MCP Server

• Single webpage scraping and full website crawling
• Configurable crawl depth and page limits
• Asynchronous crawling for efficient site traversal
• Deep research across multiple pages
• Structured data extraction using CSS selectors or LLM extraction
• OAuth and API key (Bearer token) authentication

Use cases of Crawl4AI MCP Server

• AI assistants retrieving live web content on demand
• Automated deep research of multiple linked pages
• Extracting specified data fields from web pages for analysis
• Building a searchable index of crawled content

FAQ from Crawl4AI MCP Server

Is this server ready for production use?

No. The README explicitly states the server is under development and not ready for production use.

What are the prerequisites for running the server?

You need Node.js v18+, npm, the Wrangler CLI, and a CloudFlare account. A CloudFlare KV namespace is required for storing crawl data.

How is crawled data stored?

Data is stored in a CloudFlare KV namespace configured in wrangler.toml under the binding CRAWL_DATA.

What authentication methods are supported?

The server supports both OAuth authentication (via workers-oauth-provider) and API key authentication using Bearer tokens.

Which MCP clients are compatible?

The server implements the standard Model Context Protocol and can be used with any MCP client, such as Claude Desktop.