Crawl4ai Middleware

What is Crawl4ai Middleware?

Crawl4ai Middleware is an MCP (Model Context Protocol) server that integrates crawl4ai's web scraping capabilities into MCP-compatible applications like Cursor IDE. It provides tools to create crawl tasks, retrieve results by task ID, list saved files, and read their contents.

How to use Crawl4ai Middleware?

Install dependencies with pip install mcp[cli] httpx python-dotenv, configure environment variables (API base URL and key), then add the server configuration to your MCP client (e.g., Cursor IDE's settings.json) pointing to the crawl4ai_server.py script.

Key features of Crawl4ai Middleware

• Provides four MCP tools for web scraping workflows
• Saves scraped content locally to avoid display size limits
• Supports environment variable configuration for API and output directory
• Allows retrieval of crawl results by task ID
• Lists and reads saved result files
• Works with Cursor IDE and other MCP-enabled applications

Use cases of Crawl4ai Middleware

• Scrape a web page and retrieve its content directly in Cursor IDE
• Manage multiple crawl tasks with persistent file storage
• Review previously crawled results offline at any time
• Integrate web scraping into AI agent workflows via MCP

FAQ from Crawl4ai Middleware

What tools does Crawl4ai Middleware provide?

It provides create_crawl_task, get_crawl_result, list_saved_results, and read_saved_result.

How are crawled files stored?

Files are saved in a url folder next to the server script, named extracted_domain_taskID_timestamp.txt.

What dependencies are required?

Install mcp[cli], httpx, and python-dotenv via pip.

How do I configure the server?

Set environment variables CRAWL4AI_API_BASE, CRAWL4AI_API_KEY, optionally OUTPUT_DIR and LOG_LEVEL. Default API base is http://192.168.31.12:11235 and key sk-3180623.

Can I use this with Cursor IDE?

Yes, add the server configuration to Cursor's settings.json with the command python and args pointing to the crawl4ai_server.py script.