Crawl4ai Middleware

Name: Crawl4ai Middleware
Author: Frankieli123

@Frankieli123

Repository

About Crawl4ai Middleware

No overview available yet

#crawl4ai #web-scraping #middleware

Basic information

Config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "crawl4ai": {
      "command": "python",
      "args": [
        "E:/APP/craw14ai-middleware/crawl4ai_server.py"
      ]
    }
  }
}

Tools

No tools detected

We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.

Overview

What is Crawl4ai Middleware?

Crawl4ai Middleware is an MCP (Model Context Protocol) server that integrates crawl4ai's web scraping capabilities into MCP-compatible applications like Cursor IDE. It provides tools to create crawl tasks, retrieve results by task ID, list saved files, and read their contents.

How to use Crawl4ai Middleware?

Install dependencies with pip install mcp[cli] httpx python-dotenv, configure environment variables (API base URL and key), then add the server configuration to your MCP client (e.g., Cursor IDE's settings.json) pointing to the crawl4ai_server.py script.

Key features of Crawl4ai Middleware

Provides four MCP tools for web scraping workflows
Saves scraped content locally to avoid display size limits
Supports environment variable configuration for API and output directory
Allows retrieval of crawl results by task ID
Lists and reads saved result files
Works with Cursor IDE and other MCP-enabled applications

Use cases of Crawl4ai Middleware

Scrape a web page and retrieve its content directly in Cursor IDE
Manage multiple crawl tasks with persistent file storage
Review previously crawled results offline at any time
Integrate web scraping into AI agent workflows via MCP

FAQ from Crawl4ai Middleware

What tools does Crawl4ai Middleware provide?

It provides create_crawl_task, get_crawl_result, list_saved_results, and read_saved_result.

How are crawled files stored?

Files are saved in a url folder next to the server script, named extracted_domain_taskID_timestamp.txt.

What dependencies are required?

Install mcp[cli], httpx, and python-dotenv via pip.

How do I configure the server?

Set environment variables CRAWL4AI_API_BASE, CRAWL4AI_API_KEY, optionally OUTPUT_DIR and LOG_LEVEL. Default API base is http://192.168.31.12:11235 and key sk-3180623.

Can I use this with Cursor IDE?

Yes, add the server configuration to Cursor's settings.json with the command python and args pointing to the crawl4ai_server.py script.

Comments

More Other MCP servers

🚀 Model Context Protocol (MCP) Curriculum for Beginners

microsoft

This open-source curriculum introduces the fundamentals of Model Context Protocol (MCP) through real-world, cross-language examples in .NET, Java, TypeScript, JavaScript, Rust and Python. Designed for developers, it focuses on practical techniques for building modular, scalable,

16.7K