Crawl4ai Middleware

Name: Crawl4ai Middleware
Author: Frankieli123

@Frankieli123

代码仓库

关于 Crawl4ai Middleware

暂无概览

#crawl4ai #web-scraping #middleware

基本信息

分类

其他

传输方式

stdio

发布者

Frankieli123

提交者

配置

使用下面的配置,将此服务器添加到你的 MCP 客户端。

{
  "mcpServers": {
    "crawl4ai": {
      "command": "python",
      "args": [
        "E:/APP/craw14ai-middleware/crawl4ai_server.py"
      ]
    }
  }
}

工具

未检测到工具

工具是从 README 中自动提取的。维护者可以在 ## Tools 标题下列出工具,即可填充这部分内容。

概览

What is Crawl4ai Middleware?

Crawl4ai Middleware is an MCP (Model Context Protocol) server that integrates crawl4ai's web scraping capabilities into MCP-compatible applications like Cursor IDE. It provides tools to create crawl tasks, retrieve results by task ID, list saved files, and read their contents.

How to use Crawl4ai Middleware?

Install dependencies with pip install mcp[cli] httpx python-dotenv, configure environment variables (API base URL and key), then add the server configuration to your MCP client (e.g., Cursor IDE's settings.json) pointing to the crawl4ai_server.py script.

Key features of Crawl4ai Middleware

Provides four MCP tools for web scraping workflows
Saves scraped content locally to avoid display size limits
Supports environment variable configuration for API and output directory
Allows retrieval of crawl results by task ID
Lists and reads saved result files
Works with Cursor IDE and other MCP-enabled applications

Use cases of Crawl4ai Middleware

Scrape a web page and retrieve its content directly in Cursor IDE
Manage multiple crawl tasks with persistent file storage
Review previously crawled results offline at any time
Integrate web scraping into AI agent workflows via MCP

FAQ from Crawl4ai Middleware

What tools does Crawl4ai Middleware provide?

It provides create_crawl_task, get_crawl_result, list_saved_results, and read_saved_result.

How are crawled files stored?

Files are saved in a url folder next to the server script, named extracted_domain_taskID_timestamp.txt.

What dependencies are required?

Install mcp[cli], httpx, and python-dotenv via pip.

How do I configure the server?

Set environment variables CRAWL4AI_API_BASE, CRAWL4AI_API_KEY, optionally OUTPUT_DIR and LOG_LEVEL. Default API base is http://192.168.31.12:11235 and key sk-3180623.