Crawl4AI Web Scraper MCP Server

What is Crawl4AI Web Scraper MCP Server?

This project provides an MCP (Model Context Protocol) server that uses the crawl4ai library to perform web scraping and intelligent content extraction tasks. It allows AI agents (such as Claude, or agents built with LangChain/LangGraph) to interact with web pages, retrieve content, search for specific text, and perform LLM-based extraction based on natural language instructions. The server uses FastMCP, crawl4ai, dotenv, and optionally Docker.

How to use Crawl4AI Web Scraper MCP Server?

Run the server either with Docker (recommended) or locally. For Docker: install Docker, clone the repository, create a .env file with GOOGLE_API_KEY, build the image, and run the container mapping port 8002 with --env-file .env. For local use: ensure Python ≥3.9, clone the repo, create a virtual environment, install dependencies from requirements.txt, create a .env file with API keys, and run the main Python script. The server listens on SSE at port 8002. Configure your MCP client to connect to http://127.0.0.1:8002/sse with transport "sse".

Key features of Crawl4AI Web Scraper MCP Server

• Exposes three MCP tools: scrape_url, extract_text_by_query, and smart_extract
• scrape_url returns full webpage content as Markdown
• extract_text_by_query finds up to 5 text snippet matches with optional context size
• smart_extract uses Google Gemini LLM for structured extraction from natural language instructions
• Configurable via environment variables (API keys)
• Includes Docker configuration for containerized deployment
• Communicates over Server-Sent Events (SSE) on port 8002

Use cases of Crawl4AI Web Scraper MCP Server

• AI agents scraping webpage content into Markdown format
• Searching for specific text on a page with surrounding context
• Extracting structured information (e.g., speakers list, contact email, key findings) via LLM
• Integrating web scraping capabilities into Claude, LangChain, or LangGraph agents

FAQ from Crawl4AI Web Scraper MCP Server

What API key is required for the smart_extract tool?

A valid Google Gemini API key must be set as GOOGLE_API_KEY in the .env file. OpenAI and Mistral keys are checked but not currently used.

How do I connect my MCP client to this server?

Configure your MCP client to connect to http://127.0.0.1:8002/sse with transport set to "sse".

Can the server be run without Docker?

Yes, it can be run locally by installing Python dependencies and executing the main server script.

What web scraping tools does the server expose?

Three tools: scrape_url, extract_text_by_query, and smart_extract.

What are the runtime requirements?

Python 3.9 or higher (3.10+ recommended) or Docker, plus a .env file with required API keys.