Crawlab MCP Server

What is Crawlab MCP Server?

The Crawlab MCP Server is a Model Context Protocol (MCP) server that enables AI applications to interact with Crawlab’s spider, task, and file management features through natural language. It bridges an MCP client (such as Claude Desktop) with the Crawlab API, allowing users to create, run, and manage web scraping workflows without manual API calls.

How to use Crawlab MCP Server?

Install as a Python package (pip install -e .) and run the server CLI (crawlab_mcp-mcp server), or run directly with python server.py after setting up a .env file with your Crawlab API base URL and token. A Docker image is also available. Connect an AI client to the server’s URL (e.g., http://localhost:8000) to start issuing natural language commands.

Key features of Crawlab MCP Server

• MCP‑based communication between AI apps and Crawlab
• Full spider lifecycle: create, read, update, delete
• Task management: run, cancel, restart, view logs
• File operations: list, read, write spider files
• Supports multiple deployment options (local, Docker, pip)
• Configuration via environment variables (.env)

Use cases of Crawlab MCP Server

• Create and configure new spiders using natural language
• Launch scraping tasks on demand and monitor progress
• Retrieve and update spider code files through a chat interface
• Cancel or restart tasks without leaving the AI application

FAQ from Crawlab MCP Server

What is the Crawlab MCP Server?

It is a standard MCP server that translates AI application tool calls into Crawlab API requests, enabling natural‑language control of Crawlab’s spider, task, and file management features.

How do I install the Crawlab MCP Server?

You can install it as a Python package (pip install -e .) and run via CLI, or run python server.py directly. A Docker build is also provided.

What are the prerequisites to run the server?

Python 3.8+, a running Crawlab instance, and a valid Crawlab API token. The server connects to Crawlab via its API, so the instance must be accessible.

How is authentication handled?

The server reads the Crawlab API token from the CRAWLAB_API_TOKEN environment variable (set in .env) and uses it to authenticate all requests to the Crawlab API.

What transport does the MCP server use?

The server communicates over HTTP. Clients connect to its URL (e.g., http://localhost:8000), and tool calls are executed via RESTful API calls to the Crawlab backend.