Crawlab MCP Server

Name: Crawlab MCP Server
Author: crawlab-team

@crawlab-team

代码仓库

关于 Crawlab MCP Server

暂无概览

#crawlab #mcp-server #ai-integration

基本信息

分类

其他

运行时

python

传输方式

stdio

发布者

crawlab-team

配置

使用下面的配置,将此服务器添加到你的 MCP 客户端。

{
  "mcpServers": {
    "crawlab-mcp-server": {
      "type": "http",
      "url": "http://localhost:8000"
    }
  }
}

工具

未检测到工具

工具是从 README 中自动提取的。维护者可以在 ## Tools 标题下列出工具,即可填充这部分内容。

概览

What is Crawlab MCP Server?

The Crawlab MCP Server is a Model Context Protocol (MCP) server that enables AI applications to interact with Crawlab’s spider, task, and file management features through natural language. It bridges an MCP client (such as Claude Desktop) with the Crawlab API, allowing users to create, run, and manage web scraping workflows without manual API calls.

How to use Crawlab MCP Server?

Install as a Python package (pip install -e .) and run the server CLI (crawlab_mcp-mcp server), or run directly with python server.py after setting up a .env file with your Crawlab API base URL and token. A Docker image is also available. Connect an AI client to the server’s URL (e.g., http://localhost:8000) to start issuing natural language commands.

Key features of Crawlab MCP Server

MCP‑based communication between AI apps and Crawlab
Full spider lifecycle: create, read, update, delete
Task management: run, cancel, restart, view logs
File operations: list, read, write spider files
Supports multiple deployment options (local, Docker, pip)
Configuration via environment variables (.env)

Use cases of Crawlab MCP Server

Create and configure new spiders using natural language
Launch scraping tasks on demand and monitor progress
Retrieve and update spider code files through a chat interface
Cancel or restart tasks without leaving the AI application

FAQ from Crawlab MCP Server

What is the Crawlab MCP Server?

It is a standard MCP server that translates AI application tool calls into Crawlab API requests, enabling natural‑language control of Crawlab’s spider, task, and file management features.

How do I install the Crawlab MCP Server?

You can install it as a Python package (pip install -e .) and run via CLI, or run python server.py directly. A Docker build is also provided.

What are the prerequisites to run the server?

Python 3.8+, a running Crawlab instance, and a valid Crawlab API token. The server connects to Crawlab via its API, so the instance must be accessible.

How is authentication handled?

The server reads the Crawlab API token from the CRAWLAB_API_TOKEN environment variable (set in .env) and uses it to authenticate all requests to the Crawlab API.

What transport does the MCP server use?

The server communicates over HTTP. Clients connect to its URL (e.g., http://localhost:8000), and tool calls are executed via RESTful API calls to the Crawlab backend.