Crawlbase Web MCP Server

@crawlbase

代码仓库主页

基本信息

分类

其他

传输方式

stdio

作者

crawlbase

提交者

Muhammad Waleed

关于 Crawlbase Web MCP Server

Crawlbase MCP Server connects AI agents and LLMs with real-time web data. It powers Claude, Cursor, and Windsurf integrations with battle-tested web scraping, JavaScript rendering, and anti-bot protection enabling structured, live data inside your AI workflows.

#web-scraping#web-scraping-mcp#web-crawling-mcp#web-mcp-server#web-data-mcp#web-data #ai #agents

Crawlbase Web MCP Server 配置

使用下面的配置,将此服务器添加到你的 MCP 客户端。

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "@crawlbase/mcp@latest"
      ],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

Crawlbase Web MCP Server 工具

Crawl a URL and return HTML content

Crawl a URL and extract clean markdown content

Take a screenshot of a webpage

概览

What is Crawlbase Web MCP Server?

Crawlbase Web MCP Server is a Model Context Protocol (MCP) server that bridges AI agents with live web content. It uses Crawlbase’s crawling infrastructure, trusted by over 70,000 developers, to fetch fresh, structured, real-time data from the web. It handles JavaScript rendering, proxy rotation, anti-bot evasion, and returns HTML, Markdown, or screenshots.

How to use Crawlbase Web MCP Server?

Sign up at Crawlbase to get free Normal and JavaScript tokens. Configure the MCP server in a client like Claude Desktop, Claude Code, Cursor IDE, or Windsurf IDE by adding its JSON configuration with your tokens. Then use commands like crawl, crawl_markdown, or crawl_screenshot to fetch web data. An HTTP transport mode is also available for multi‑user or custom integrations.

Key features of Crawlbase Web MCP Server

Fetches live web data for AI agents
JavaScript rendering for modern web apps
Proxy rotation and anti‑bot evasion
Structured outputs: HTML, Markdown, screenshots
Async crawling with Crawlbase Cloud Storage
HTTP transport with per‑request token authentication

Use cases of Crawlbase Web MCP Server

Market research by pulling live competitor data and reports
E‑commerce monitoring for real‑time product, review, and price tracking
News and finance feeds to keep AI agents current with events
Autonomous AI agents that can act on fresh web data

常见问题

What do I need to get started?

You need free tokens from Crawlbase (a Normal token for HTML and a JavaScript token for JS‑rendered pages). Then configure the MCP server in your AI client.

Which AI clients are supported?

Claude Desktop, Claude Code, Cursor IDE, and Windsurf IDE are explicitly supported via stdio transport. HTTP transport mode is also available for any HTTP‑capable client.

Does it handle JavaScript‑heavy sites?

Yes. The server uses Crawlbase’s JavaScript rendering capability, which requires a JavaScript token. It handles modern web apps and anti‑bot measures.

Can I store crawl results for later use?

Yes. Crawlbase MCP can push results to Crawlbase Cloud Storage. Your AI agent can later read, list, or delete stored pages across sessions.

How does authentication work in HTTP mode?

Authentication is via environment variables or per‑request headers (`X-Crawlbase-Token` and `X-Crawlbase-JS-Token`). Headers override environment variables, enabling multi‑tenant deployments.

其他分类下的更多 MCP 服务器

Handl

Handl speaks the Model Context Protocol. Your whole billing operation — projects, invoices, clients, reports, approvals — as tools any AI assistant can call, inside your autonomy rules.

Klarix Intelligence Engine

Klarix

B2B competitive intelligence, live sales battlecards, 5-axis ICP prospect fit scoring, and tech stack teardowns over MCP and REST with cited public-web evidence.

Screenpipe

screenpipe

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

22.0K