Crawlbase Web MCP Server

@crawlbase

リポジトリホームページ

基本情報

カテゴリ

その他

トランスポート

stdio

作者

crawlbase

投稿者

Muhammad Waleed

Crawlbase Web MCP Server について

Crawlbase MCP Server connects AI agents and LLMs with real-time web data. It powers Claude, Cursor, and Windsurf integrations with battle-tested web scraping, JavaScript rendering, and anti-bot protection enabling structured, live data inside your AI workflows.

#web-scraping#web-scraping-mcp#web-crawling-mcp#web-mcp-server#web-data-mcp#web-data #ai #agents

Crawlbase Web MCP Server の設定

以下の設定を使って、このサーバーを MCP 対応クライアントに追加してください。

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "@crawlbase/mcp@latest"
      ],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

Crawlbase Web MCP Server のツール

Crawl a URL and return HTML content

Crawl a URL and extract clean markdown content

Take a screenshot of a webpage

概要

What is Crawlbase Web MCP Server?

Crawlbase Web MCP Server is a Model Context Protocol (MCP) server that bridges AI agents with live web content. It uses Crawlbase’s crawling infrastructure, trusted by over 70,000 developers, to fetch fresh, structured, real-time data from the web. It handles JavaScript rendering, proxy rotation, anti-bot evasion, and returns HTML, Markdown, or screenshots.

How to use Crawlbase Web MCP Server?

Sign up at Crawlbase to get free Normal and JavaScript tokens. Configure the MCP server in a client like Claude Desktop, Claude Code, Cursor IDE, or Windsurf IDE by adding its JSON configuration with your tokens. Then use commands like crawl, crawl_markdown, or crawl_screenshot to fetch web data. An HTTP transport mode is also available for multi‑user or custom integrations.

Key features of Crawlbase Web MCP Server

Fetches live web data for AI agents
JavaScript rendering for modern web apps
Proxy rotation and anti‑bot evasion
Structured outputs: HTML, Markdown, screenshots
Async crawling with Crawlbase Cloud Storage
HTTP transport with per‑request token authentication

Use cases of Crawlbase Web MCP Server

Market research by pulling live competitor data and reports
E‑commerce monitoring for real‑time product, review, and price tracking
News and finance feeds to keep AI agents current with events
Autonomous AI agents that can act on fresh web data

よくある質問

What do I need to get started?

You need free tokens from Crawlbase (a Normal token for HTML and a JavaScript token for JS‑rendered pages). Then configure the MCP server in your AI client.

Which AI clients are supported?

Claude Desktop, Claude Code, Cursor IDE, and Windsurf IDE are explicitly supported via stdio transport. HTTP transport mode is also available for any HTTP‑capable client.

Does it handle JavaScript‑heavy sites?

Yes. The server uses Crawlbase’s JavaScript rendering capability, which requires a JavaScript token. It handles modern web apps and anti‑bot measures.

Can I store crawl results for later use?

Yes. Crawlbase MCP can push results to Crawlbase Cloud Storage. Your AI agent can later read, list, or delete stored pages across sessions.

How does authentication work in HTTP mode?

Authentication is via environment variables or per‑request headers (`X-Crawlbase-Token` and `X-Crawlbase-JS-Token`). Headers override environment variables, enabling multi‑tenant deployments.

「その他」の他のコンテンツ

The Living Bread

A free remote MCP server for The Living Bread. Bible passages read word for word from a stored public domain text with an evidence label, keyword search and cross references, real churches, gatherings and prayer groups w

DeepMark

DeepMark helps teachers deliver rapid, consistent marking with meaningful feedback for every student — in a fraction of the time. What once took a week, now takes one free period.