Crawlbase Web MCP Server

@crawlbase

Basic information

About Crawlbase Web MCP Server

Crawlbase MCP Server connects AI agents and LLMs with real-time web data. It powers Claude, Cursor, and Windsurf integrations with battle-tested web scraping, JavaScript rendering, and anti-bot protection enabling structured, live data inside your AI workflows.

#web-scraping#web-scraping-mcp#web-crawling-mcp#web-mcp-server#web-data-mcp#web-data #ai #agents

Crawlbase Web MCP Server config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "@crawlbase/mcp@latest"
      ],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

Crawlbase Web MCP Server tools

Crawl a URL and return HTML content

Crawl a URL and extract clean markdown content

Take a screenshot of a webpage

Overview

What is Crawlbase Web MCP Server?

Crawlbase Web MCP Server is a Model Context Protocol (MCP) server that bridges AI agents with live web content. It uses Crawlbase’s crawling infrastructure, trusted by over 70,000 developers, to fetch fresh, structured, real-time data from the web. It handles JavaScript rendering, proxy rotation, anti-bot evasion, and returns HTML, Markdown, or screenshots.

How to use Crawlbase Web MCP Server?

Sign up at Crawlbase to get free Normal and JavaScript tokens. Configure the MCP server in a client like Claude Desktop, Claude Code, Cursor IDE, or Windsurf IDE by adding its JSON configuration with your tokens. Then use commands like crawl, crawl_markdown, or crawl_screenshot to fetch web data. An HTTP transport mode is also available for multi‑user or custom integrations.

Key features of Crawlbase Web MCP Server

Fetches live web data for AI agents
JavaScript rendering for modern web apps
Proxy rotation and anti‑bot evasion
Structured outputs: HTML, Markdown, screenshots
Async crawling with Crawlbase Cloud Storage
HTTP transport with per‑request token authentication

Use cases of Crawlbase Web MCP Server

Market research by pulling live competitor data and reports
E‑commerce monitoring for real‑time product, review, and price tracking
News and finance feeds to keep AI agents current with events
Autonomous AI agents that can act on fresh web data

Frequently asked questions

What do I need to get started?

You need free tokens from Crawlbase (a Normal token for HTML and a JavaScript token for JS‑rendered pages). Then configure the MCP server in your AI client.

Which AI clients are supported?

Claude Desktop, Claude Code, Cursor IDE, and Windsurf IDE are explicitly supported via stdio transport. HTTP transport mode is also available for any HTTP‑capable client.

Does it handle JavaScript‑heavy sites?

Yes. The server uses Crawlbase’s JavaScript rendering capability, which requires a JavaScript token. It handles modern web apps and anti‑bot measures.

Can I store crawl results for later use?

Yes. Crawlbase MCP can push results to Crawlbase Cloud Storage. Your AI agent can later read, list, or delete stored pages across sessions.

How does authentication work in HTTP mode?

Authentication is via environment variables or per‑request headers (`X-Crawlbase-Token` and `X-Crawlbase-JS-Token`). Headers override environment variables, enabling multi‑tenant deployments.

Comments

More Other MCP servers

Awesome Mlops

visenger

A curated list of references for MLOps

14.0K

Reactive Resume

amruthpillai

A one-of-a-kind resume builder that keeps your privacy in mind. Completely secure, customizable, portable, open-source and free forever. Try it out today!

39.5K

Blender

ahujasid

Open-source MCP to use Blender with any LLM

23.6K

Codelf

unbug

A search tool helps dev to solve the naming things problem.

14.2K

DeepMark

DeepMark helps teachers deliver rapid, consistent marking with meaningful feedback for every student — in a fraction of the time. What once took a week, now takes one free period.