Fastcrw MCP Server

@us

代码仓库主页

基本信息

分类

浏览器自动化

许可证

AGPL-3.0

传输方式

stdio

作者

提交者

Recep Saritekin

关于 Fastcrw MCP Server

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary.

#web-scraping #crawling #data-extraction #ai-agents #rag #llm #markdown #search #self-hosted#firecrawl-alternative#browser-automation #structured-data #mcp-server #claude#curso

Fastcrw MCP Server 配置

使用下面的配置,将此服务器添加到你的 MCP 客户端。

{
  "mcpServers": {
    "crw": {
      "command": "npx",
      "args": [
        "crw-mcp"
      ],
      "env": {
        "CRW_API_KEY": "<YOUR_API_KEY>"
      }
    }
  }
}

Fastcrw MCP Server 工具

Scrape one URL to markdown, HTML, or links.

Start an async site crawl; returns a job id to poll with crw_check_crawl_status.

Poll an async crawl job and retrieve its pages.

Discover URLs on a site via sitemap and/or a short crawl. Returns a URL list only, no page content.

Extract structured JSON from URLs via a prompt and/or JSON schema. Async job — poll crw_check_extract_status with the returned id. Needs an LLM.

Poll an extract job; returns status and, when complete, a per-URL results array.

Request cancellation of an extract job. Returns the canonical status; cancelling remains non-terminal until the claimed URL settles.

Parse a local PDF (base64 in contentBase64) to markdown. No OCR: scanned PDFs return empty markdown with a warning.

概览

What is Fastcrw?

Fastcrw is a self-hosted, Rust-native web crawler and scraper designed for AI agents. It serves as an open-source alternative to Firecrawl, offering a single static binary with a native REST API under /v1/* (scrape, crawl, map, search, structured extraction, and change tracking) plus a /firecrawl/v2/* compatibility layer for migrating from Firecrawl. It can be self-hosted for free under AGPL-3.0 or used via the managed API at api.fastcrw.com.

How to use Fastcrw?

Use Fastcrw by running the self-hosted binary (e.g., docker run -p 3000:3000 ghcr.io/us/crw) and sending requests to its REST endpoints (/v1/scrape, /v1/crawl, etc.) with an API key. Alternatively, install the built-in MCP server via npm install -g crw-mcp for use with MCP-compatible AI agents like Claude Code, Cursor, or Windsurf.

Key features of Fastcrw

Single static binary (~8 MB), no Redis, Node.js, or Python required.
~50 MB RAM idle, runs on a $5 VPS.
Native /v1/* API and Firecrawl v2 compatibility layer.
Change tracking with markdown git-diff and optional LLM judge.
AGPL-3.0 open core with managed commercial option.
Built-in MCP server and SDKs (npm, PyPI, Homebrew, Cargo).

Use cases of Fastcrw

AI agents scraping web content (markdown, HTML, JSON) programmatically.
Crawling multi-page documentation sites for knowledge base ingestion.
Mapping website link structures for sitemap generation or audit.
Structured JSON extraction from web pages using a JSON Schema.
Monitoring page changes with diff snapshots for alerting or tracking.

常见问题

How does Fastcrw compare to Firecrawl?

Fastcrw is a Rust-native alternative that runs as a single static binary (~8 MB, ~50 MB RAM idle) compared to Firecrawl’s multi-container setup. It offers a `/firecrawl/v2/*` compatibility layer for easy migration and natively supports change tracking and MCP server integration.

What are the runtime dependencies?

None beyond the binary itself. There is no need for Redis, Node.js, Python, or a headless browser alongside the request path. Self-hosted search requires a SearXNG sidecar (via Docker Compose) if used.

Where does my data live?

Data stays entirely on your infrastructure when self-hosting (the binary runs locally). If using the managed API at `api.fastcrw.com`, data passes through their proxy network and dashboard.

What transports and authentication does Fastcrw support?

Transport is HTTP REST (TCP) with Bearer token authentication via `Authorization: Bearer <key>` header. The built-in MCP server uses the MCP protocol (stdio or SSE). Embedded mode runs in-process with no external transport.

Is there a usage limit for the self-hosted version?

The README does not specify hard limits. The memory baseline is ~50 MB idle, scaling with crawl queue size. The managed API offers 500 free credits on sign-up.

浏览器自动化分类下的更多 MCP 服务器

Apify Model Context Protocol (MCP) Server

apify

The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automation tools available on the Apify Store.

1.7K