Fastcrw MCP Server

@us

リポジトリホームページ

基本情報

カテゴリ

ブラウザ自動化

ライセンス

AGPL-3.0

トランスポート

stdio

作者

投稿者

Recep Saritekin

Fastcrw MCP Server について

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary.

#web-scraping #crawling #data-extraction #ai-agents #rag #llm #markdown #search #self-hosted#firecrawl-alternative#browser-automation #structured-data #mcp-server #claude#curso

Fastcrw MCP Server の設定

以下の設定を使って、このサーバーを MCP 対応クライアントに追加してください。

{
  "mcpServers": {
    "crw": {
      "command": "npx",
      "args": [
        "crw-mcp"
      ],
      "env": {
        "CRW_API_KEY": "<YOUR_API_KEY>"
      }
    }
  }
}

Fastcrw MCP Server のツール

Scrape one URL to markdown, HTML, or links.

Start an async site crawl; returns a job id to poll with crw_check_crawl_status.

Poll an async crawl job and retrieve its pages.

Discover URLs on a site via sitemap and/or a short crawl. Returns a URL list only, no page content.

Extract structured JSON from URLs via a prompt and/or JSON schema. Async job — poll crw_check_extract_status with the returned id. Needs an LLM.

Poll an extract job; returns status and, when complete, a per-URL results array.

Request cancellation of an extract job. Returns the canonical status; cancelling remains non-terminal until the claimed URL settles.

Parse a local PDF (base64 in contentBase64) to markdown. No OCR: scanned PDFs return empty markdown with a warning.

概要

What is Fastcrw?

Fastcrw is a self-hosted, Rust-native web crawler and scraper designed for AI agents. It serves as an open-source alternative to Firecrawl, offering a single static binary with a native REST API under /v1/* (scrape, crawl, map, search, structured extraction, and change tracking) plus a /firecrawl/v2/* compatibility layer for migrating from Firecrawl. It can be self-hosted for free under AGPL-3.0 or used via the managed API at api.fastcrw.com.

How to use Fastcrw?

Use Fastcrw by running the self-hosted binary (e.g., docker run -p 3000:3000 ghcr.io/us/crw) and sending requests to its REST endpoints (/v1/scrape, /v1/crawl, etc.) with an API key. Alternatively, install the built-in MCP server via npm install -g crw-mcp for use with MCP-compatible AI agents like Claude Code, Cursor, or Windsurf.

Key features of Fastcrw

Single static binary (~8 MB), no Redis, Node.js, or Python required.
~50 MB RAM idle, runs on a $5 VPS.
Native /v1/* API and Firecrawl v2 compatibility layer.
Change tracking with markdown git-diff and optional LLM judge.
AGPL-3.0 open core with managed commercial option.
Built-in MCP server and SDKs (npm, PyPI, Homebrew, Cargo).

Use cases of Fastcrw

AI agents scraping web content (markdown, HTML, JSON) programmatically.
Crawling multi-page documentation sites for knowledge base ingestion.
Mapping website link structures for sitemap generation or audit.
Structured JSON extraction from web pages using a JSON Schema.
Monitoring page changes with diff snapshots for alerting or tracking.

よくある質問

How does Fastcrw compare to Firecrawl?

Fastcrw is a Rust-native alternative that runs as a single static binary (~8 MB, ~50 MB RAM idle) compared to Firecrawl’s multi-container setup. It offers a `/firecrawl/v2/*` compatibility layer for easy migration and natively supports change tracking and MCP server integration.

What are the runtime dependencies?

None beyond the binary itself. There is no need for Redis, Node.js, Python, or a headless browser alongside the request path. Self-hosted search requires a SearXNG sidecar (via Docker Compose) if used.

Where does my data live?

Data stays entirely on your infrastructure when self-hosting (the binary runs locally). If using the managed API at `api.fastcrw.com`, data passes through their proxy network and dashboard.

What transports and authentication does Fastcrw support?

Transport is HTTP REST (TCP) with Bearer token authentication via `Authorization: Bearer <key>` header. The built-in MCP server uses the MCP protocol (stdio or SSE). Embedded mode runs in-process with no external transport.

Is there a usage limit for the self-hosted version?

The README does not specify hard limits. The memory baseline is ~50 MB idle, scaling with crawl queue size. The managed API offers 500 free credits on sign-up.

「ブラウザ自動化」の他のコンテンツ

Fetch

modelcontextprotocol

Model Context Protocol Servers

88.1K

Loadster

loadster

Loadster is a load testing, stress testing, and synthetic monitoring platform. Run Playwright, Browser Bots, and Protocol Bots to test and monitor your websites, web apps, and APIs. Loadster MCP gives your agents the abi