Fastcrw

What is Fastcrw?

Fastcrw is a self-hosted, Rust-native web crawler and scraper designed for AI agents. It serves as an open-source alternative to Firecrawl, offering a single static binary with a native REST API under /v1/* (scrape, crawl, map, search, structured extraction, and change tracking) plus a /firecrawl/v2/* compatibility layer for migrating from Firecrawl. It can be self-hosted for free under AGPL-3.0 or used via the managed API at api.fastcrw.com.

How to use Fastcrw?

Use Fastcrw by running the self-hosted binary (e.g., docker run -p 3000:3000 ghcr.io/us/crw) and sending requests to its REST endpoints (/v1/scrape, /v1/crawl, etc.) with an API key. Alternatively, install the built-in MCP server via npm install -g crw-mcp for use with MCP-compatible AI agents like Claude Code, Cursor, or Windsurf.

Key features of Fastcrw

• Single static binary (~8 MB), no Redis, Node.js, or Python required.
• ~50 MB RAM idle, runs on a $5 VPS.
• Native /v1/* API and Firecrawl v2 compatibility layer.
• Change tracking with markdown git-diff and optional LLM judge.
• AGPL-3.0 open core with managed commercial option.
• Built-in MCP server and SDKs (npm, PyPI, Homebrew, Cargo).

Use cases of Fastcrw

• AI agents scraping web content (markdown, HTML, JSON) programmatically.
• Crawling multi-page documentation sites for knowledge base ingestion.
• Mapping website link structures for sitemap generation or audit.
• Structured JSON extraction from web pages using a JSON Schema.
• Monitoring page changes with diff snapshots for alerting or tracking.

FAQ from Fastcrw

How does Fastcrw compare to Firecrawl?

Fastcrw is a Rust-native alternative that runs as a single static binary (~8 MB, ~50 MB RAM idle) compared to Firecrawl’s multi-container setup. It offers a /firecrawl/v2/* compatibility layer for easy migration and natively supports change tracking and MCP server integration.

What are the runtime dependencies?

None beyond the binary itself. There is no need for Redis, Node.js, Python, or a headless browser alongside the request path. Self-hosted search requires a SearXNG sidecar (via Docker Compose) if used.

Where does my data live?

Data stays entirely on your infrastructure when self-hosting (the binary runs locally). If using the managed API at api.fastcrw.com, data passes through their proxy network and dashboard.

What transports and authentication does Fastcrw support?

Transport is HTTP REST (TCP) with Bearer token authentication via Authorization: Bearer <key> header. The built-in MCP server uses the MCP protocol (stdio or SSE). Embedded mode runs in-process with no external transport.

Is there a usage limit for the self-hosted version?

The README does not specify hard limits. The memory baseline is ~50 MB idle, scaling with crawl queue size. The managed API offers 500 free credits on sign-up.