Anycrawl - Turn Any Website Into Llm Ready

What is Anycrawl?

Anycrawl is an MCP server that turns any website into LLM-ready data through web scraping and crawling. It integrates with Cursor, Claude, and other MCP clients to extract content from single URLs or entire websites, supporting multiple output formats and scraping engines. It is designed for developers and AI users who need structured web data for LLM consumption.

How to use Anycrawl?

Install and run via npx with your API key: ANYCRAWL_API_KEY=YOUR-API-KEY npx -y anycrawl-mcp. For cloud usage, set the API key only; for self-hosted, additionally set ANYCRAWL_BASE_URL. The server runs in STDIO mode by default; switch to MCP (HTTP) or SSE mode by setting ANYCRAWL_MODE. Then configure your MCP client (Cursor, Claude Desktop, etc.) to connect to Anycrawl using the provided tool definitions.

Key features of Anycrawl

• Scrape single URLs with multiple output formats
• Crawl entire websites with configurable depth and limits
• Search engine integration for web search and scraping
• Multiple engines: auto, Playwright, Cheerio, Puppeteer
• Flexible output: Markdown, HTML, text, screenshots, JSON
• Async operations with status monitoring for crawl jobs

Use cases of Anycrawl

• Extracting structured content from a single page for LLM ingestion
• Collecting data from multiple related pages (e.g., blog posts, documentation)
• Gathering web content to build retrieval-augmented generation (RAG) datasets
• Monitoring website changes via periodic scraping
• Integrating web data into AI workflows in tools like Cursor or Claude

FAQ from Anycrawl

How do I get an API key?

Visit AnyCrawl, sign up or log in, then go to the Dashboard → API Keys → Copy your key. Free sign-up gives 1,500 credits.

Can I use Anycrawl without hosting my own server?

Yes. The cloud service at mcp.anycrawl.dev is recommended — no server setup required. Just set your API key and use the provided endpoints.

What scraping engines are available?

Four engines: auto (intelligent selection), playwright, cheerio, and puppeteer. You can specify one per scrape or crawl.

What output formats does Anycrawl support?

Markdown, HTML, text, screenshot (full-page optional), raw HTML, and structured JSON. Multiple formats can be requested in a single request.

What transport modes does Anycrawl support?

STDIO (default), MCP over HTTP (streamable HTTP), and SSE (Server-Sent Events). Mode is selected via the ANYCRAWL_MODE environment variable.