Anycrawl - Turn Any Website Into Llm Ready MCP Server

@any4ai

Repository

Basic information

About Anycrawl - Turn Any Website Into Llm Ready MCP Server

AnyCrawl MCP Server, with Scrape, Crawl and SERP.

#anycrawl#web-crawler#ai-scrape

Anycrawl - Turn Any Website Into Llm Ready MCP Server config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "anycrawl-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "anycrawl-mcp-server"
      ],
      "env": {
        "ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
        "ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
        "LOG_LEVEL": "info"
      }
    }
  }
}

Anycrawl - Turn Any Website Into Llm Ready MCP Server tools

Scrape a single URL and extract content in selected formats. Best for: One known page (articles, docs, product pages). Not recommended for: Multi-page coverage (use anycrawl_crawl) or open-ended discovery (use anycrawl_search). RECOMMENDED: Use 'playwright' engine for best results with dynamic content and modern websites. Usage (parameters): - url: HTTP/HTTPS URL to scrape (string, required) - engine: 'playwright' | 'cheerio' | 'puppeteer' (required, default: 'playwright') - proxy: Proxy URL (string, optional) - formats: Output formats ['markdown'|'html'|'text'|'screenshot'|'screenshot@fullPage'|'rawHtml'|'json'] (optional) - timeout: Request timeout in ms (number, optional) - retry: Enable auto-retry on failure (boolean, optional) - wait_for: Wait in ms for dynamic pages (number, optional) - include_tags: HTML tags to include (string[], optional) - exclude_tags: HTML tags to exclude (string[], optional) - json_options: { schema?, user_prompt?, schema_name?, schema_description? } (optional) - extract_source: 'html' | 'markdown' (optional) Returns: { url, status, jobId?, title?, html?, markdown?, metadata?, timestamp? } Examples: - Recommended: { "url": "https://example.com", "engine": "playwright" } - With JSON extraction: { "url": "https://news.ycombinator.com", "engine": "playwright", "formats": ["markdown"], "json_options": { "user_prompt": "Extract titles", "schema_name": "Articles" } } - With JSON schema extraction: { "url": "https://example.com/article", "engine": "playwright", "json_options": { "schema_name": "Article", "schema_description": "Extract article metadata and content", "schema": { "type": "object", "properties": { "title": { "type": "string" }, "author": { "type": "string" }, "date": { "type": "string" }, "content": { "type": "string" } }, "required": ["title", "content"] } } }

Crawl an entire website with configurable depth and limits. Best for: Multi-page coverage, site mapping, content discovery. Not recommended for: Single pages (use anycrawl_scrape) or open-ended discovery (use anycrawl_search). RECOMMENDED: Use 'playwright' engine for best results with dynamic content and modern websites. Usage (parameters): - url: Starting URL to crawl (string, required) - engine: 'playwright' | 'cheerio' | 'puppeteer' (required, default: 'playwright') - max_depth: Maximum crawl depth (number, optional, default: 10) - limit: Maximum pages to crawl (number, optional, default: 100) - strategy: Crawl strategy 'all' | 'same-domain' | 'same-hostname' | 'same-origin' (optional, default: 'same-domain') - include_paths: Path patterns to include (string[], optional) - exclude_paths: Path patterns to exclude (string[], optional) - retry: Enable auto-retry on failure (boolean, optional) - poll_seconds: Polling interval for job status (number, optional) - poll_interval_ms: Polling interval in milliseconds (number, optional) - timeout_ms: Job timeout in milliseconds (number, optional) - scrape_options: Nested scrape options for each page (object, optional) Returns: { job_id, status, message } for async jobs Examples: - Recommended: { "url": "https://example.com", "engine": "playwright", "limit": 50 } - Deep crawl: { "url": "https://docs.example.com", "engine": "playwright", "max_depth": 5, "limit": 200 } - Filtered crawl: { "url": "https://blog.example.com", "engine": "playwright", "include_paths": ["/posts/*"], "exclude_paths": ["/admin/*"] }

Search the web and optionally scrape results. Best for: Open-ended discovery, finding relevant content. Not recommended for: Known URLs (use anycrawl_scrape) or comprehensive site coverage (use anycrawl_crawl). RECOMMENDED: Use limit=5 for balanced performance and cost. Use 'playwright' engine for scraping results. Usage (parameters): - query: Search query string (string, required) - engine: Search engine 'google' (optional, default: 'google') - limit: Number of results to return (number, optional, default: 5) - offset: Number of results to skip (number, optional, default: 0) - pages: Number of search result pages to process (number, optional) - lang: Language code (string, optional) - country: Country code (string, optional) - safeSearch: Safe search level 0-2 (number, optional) - scrape_options: Options for scraping search results (object, optional) Returns: Array of search results with optional scraped content Examples: - Recommended: { "query": "artificial intelligence news", "limit": 5 } - With scraping: { "query": "TypeScript tutorials", "limit": 5, "scrape_options": { "formats": ["markdown"], "engine": "playwright" } } - Localized search: { "query": "machine learning", "lang": "es", "country": "ES", "limit": 5 }

Get the status of a crawl job. Check progress, completion status, and statistics for an ongoing or completed crawl job. Usage (parameters): - job_id: The crawl job ID (string, required) Returns: { job_id, status, start_time, expires_at, credits_used, total, completed, failed } Examples: - Check status: { "job_id": "crawl_12345" }

Get the results of a completed crawl job. Retrieve the scraped content and metadata from a completed crawl job. Usage (parameters): - job_id: The crawl job ID (string, required) - skip: Number of results to skip (number, optional, default: 0) Returns: { status, total, completed, creditsUsed, next?, data[] } Examples: - Get all results: { "job_id": "crawl_12345" } - Paginated results: { "job_id": "crawl_12345", "skip": 50 }

Cancel a running crawl job. Stop an ongoing crawl job and prevent further processing. Usage (parameters): - job_id: The crawl job ID to cancel (string, required) Returns: { success: boolean, message: string } Examples: - Cancel job: { "job_id": "crawl_12345" }

Overview

What is Anycrawl?

Anycrawl is an MCP server that turns any website into LLM-ready data through web scraping and crawling. It integrates with Cursor, Claude, and other MCP clients to extract content from single URLs or entire websites, supporting multiple output formats and scraping engines. It is designed for developers and AI users who need structured web data for LLM consumption.

How to use Anycrawl?

Install and run via npx with your API key: ANYCRAWL_API_KEY=YOUR-API-KEY npx -y anycrawl-mcp. For cloud usage, set the API key only; for self-hosted, additionally set ANYCRAWL_BASE_URL. The server runs in STDIO mode by default; switch to MCP (HTTP) or SSE mode by setting ANYCRAWL_MODE. Then configure your MCP client (Cursor, Claude Desktop, etc.) to connect to Anycrawl using the provided tool definitions.

Key features of Anycrawl

Scrape single URLs with multiple output formats
Crawl entire websites with configurable depth and limits
Search engine integration for web search and scraping
Multiple engines: auto, Playwright, Cheerio, Puppeteer
Flexible output: Markdown, HTML, text, screenshots, JSON
Async operations with status monitoring for crawl jobs

Use cases of Anycrawl

Extracting structured content from a single page for LLM ingestion
Collecting data from multiple related pages (e.g., blog posts, documentation)
Gathering web content to build retrieval-augmented generation (RAG) datasets
Monitoring website changes via periodic scraping
Integrating web data into AI workflows in tools like Cursor or Claude

Frequently asked questions

How do I get an API key?

Visit [AnyCrawl](https://anycrawl.dev), sign up or log in, then go to the Dashboard → API Keys → Copy your key. Free sign-up gives 1,500 credits.

Can I use Anycrawl without hosting my own server?

Yes. The cloud service at `mcp.anycrawl.dev` is recommended — no server setup required. Just set your API key and use the provided endpoints.

What scraping engines are available?

Four engines: `auto` (intelligent selection), `playwright`, `cheerio`, and `puppeteer`. You can specify one per scrape or crawl.

What output formats does Anycrawl support?

Markdown, HTML, text, screenshot (full-page optional), raw HTML, and structured JSON. Multiple formats can be requested in a single request.

What transport modes does Anycrawl support?

STDIO (default), MCP over HTTP (streamable HTTP), and SSE (Server-Sent Events). Mode is selected via the `ANYCRAWL_MODE` environment variable.

Comments

More AI & Agents MCP servers

Legion MCP

faulkj

MCP-native LLM councils for debates, juries, blind panels, voting, refinement, and custom multi-model deliberation.

Transform MCP

Unstructured

The Unstructured Transform MCP Server connects your ADK agent to Unstructured, a document processing platform that turns raw files into structured, AI-ready data. This integration gives your agent the ability to parse PD

AskOne: Live Q&A and Polls

phuryn

Run free live audience Q&A and polls from your agent: start an AskOne room, add and launch a poll, approve or hide questions and mark them answered, close the room, then read the questions as a FAQ draft and get the resu

Gemina

Gemina Tech Ltd.

Extract, search and tag any document — invoices, receipts, contracts, forms, or your own templates. 13 tools, one sign-in (OAuth 2.1) or API key, free FileTag tier, data residency in the EU, US, Israel or Asia.

Poly-Glot AI Workspace

hmoses

1,000+ fully localized prompt templates across 40+ categories. Compare Mode — send the same prompt to up to 9 AI providers (ChatGPT, Claude, Gemini, Grok, Groq, Copilot, Mistral, Perplexity, DuckDuckGo AI) and compare an