Anycrawl - Turn Any Website Into Llm Ready
@any4ai
About Anycrawl - Turn Any Website Into Llm Ready
AnyCrawl MCP Server, with Scrape, Crawl and SERP.
Basic information
Config
Add this server to your MCP-compatible client using the configuration below.
{
"mcpServers": {
"anycrawl-mcp": {
"command": "npx",
"args": [
"-y",
"anycrawl-mcp-server"
],
"env": {
"ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
"ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
"LOG_LEVEL": "info"
}
}
}
}Tools
6🚀 Scrape a single URL and extract content in various formats. AnyCrawl turns websites into LLM-ready structured data with high performance multi-threading. Best for: Extracting content from a known, single page (article, docs page, product page). Not recommended for: Broad discovery across many pages (use anycrawl_crawl); open-ended questions across the web (use anycrawl_search). Common mistakes: Using a headless engine unnecessarily (prefer cheerio for static pages); requesting heavy formats (screenshots/rawHtml) when not needed; setting large timeouts without cause. Prompt example: "Scrape this page and return clean markdown: https://example.com/blog/post" Engine guidance: Use cheerio for static HTML, playwright for dynamic apps, puppeteer for Chrome automation. Usage example without formats: { "name": "anycrawl_scrape", "arguments": { "url": "https://news.ycombinator.com", "engine": "cheerio" } } Usage example with formats: { "name": "anycrawl_scrape", "arguments": { "url": "https://example.com/docs/page", "engine": "playwright", "formats": ["markdown"], "wait_for": 1500 } } Curl examples (JSON extraction via json_options): 1) Using user_prompt for ad-hoc extraction curl -X POST https://api.anycrawl.dev/v1/scrape -H 'Content-Type: application/json' -H 'Authorization: Bearer YOUR_ANYCRAWL_API_KEY' -d '{ "url": "https://example.com/", "engine": "playwright", "formats": [ "markdown", "json" ], "json_options": { "user_prompt": "Extract the page title and the main paragraph content as plain text." } }' 2) Using a JSON Schema with user_prompt for structured extraction curl -X POST https://api.anycrawl.dev/v1/scrape -H 'Content-Type: application/json' -H 'Authorization: Bearer YOUR_ANYCRAWL_API_KEY' -d '{ "url": "https://example.com/", "engine": "playwright", "formats": [ "markdown", "json" ], "json_options": { "schema": { "type": "object", "properties": { "title": { "type": "string", "description": "title of web" } }, "required": [ "title" ] }, "user_prompt": "Extract the page title and the main paragraph content as plain text." } }' Returns: { url, status, jobId, title, html?, markdown?, metadata?, timestamp }
🌐 Start an asynchronous crawl job to scrape multiple pages from a website. Perfect for comprehensive site analysis, content aggregation, and bulk data collection with native multi-threading. Best for: Multi-page coverage of a site or section (docs, blogs, categories). Not recommended for: A single known page (use anycrawl_scrape); open-ended web-wide queries (use anycrawl_search). Common mistakes: Setting limit too high; using strategy="all" unintentionally; requesting heavy formats (e.g., full-page screenshots) across many pages; deep max_depth without need. Prompt example: "Crawl the docs section and return markdown for up to 100 pages." Strategy guidance: - same-domain (default) is safest; - same-hostname for subdomain specificity; - same-origin for strict protocol+domain; - all for external links (use cautiously). Usage example (basic): { "name": "anycrawl_crawl", "arguments": { "url": "https://docs.example.com/*", "engine": "cheerio", "limit": 100, "max_depth": 5 } } Usage example (with formats and filters): { "name": "anycrawl_crawl", "arguments": { "url": "https://example.com/blog/*", "engine": "cheerio", "formats": ["markdown"], "exclude_paths": ["/tags/*", "*.pdf"], "include_tags": ["article", "main"], "limit": 50 } } Curl examples (JSON extraction across pages via json_options): 1) Create a crawl with user_prompt curl -X POST https://api.anycrawl.dev/v1/crawl -H 'Content-Type: application/json' -H 'Authorization: Bearer YOUR_ANYCRAWL_API_KEY' -d '{ "url": "https://example.com/blog/*", "engine": "cheerio", "limit": 50, "formats": ["markdown", "json"], "json_options": { "user_prompt": "Extract the article title and author for each page." } }' 2) Create a crawl with a JSON Schema curl -X POST https://api.anycrawl.dev/v1/crawl -H 'Content-Type: application/json' -H 'Authorization: Bearer YOUR_ANYCRAWL_API_KEY' -d '{ "url": "https://example.com/docs/*", "engine": "cheerio", "max_depth": 5, "formats": ["markdown", "json"], "json_options": { "schema": { "type": "object", "properties": { "title": { "type": "string" }, "slug": { "type": "string" } }, "required": ["title"] }, "user_prompt": "Extract the page title and compute a slug." } }' Returns: Job creation info { job_id, status, message }. Use anycrawl_crawl_status and anycrawl_crawl_results to monitor and retrieve data.
📊 Check the status of an asynchronous crawl job. Monitor progress, view statistics, and track completion status. Best for: Ongoing monitoring of a crawl created with anycrawl_crawl. Not recommended for: Fetching page content (use anycrawl_crawl_results). Usage example: { "name": "anycrawl_crawl_status", "arguments": { "job_id": "7a2e165d-8f81-4be6-9ef7-23222330a396" } } Returns: { job_id, status, start_time, expires_at, credits_used, total, completed, failed }
📄 Get results from a completed or in-progress crawl job. Supports pagination for large crawls with thousands of pages. Best for: Retrieving crawled page data and metadata after or during a crawl. Common mistakes: Forgetting to paginate via skip when next is present; requesting extremely large pages of data. Usage example: { "name": "anycrawl_crawl_results", "arguments": { "job_id": "7a2e165d-8f81-4be6-9ef7-23222330a396", "skip": 0 } } Returns: { status, total, completed, creditsUsed, next?, data: Array<pageResult> } where next can be used as the next skip value for pagination.
🛑 Cancel a pending or running crawl job. Useful for stopping long-running crawls or correcting configuration mistakes. Best for: Stopping crawls that are no longer needed or misconfigured. Not recommended for: Completed jobs (cancellation has no effect). Usage example: { "name": "anycrawl_cancel_crawl", "arguments": { "job_id": "7a2e165d-8f81-4be6-9ef7-23222330a396" } } Returns: Confirmation { job_id, status } indicating cancellation state.
🔍 Search the web using AnyCrawl's powerful search engine integration. Get SERP (Search Engine Results Page) data with optional content scraping for comprehensive research. Best for: Finding specific information across multiple websites when you don't know which site has it; retrieving the most relevant content for an open-ended query. Not recommended for: Searching the filesystem; when you already know the exact website to extract (use anycrawl_scrape); when you need comprehensive coverage of a single site (use anycrawl_crawl). Common mistakes: Using crawl for open-ended questions; requesting heavy scrape_options (large formats/timeouts) unnecessarily. Prompt example: "Find the latest research papers on AI published in 2023." Sources: web (default). Image/news verticals are not yet supported in this tool. Scrape options: Only set scrape_options when absolutely necessary. Prefer small limits (≤5) and minimal formats (e.g., ["markdown"]) to avoid timeouts. Usage example without formats: { "name": "anycrawl_search", "arguments": { "query": "top AI companies", "limit": 5, "scrape_options": { "engine": "cheerio" } } } Usage example with formats: { "name": "anycrawl_search", "arguments": { "query": "latest AI research papers 2023", "limit": 5, "lang": "en", "country": "US", "scrape_options": { "engine": "cheerio", "formats": ["markdown"], "wait_for": 1000 } } } Curl examples (with scrape_options and optional JSON extraction): 1) Basic search with scraping each result page curl -X POST https://api.anycrawl.dev/v1/search -H 'Content-Type: application/json' -H 'Authorization: Bearer YOUR_ANYCRAWL_API_KEY' -d '{ "query": "python web scraping tutorial", "limit": 5, "scrape_options": { "engine": "cheerio", "formats": ["markdown"] } }' 2) Search with JSON Schema extraction from result pages curl -X POST https://api.anycrawl.dev/v1/search -H 'Content-Type: application/json' -H 'Authorization: Bearer YOUR_ANYCRAWL_API_KEY' -d '{ "query": "best restaurants in New York", "limit": 5, "lang": "en", "country": "US", "scrape_options": { "engine": "cheerio", "formats": ["markdown", "json"], "json_options": { "schema": { "type": "object", "properties": { "name": { "type": "string" }, "rating": { "type": "number" } }, "required": ["name"] }, "user_prompt": "Extract the restaurant name and rating if available." } } }' Returns: Array of search results with optional scraped content for each result URL.
Overview
What is Anycrawl?
Anycrawl is an MCP server that turns any website into LLM-ready data through web scraping and crawling. It integrates with Cursor, Claude, and other MCP clients to extract content from single URLs or entire websites, supporting multiple output formats and scraping engines. It is designed for developers and AI users who need structured web data for LLM consumption.
How to use Anycrawl?
Install and run via npx with your API key: ANYCRAWL_API_KEY=YOUR-API-KEY npx -y anycrawl-mcp. For cloud usage, set the API key only; for self-hosted, additionally set ANYCRAWL_BASE_URL. The server runs in STDIO mode by default; switch to MCP (HTTP) or SSE mode by setting ANYCRAWL_MODE. Then configure your MCP client (Cursor, Claude Desktop, etc.) to connect to Anycrawl using the provided tool definitions.
Key features of Anycrawl
- Scrape single URLs with multiple output formats
- Crawl entire websites with configurable depth and limits
- Search engine integration for web search and scraping
- Multiple engines: auto, Playwright, Cheerio, Puppeteer
- Flexible output: Markdown, HTML, text, screenshots, JSON
- Async operations with status monitoring for crawl jobs
Use cases of Anycrawl
- Extracting structured content from a single page for LLM ingestion
- Collecting data from multiple related pages (e.g., blog posts, documentation)
- Gathering web content to build retrieval-augmented generation (RAG) datasets
- Monitoring website changes via periodic scraping
- Integrating web data into AI workflows in tools like Cursor or Claude
FAQ from Anycrawl
How do I get an API key?
Visit AnyCrawl, sign up or log in, then go to the Dashboard → API Keys → Copy your key. Free sign-up gives 1,500 credits.
Can I use Anycrawl without hosting my own server?
Yes. The cloud service at mcp.anycrawl.dev is recommended — no server setup required. Just set your API key and use the provided endpoints.
What scraping engines are available?
Four engines: auto (intelligent selection), playwright, cheerio, and puppeteer. You can specify one per scrape or crawl.
What output formats does Anycrawl support?
Markdown, HTML, text, screenshot (full-page optional), raw HTML, and structured JSON. Multiple formats can be requested in a single request.
What transport modes does Anycrawl support?
STDIO (default), MCP over HTTP (streamable HTTP), and SSE (Server-Sent Events). Mode is selected via the ANYCRAWL_MODE environment variable.
More AI & Agents MCP servers
21st.dev Magic AI Agent
21st-devIt's like v0 but in your Cursor/WindSurf/Cline. 21st dev Magic MCP server for working with your frontend like Magic
Open Multi-Agent Canvas
CopilotKitThe open-source multi-agent chat interface that lets you manage multiple agents in one dynamic conversation and add MCP servers for deep research
MCP-LLM Bridge
patruffBridge between Ollama and MCP servers, enabling local LLMs to use Model Context Protocol tools
MCP Claude Code
SDGLBLMCP implementation of Claude Code capabilities and more
Perplexity Ask MCP Server
ppl-aiThe official MCP server implementation for the Perplexity API Platform
Comments