Overview
What is WebScraping.AI MCP Server?
WebScraping.AI MCP Server is a Model Context Protocol (MCP) server that integrates with the WebScraping.AI API to provide web data extraction capabilities to MCP-enabled LLMs. It allows AI agents to scrape web pages, answer questions about content, extract structured data, and retrieve HTML or text with JavaScript rendering and proxy support.
How to use WebScraping.AI MCP Server?
Install and run it via npx with your API key: env WEBSCRAPING_AI_API_KEY=your_api_key npx -y webscraping-ai-mcp. Configure it in Cursor (.cursor/mcp.json) or Claude Desktop (claude_desktop_config.json) by specifying the command and environment variables. All scraping tools accept parameters like url, timeout, js, proxy, country, device, and wait_for.
Key features of WebScraping.AI MCP Server
- Question answering about web page content
- Structured data extraction with custom fields
- Full HTML and plain text retrieval
- CSS selector–based content extraction
- Support for datacenter and residential proxies with country selection
- JavaScript rendering using headless Chrome/Chromium
Use cases of WebScraping.AI MCP Server
- Ask an LLM to summarize or find specific information on a webpage
- Extract product details (title, price, description) from e-commerce sites
- Retrieve the full HTML of a page for further analysis or archiving
- Monitor account usage (API requests remaining) via the account tool
- Scrape content from multiple page sections using multiple CSS selectors
FAQ from WebScraping.AI MCP Server
What is the difference between WebScraping.AI MCP Server and direct web scraping?
This server wraps WebScraping.AI’s API into the MCP format, making it accessible to any MCP-compatible LLM. It handles rate limiting, retries, proxy selection, and JavaScript rendering automatically, so the LLM can focus on extraction tasks.
What are the runtime dependencies?
You need Node.js and an active WebScraping.AI API key. The server runs via npx or can be installed from npm; no other external services are required.
Where does scraped data live?
All web page data is fetched through the WebScraping.AI API. The server does not store data locally; responses are returned directly to the calling LLM.
Are there any rate limits or concurrency restrictions?
Yes. The default concurrency limit is 5 requests (configurable via WEBSCRAPING_AI_CONCURRENCY_LIMIT). The server includes automatic backoff for rate-limit errors (HTTP 429). Timeouts are capped at 30 s for page retrieval and 2 s for JavaScript rendering (configurable).
What transport and authentication does the server use?
The server communicates over stdio (standard input/output) using the MCP protocol. Authentication is via an API key passed as the environment variable WEBSCRAPING_AI_API_KEY.
