olostep-mcp
@olostep
MCP server for Olostep — the web scraping, crawling, and search infrastructure used by top AI companies. Gives any MCP-compatible AI agent the ability to scrape, crawl, batch-extract, and search the web in real time.
Overview
What is olostep-mcp?
A Model Context Protocol (MCP) server that integrates with Olostep for web scraping, content extraction, and web search. It requires an Olostep API key (obtainable at olostep.com/auth). Designed for developers and AI agents needing structured data from websites.
How to use olostep-mcp?
Install via npx (npx -y olostep-mcp), Docker (docker pull olostep/mcp-server), or connect to the remote endpoint (https://mcp.olostep.com/mcp). Set the OLOSTEP_API_KEY environment variable. Configure in your MCP client (Cursor, Claude Desktop, Windsurf, VS Code, Metorial) using the provided JSON examples.
Key features of olostep-mcp
- Scrape websites in HTML, Markdown, JSON, or Plain Text
- Parser-based web search with structured results
- AI Answers with citations and optional JSON-shaped outputs
- Batch scraping of up to 10,000 URLs
- Autonomous site crawling from a start URL
- Website URL discovery with include/exclude filters
- Country-specific request routing for geo-targeted content
- Configurable wait times for JavaScript-heavy websites
Use cases of olostep-mcp
- Extract clean markdown from a single documentation page
- Search the web and return structured data (e.g., product listings)
- Get AI-powered answers with cited sources for research
- Batch scrape thousands of URLs for large-scale data collection
- Crawl an entire website to map its structure and content
FAQ from olostep-mcp
How do I get an Olostep API key?
Sign up at olostep.com/auth to obtain your API key.
What output formats are supported?
Scraped content can be returned as HTML, Markdown, JSON, or Plain Text (configurable per request).
Can I scrape multiple URLs at once?
Yes, use the batch_scrape_urls tool to scrape up to 10,000 URLs in a single job. Results are retrieved later via get_batch_results.
How does website crawling work?
Crawling is asynchronous: call create_crawl to start a crawl and get a crawl_id, then poll get_crawl_results to retrieve the scraped pages.
What authentication method is used?
Authentication uses a Bearer token in the Authorization header (for the remote endpoint) or the OLOSTEP_API_KEY environment variable (for local/docker setups).
