How do I get an Olostep API key?
Sign up at [olostep.com/auth](https://olostep.com/auth) to obtain your API key.
Olostep MCP Server
@olostep
基本信息
关于 Olostep MCP Server
MCP server for Olostep — the web scraping, crawling, and search infrastructure used by top AI companies. Gives any MCP-compatible AI agent the ability to scrape, crawl, batch-extract, and search the web in real time.
Olostep MCP Server 配置
使用下面的配置,将此服务器添加到你的 MCP 客户端。
{
"mcpServers": {
"olostep-mcp": {
"command": "npx",
"args": [
"-y",
"olostep-mcp"
],
"env": {
"OLOSTEP_API_KEY": "YOUR_API_KEY_HERE"
}
}
}
}Olostep MCP Server 工具
10Get a LIST of URLs on a website (URL discovery only — does NOT scrape content). Use when the user wants a list of links: 'show me all URLs on this site', 'map this website', or when you want to surface candidate URLs to the user before scraping a subset. Prefer `create_crawl` if the goal is to scrape the whole site — it discovers AND scrapes in one workflow. Use this only when the URL list itself is the deliverable.
**PREFERRED tool for crawling a website.** Use this whenever the user says 'crawl', 'scrape the whole site', 'get all pages from a site', or wants multiple pages from a single domain. This is the CORRECT tool for any whole-site scraping task. **Do NOT use `batch_scrape_urls` for crawling** - that tool is only for when you already have a specific list of unrelated URLs from different domains. Starts an ASYNC crawl that autonomously discovers and scrapes pages by following links from a start URL. Returns a crawl_id - the crawl runs in the background. You MUST then call `get_crawl_results` with the returned crawl_id to poll status and retrieve the scraped pages. Do NOT call `get_batch_results` with a crawl_id - crawls and batches are separate resources.
Retrieve the status and scraped pages for a crawl job. Pass the crawl_id returned by create_crawl. If the crawl is still in_progress, returns the current status so you can call again later (poll every ~10 seconds). Once completed, returns the list of discovered pages with their scraped content in the requested formats. This is the REQUIRED companion to create_crawl — create_crawl only kicks off the async job, this tool is how you actually get the content.
Scrape a SPECIFIC, KNOWN list of URLs (typically from different domains). **Do NOT use this for crawling a website** - if the user wants to scrape a whole site or 'crawl' a domain, use `create_crawl` instead. Use this only when you already have an explicit list of URLs to scrape (e.g., user provides a CSV of URLs, or you need to scrape unrelated pages). Returns a batch_id immediately. Use `get_batch_results` with the batch_id to fetch the scraped content once the batch completes (~5–8 min). Set `wait_for_completion_seconds` to poll automatically.
Retrieve the status and scraped content for a batch job. Pass the batch_id returned by batch_scrape_urls. If the batch is completed, returns the scraped content for each URL. If still in_progress, returns the current status so you can call again later.
Answer a factual question using web search, optionally shaped into a flat JSON object of fields (returned with sources and citations). Best for a bounded, factual answer (e.g. a company's founding year, a product's current price). It is NOT reliable for enumerating a live list from a page (e.g. 'the latest N blog posts with titles and dates'). For that, use create_map or get_webpage_content on the page and read the results instead.
Search the web for a given query and return structured results (non-AI, parser-based).
Extract content from a single URL. Supports multiple formats and JavaScript rendering.
Retrieve content of a webpage in markdown
Search and retrieve relevant URLs from a website (URL discovery only - does NOT scrape content). Use this only when the user wants a *filtered list of links* matching a search query. **Do NOT use this as a precursor to scraping** - if the user wants to scrape/crawl a site, use `create_crawl` directly.
概览
What is olostep-mcp?
A Model Context Protocol (MCP) server that integrates with Olostep for web scraping, content extraction, and web search. It requires an Olostep API key (obtainable at olostep.com/auth). Designed for developers and AI agents needing structured data from websites.
How to use olostep-mcp?
Install via npx (npx -y olostep-mcp), Docker (docker pull olostep/mcp-server), or connect to the remote endpoint (https://mcp.olostep.com/mcp). Set the OLOSTEP_API_KEY environment variable. Configure in your MCP client (Cursor, Claude Desktop, Windsurf, VS Code, Metorial) using the provided JSON examples.
Key features of olostep-mcp
- Scrape websites in HTML, Markdown, JSON, or Plain Text
- Parser-based web search with structured results
- AI Answers with citations and optional JSON-shaped outputs
- Batch scraping of up to 10,000 URLs
- Autonomous site crawling from a start URL
- Website URL discovery with include/exclude filters
- Country-specific request routing for geo-targeted content
- Configurable wait times for JavaScript-heavy websites
Use cases of olostep-mcp
- Extract clean markdown from a single documentation page
- Search the web and return structured data (e.g., product listings)
- Get AI-powered answers with cited sources for research
- Batch scrape thousands of URLs for large-scale data collection
- Crawl an entire website to map its structure and content
常见问题
What output formats are supported?
Scraped content can be returned as HTML, Markdown, JSON, or Plain Text (configurable per request).
Can I scrape multiple URLs at once?
Yes, use the `batch_scrape_urls` tool to scrape up to 10,000 URLs in a single job. Results are retrieved later via `get_batch_results`.
How does website crawling work?
Crawling is asynchronous: call `create_crawl` to start a crawl and get a `crawl_id`, then poll `get_crawl_results` to retrieve the scraped pages.
What authentication method is used?
Authentication uses a Bearer token in the `Authorization` header (for the remote endpoint) or the `OLOSTEP_API_KEY` environment variable (for local/docker setups).
其他 分类下的更多 MCP 服务器
Volley.sh
VolleyA good sequence diagram can solve almost any communication problem in engineering. Volley lets your agent write them as plain text, with the real request and response payloads on every arrow. It checks its own syntax wi
The Living Bread
The Living BreadA free remote MCP server for The Living Bread. Bible passages read word for word from a stored public domain text with an evidence label, keyword search and cross references, real churches, gatherings and prayer groups w
MaxKB
1Panel-dev🔥 MaxKB is an open-source platform for building enterprise-grade agents. 强大易用的开源企业级智能体平台。
22.0K
DeepMark
DeepMark helps teachers deliver rapid, consistent marking with meaningful feedback for every student — in a fraction of the time. What once took a week, now takes one free period.
Screenpipe
screenpipeYC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
22.0K
评论