How do I get an Olostep API key?
Sign up at [olostep.com/auth](https://olostep.com/auth) to obtain your API key.
Olostep MCP Server
@olostep
基本情報
Olostep MCP Server について
MCP server for Olostep — the web scraping, crawling, and search infrastructure used by top AI companies. Gives any MCP-compatible AI agent the ability to scrape, crawl, batch-extract, and search the web in real time.
Olostep MCP Server の設定
以下の設定を使って、このサーバーを MCP 対応クライアントに追加してください。
{
"mcpServers": {
"olostep-mcp": {
"command": "npx",
"args": [
"-y",
"olostep-mcp"
],
"env": {
"OLOSTEP_API_KEY": "YOUR_API_KEY_HERE"
}
}
}
}Olostep MCP Server のツール
10Get a LIST of URLs on a website (URL discovery only — does NOT scrape content). Use when the user wants a list of links: 'show me all URLs on this site', 'map this website', or when you want to surface candidate URLs to the user before scraping a subset. Prefer `create_crawl` if the goal is to scrape the whole site — it discovers AND scrapes in one workflow. Use this only when the URL list itself is the deliverable.
**PREFERRED tool for crawling a website.** Use this whenever the user says 'crawl', 'scrape the whole site', 'get all pages from a site', or wants multiple pages from a single domain. This is the CORRECT tool for any whole-site scraping task. **Do NOT use `batch_scrape_urls` for crawling** - that tool is only for when you already have a specific list of unrelated URLs from different domains. Starts an ASYNC crawl that autonomously discovers and scrapes pages by following links from a start URL. Returns a crawl_id - the crawl runs in the background. You MUST then call `get_crawl_results` with the returned crawl_id to poll status and retrieve the scraped pages. Do NOT call `get_batch_results` with a crawl_id - crawls and batches are separate resources.
Retrieve the status and scraped pages for a crawl job. Pass the crawl_id returned by create_crawl. If the crawl is still in_progress, returns the current status so you can call again later (poll every ~10 seconds). Once completed, returns the list of discovered pages with their scraped content in the requested formats. This is the REQUIRED companion to create_crawl — create_crawl only kicks off the async job, this tool is how you actually get the content.
Scrape a SPECIFIC, KNOWN list of URLs (typically from different domains). **Do NOT use this for crawling a website** - if the user wants to scrape a whole site or 'crawl' a domain, use `create_crawl` instead. Use this only when you already have an explicit list of URLs to scrape (e.g., user provides a CSV of URLs, or you need to scrape unrelated pages). Returns a batch_id immediately. Use `get_batch_results` with the batch_id to fetch the scraped content once the batch completes (~5–8 min). Set `wait_for_completion_seconds` to poll automatically.
Retrieve the status and scraped content for a batch job. Pass the batch_id returned by batch_scrape_urls. If the batch is completed, returns the scraped content for each URL. If still in_progress, returns the current status so you can call again later.
Answer a factual question using web search, optionally shaped into a flat JSON object of fields (returned with sources and citations). Best for a bounded, factual answer (e.g. a company's founding year, a product's current price). It is NOT reliable for enumerating a live list from a page (e.g. 'the latest N blog posts with titles and dates'). For that, use create_map or get_webpage_content on the page and read the results instead.
Search the web for a given query and return structured results (non-AI, parser-based).
Extract content from a single URL. Supports multiple formats and JavaScript rendering.
Retrieve content of a webpage in markdown
Search and retrieve relevant URLs from a website (URL discovery only - does NOT scrape content). Use this only when the user wants a *filtered list of links* matching a search query. **Do NOT use this as a precursor to scraping** - if the user wants to scrape/crawl a site, use `create_crawl` directly.
概要
What is olostep-mcp?
A Model Context Protocol (MCP) server that integrates with Olostep for web scraping, content extraction, and web search. It requires an Olostep API key (obtainable at olostep.com/auth). Designed for developers and AI agents needing structured data from websites.
How to use olostep-mcp?
Install via npx (npx -y olostep-mcp), Docker (docker pull olostep/mcp-server), or connect to the remote endpoint (https://mcp.olostep.com/mcp). Set the OLOSTEP_API_KEY environment variable. Configure in your MCP client (Cursor, Claude Desktop, Windsurf, VS Code, Metorial) using the provided JSON examples.
Key features of olostep-mcp
- Scrape websites in HTML, Markdown, JSON, or Plain Text
- Parser-based web search with structured results
- AI Answers with citations and optional JSON-shaped outputs
- Batch scraping of up to 10,000 URLs
- Autonomous site crawling from a start URL
- Website URL discovery with include/exclude filters
- Country-specific request routing for geo-targeted content
- Configurable wait times for JavaScript-heavy websites
Use cases of olostep-mcp
- Extract clean markdown from a single documentation page
- Search the web and return structured data (e.g., product listings)
- Get AI-powered answers with cited sources for research
- Batch scrape thousands of URLs for large-scale data collection
- Crawl an entire website to map its structure and content
よくある質問
What output formats are supported?
Scraped content can be returned as HTML, Markdown, JSON, or Plain Text (configurable per request).
Can I scrape multiple URLs at once?
Yes, use the `batch_scrape_urls` tool to scrape up to 10,000 URLs in a single job. Results are retrieved later via `get_batch_results`.
How does website crawling work?
Crawling is asynchronous: call `create_crawl` to start a crawl and get a `crawl_id`, then poll `get_crawl_results` to retrieve the scraped pages.
What authentication method is used?
Authentication uses a Bearer token in the `Authorization` header (for the remote endpoint) or the `OLOSTEP_API_KEY` environment variable (for local/docker setups).
「その他」の他のコンテンツ
Figs, Relationship Expert
Figs O'Sullivan, LMFTCouples therapist Figs O'Sullivan (LMFT, 17 years, endorsed by Sue Johnson, creator of EFT) inside your AI. Ask relationship questions answered from his clinical writing, name the pattern in a fight, get a three-step rep
Yuppsy
Yuppsy MCP: build and publish real estate websites from your AI assistant. Yuppsy lets your AI assistant create live websites for real estate agents and individual property listings. You don't need a designer or any code
FastMCP v2 🚀
jlowin🚀 The fast, Pythonic way to build MCP servers and clients.
26.0K
MaxKB
1Panel-dev🔥 MaxKB is an open-source platform for building enterprise-grade agents. 强大易用的开源企业级智能体平台。
22.0K
Reactive Resume
amruthpillaiA one-of-a-kind resume builder that keeps your privacy in mind. Completely secure, customizable, portable, open-source and free forever. Try it out today!
39.5K
コメント