AI-First Scraper

Name: AI-First Scraper
Author: yubinkim444

@yubinkim444

代码仓库

关于 AI-First Scraper

Ad-free web scraping and search exposed as 3 MCP tools

#web-scraping #markdown #search #ai-agents #rag

基本信息

分类

浏览器自动化

传输方式

stdio

发布者

yubinkim444

提交者

YooBin Kim

配置

使用下面的配置,将此服务器添加到你的 MCP 客户端。

{
  "mcpServers": {
    "ai-first-scraper": {
      "command": "uvx",
      "args": [
        "ai-first-scraper-mcp"
      ]
    }
  }
}

工具

未检测到工具

工具是从 README 中自动提取的。维护者可以在 ## Tools 标题下列出工具,即可填充这部分内容。

概览

What is AI-First Scraper?

AI-First Scraper is an MCP server that adds three web-scraping and search tools to any MCP-compatible agent (Claude Desktop, Cursor, Cline). It fetches URLs and returns clean Markdown, stripping ads and HTML clutter, and can also run web searches that return top results already converted to Markdown.

How to use AI-First Scraper?

Install via uvx (no install, runs from PyPI) or pip install ai-first-scraper-mcp. Add the server configuration to your client’s settings JSON (e.g., claude_desktop_config.json, ~/.cursor/mcp.json, or cline_mcp_settings.json). Restart the client; the three tools (fetch_page, fetch_pages_batch, search_web) appear automatically.

Key features of AI-First Scraper

fetch_page: convert one URL to clean Markdown (HTML or PDF).
fetch_pages_batch: fetch up to 25 URLs in parallel, return Markdown.
search_web: run a web search and return top-k results as Markdown.
Backed by ai-first-scraper and ai-first-search APIs.
Supports self‑hosted backends via environment variables.
Requires Python 3.10+.

Use cases of AI-First Scraper

Let an AI agent summarize recent articles without raw HTML.
Batch‑scrape multiple pages for competitive analysis.
Search the web and immediately pass clean content to an LLM.
Replace manual curl+parsing workflows in MCP‑powered automation.

FAQ from AI-First Scraper

What tools does AI-First Scraper provide?

It provides fetch_page (single URL to Markdown), fetch_pages_batch (up to 25 URLs in parallel to Markdown), and search_web (web search with Markdown results).

How do I install AI-First Scraper?

The fastest method is uvx (no install). Alternatively, run pip install ai-first-scraper-mcp and then set the command to ai-first-scraper-mcp in your MCP client config.

Can I run my own backend instead of the public APIs?

Yes. Set environment variables SCRAPER_URL and SEARCH_URL in the MCP server config to point at your self‑hosted instances. You can also set AFS_TIMEOUT.

Where do I put the configuration file?

For Claude Desktop (macOS): ~/Library/Application Support/Claude/claude_desktop_config.json. For Claude Desktop (Windows): %APPDATA%\Claude\claude_desktop_config.json. For Cursor: ~/.cursor/mcp.json. For Cline (VS Code): ~/Library/Application Support/Code/User/globalStorage/saoudrizwan.claude-dev/settings/cline_mcp_settings.json.

What dependencies does AI-First Scraper have?

It requires Python 3.10 or higher and uses the external ai-first-scraper and ai-first-search APIs (public or self‑hosted). The server communicates over MCP via stdio.

浏览器自动化分类下的更多 MCP 服务器

Scrapling Fetch MCP

cyberchitta

Helps AI assistants access text content from bot-protected websites. MCP server that fetches HTML/markdown from sites with anti-automation measures using Scrapling.

101

BrowserTools MCP

AgentDeskAI

Monitor browser logs directly from Cursor and other MCP compatible IDEs.

7.3K

Apify Model Context Protocol (MCP) Server

apify

The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automation tools available on the Apify Store.

1.7K