Web Scraping with Anthropic’s MCP Server

@luminati-io

リポジトリホームページ

基本情報

カテゴリ

ブラウザ自動化

トランスポート

stdio

作者

luminati-io

Web Scraping with Anthropic’s MCP Server について

Example MCP server and instructions for connecting Anthropic LLMs to external web scraping tools, with real-world examples and Bright Data integration.

#mcp #web-scraping #anthropic-claude

Web Scraping with Anthropic’s MCP Server の設定

以下の設定を使って、このサーバーを MCP 対応クライアントに追加してください。

{
  "mcpServers": {
    "web-scraping-mcp": {
      "command": "python",
      "args": [
        "-m",
        "venv",
        "mcp-amazon-scraper"
      ]
    }
  }
}

Web Scraping with Anthropic’s MCP Server のツール

ツール一覧はまだありません

この掲載には表示できるツール一覧がまだありません。サーバーにツールがないという意味ではありません。対応機能はプロジェクトのドキュメントをご確認ください。

ツールは README から自動的に抽出されます。メンテナーは ## Tools という見出しの下に記載することで、このタブに反映できます。

概要

What is Web Scraping with Anthropic’s MCP?

This guide explains how to build and run an MCP (Model Context Protocol) server for on-demand web data extraction, enabling LLMs like Claude to retrieve live information from websites. It uses Python, Playwright, and lxml to create a server that scrapes Amazon product pages, and it covers connecting the server to development tools like Cursor or Claude Desktop. It also introduces Bright Data’s MCP for professional-grade extraction.

How to use Web Scraping with Anthropic’s MCP?

Set up a Python 3 virtual environment, install dependencies (pip install mcp playwright lxml, then python -m playwright install), and create a file amazon_scraper_mcp.py using the provided code. Run the server, then connect it via an MCP client in Cursor or Claude Desktop. The server exposes two tools: fetch_page (downloads HTML) and extract_info (parses structured data from saved HTML).

Key features of Web Scraping with Anthropic’s MCP

fetch_page tool downloads HTML via Playwright headless browser.
extract_info tool parses product data using lxml XPath selectors.
Handles JavaScript-heavy pages with configurable delays.
Saves HTML to a temporary file for structured extraction.
Designed for Amazon product pages but adaptable to any URL.

Use cases of Web Scraping with Anthropic’s MCP

Extract real-time product details (price, title, description) from Amazon.
Enable LLMs to answer questions based on current web data.
Automate data collection for price monitoring or comparison sites.
Integrate live web scraping into AI assistants and IDEs.

よくある質問

What is MCP?

Model Context Protocol (MCP) is an open standard by Anthropic that allows LLMs to interact with external tools, APIs, and data sources in a consistent, secure way. It uses a JSON-RPC interface over transports like `stdio`.

Why does an LLM need an MCP server for web scraping?

LLMs cannot naturally browse the internet or fetch live data. MCP provides a standardized bridge that lets them call tools to download and extract information from websites on demand.

What dependencies are required?

Python 3, the MCP Python SDK (`mcp`), Playwright for browser automation, and lxml for HTML parsing. Playwright browser binaries must be installed separately.