Web Scraper MCP
@navin4078
Web Scraper MCP について
Scrape websites and let them talk to your LLM
基本情報
設定
以下の設定を使って、このサーバーを MCP 対応クライアントに追加してください。
{
"mcpServers": {
"web-scraper": {
"command": "/full/path/to/your/venv/bin/python",
"args": [
"/full/path/to/your/app_mcp.py"
]
}
}
}ツール
ツールは検出されませんでした
ツールは README から自動的に抽出されます。メンテナーは ## Tools という見出しの下に記載することで、このタブに反映できます。
概要
What is Web Scraper MCP?
A lightweight and efficient web scraping MCP server using the direct STDIO protocol, designed for integration with Claude Desktop. It enables users to extract text, links, images, tables, metadata, and headlines from any webpage using natural language commands.
How to use Web Scraper MCP?
Install via the automated setup script (setup.sh) or manually with pip install -r requirements.txt in a Python virtual environment. Configure Claude Desktop by adding the server to claude_desktop_config.json with absolute paths to the Python interpreter and app_mcp.py. Once configured, invoke scraping tools like scrape_website, extract_headlines, extract_metadata, or get_page_info through natural language prompts in Claude.
Key features of Web Scraper MCP
- Text extraction with CSS selectors
- Link, image, and table data extraction
- Comprehensive metadata and Open Graph tags
- Headline extraction with hierarchy
- Custom CSS selector support and result limits
- Native Claude Desktop integration (STDIO)
Use cases of Web Scraper MCP
- Scrape text content from a blog or news article
- Extract all links from a link aggregator like Hacker News
- Get headlines and structure from a news website
- Retrieve images with alt text from a gallery page
- Extract metadata and Open Graph tags from any URL
FAQ from Web Scraper MCP
What is the MCP protocol and how does it work?
The MCP (Model Context Protocol) server operates using direct STDIO communication, meaning no HTTP server is required. It integrates natively with Claude Desktop for automatic lifecycle management.
What dependencies does Web Scraper MCP require?
The minimal dependencies are: requests, beautifulsoup4, lxml, and the mcp library. All are installed via pip install -r requirements.txt.
How do I configure Web Scraper MCP with Claude Desktop?
Add a new entry under mcpServers in your claude_desktop_config.json file (located in ~/Library/Application Support/Claude/ on macOS, %APPDATA%\Claude\ on Windows, or ~/.config/Claude/ on Linux). The entry must contain the absolute path to the Python interpreter in your virtual environment and the absolute path to app_mcp.py as arguments.
What should I do if the MCP server is not appearing in Claude Desktop?
Verify that the file paths in your configuration are absolute and correct. Check that the virtual environment is activated and dependencies are installed. Test the server manually by running python app_mcp.py from the project directory. Ensure your JSON configuration is valid (no trailing commas).
What security measures are included?
The server enforces a 10-second request timeout, validates URLs and parameters, uses modern browser User-Agent headers, and includes configurable result limits to prevent overload. Users are reminded to respect robots.txt and practice rate limiting.
「ブラウザ自動化」の他のコンテンツ
App Store Scraper MCP Server
appreply-coThis is an MCP server that provides tools to LLMs for searching and analyzing apps from both Google Play Store and Apple App Store – perfect for ASO.
Webpage Screenshot MCP Server
ananddtyagiAn MCP that gives your agent the ability to snap a screenshot of webpages. Useful when you want your agent to check its progress during development.
@mendableai/firecrawl Mcp Server
mendableaiThe API to search, scrape, and interact with the web at scale. 🔥
Fetcher MCP
jae-jaeMCP server for fetch web page content using Playwright headless browser.
Scrapling Fetch MCP
cyberchittaHelps AI assistants access text content from bot-protected websites. MCP server that fetches HTML/markdown from sites with anti-automation measures using Scrapling.
コメント