What is the MCP protocol and how does it work?
The MCP (Model Context Protocol) server operates using direct STDIO communication, meaning no HTTP server is required. It integrates natively with Claude Desktop for automatic lifecycle management.
W
Web Scraper MCP Server
@navin4078
基本信息
关于 Web Scraper MCP Server
Scrape websites and let them talk to your LLM
Web Scraper MCP Server 配置
使用下面的配置,将此服务器添加到你的 MCP 客户端。
{
"mcpServers": {
"web-scraper": {
"command": "/full/path/to/your/venv/bin/python",
"args": [
"/full/path/to/your/app_mcp.py"
]
}
}
}Web Scraper MCP Server 工具
工具列表暂未提供
当前收录暂时没有可展示的工具列表，不代表服务没有工具。请查阅项目文档了解支持的能力。
工具是从 README 中自动提取的。维护者可以在 ## Tools 标题下列出工具,即可填充这部分内容。
概览
What is Web Scraper MCP?
A lightweight and efficient web scraping MCP server using the direct STDIO protocol, designed for integration with Claude Desktop. It enables users to extract text, links, images, tables, metadata, and headlines from any webpage using natural language commands.
How to use Web Scraper MCP?
Install via the automated setup script (setup.sh) or manually with pip install -r requirements.txt in a Python virtual environment. Configure Claude Desktop by adding the server to claude_desktop_config.json with absolute paths to the Python interpreter and app_mcp.py. Once configured, invoke scraping tools like scrape_website, extract_headlines, extract_metadata, or get_page_info through natural language prompts in Claude.
Key features of Web Scraper MCP
- Text extraction with CSS selectors
- Link, image, and table data extraction
- Comprehensive metadata and Open Graph tags
- Headline extraction with hierarchy
- Custom CSS selector support and result limits
- Native Claude Desktop integration (STDIO)
Use cases of Web Scraper MCP
- Scrape text content from a blog or news article
- Extract all links from a link aggregator like Hacker News
- Get headlines and structure from a news website
- Retrieve images with alt text from a gallery page
- Extract metadata and Open Graph tags from any URL
常见问题
What dependencies does Web Scraper MCP require?
The minimal dependencies are: `requests`, `beautifulsoup4`, `lxml`, and the `mcp` library. All are installed via `pip install -r requirements.txt`.
How do I configure Web Scraper MCP with Claude Desktop?
Add a new entry under `mcpServers` in your `claude_desktop_config.json` file (located in `~/Library/Application Support/Claude/` on macOS, `%APPDATA%\Claude\` on Windows, or `~/.config/Claude/` on Linux). The entry must contain the absolute path to the Python interpreter in your virtual environment and the absolute path to `app_mcp.py` as arguments.
What should I do if the MCP server is not appearing in Claude Desktop?
Verify that the file paths in your configuration are absolute and correct. Check that the virtual environment is activated and dependencies are installed. Test the server manually by running `python app_mcp.py` from the project directory. Ensure your JSON configuration is valid (no trailing commas).
What security measures are included?
The server enforces a 10-second request timeout, validates URLs and parameters, uses modern browser User-Agent headers, and includes configurable result limits to prevent overload. Users are reminded to respect `robots.txt` and practice rate limiting.
浏览器自动化 分类下的更多 MCP 服务器
Playwright Mcp
microsoftPlaywright MCP server
34.8K
BrowserTools MCP
AgentDeskAIMonitor browser logs directly from Cursor and other MCP compatible IDEs.
7.3K
browser-use-mcp-server
co-browserBrowse the web, directly from Cursor etc.
827
playwright-server MCP server
blackwhite084189
Puppeteer
modelcontextprotocolModel Context Protocol Servers
88.1K
评论