What is the MCP protocol and how does it work?
The MCP (Model Context Protocol) server operates using direct STDIO communication, meaning no HTTP server is required. It integrates natively with Claude Desktop for automatic lifecycle management.
W
Web Scraper MCP Server
@navin4078
Basic information
About Web Scraper MCP Server
Scrape websites and let them talk to your LLM
Web Scraper MCP Server config
Add this server to your MCP-compatible client using the configuration below.
{
"mcpServers": {
"web-scraper": {
"command": "/full/path/to/your/venv/bin/python",
"args": [
"/full/path/to/your/app_mcp.py"
]
}
}
}Web Scraper MCP Server tools
Tool list not available yet
This listing does not currently have a tool list to display. This does not mean the server has no tools; check its documentation for supported capabilities.
We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.
Overview
What is Web Scraper MCP?
A lightweight and efficient web scraping MCP server using the direct STDIO protocol, designed for integration with Claude Desktop. It enables users to extract text, links, images, tables, metadata, and headlines from any webpage using natural language commands.
How to use Web Scraper MCP?
Install via the automated setup script (setup.sh) or manually with pip install -r requirements.txt in a Python virtual environment. Configure Claude Desktop by adding the server to claude_desktop_config.json with absolute paths to the Python interpreter and app_mcp.py. Once configured, invoke scraping tools like scrape_website, extract_headlines, extract_metadata, or get_page_info through natural language prompts in Claude.
Key features of Web Scraper MCP
- Text extraction with CSS selectors
- Link, image, and table data extraction
- Comprehensive metadata and Open Graph tags
- Headline extraction with hierarchy
- Custom CSS selector support and result limits
- Native Claude Desktop integration (STDIO)
Use cases of Web Scraper MCP
- Scrape text content from a blog or news article
- Extract all links from a link aggregator like Hacker News
- Get headlines and structure from a news website
- Retrieve images with alt text from a gallery page
- Extract metadata and Open Graph tags from any URL
Frequently asked questions
What dependencies does Web Scraper MCP require?
The minimal dependencies are: `requests`, `beautifulsoup4`, `lxml`, and the `mcp` library. All are installed via `pip install -r requirements.txt`.
How do I configure Web Scraper MCP with Claude Desktop?
Add a new entry under `mcpServers` in your `claude_desktop_config.json` file (located in `~/Library/Application Support/Claude/` on macOS, `%APPDATA%\Claude\` on Windows, or `~/.config/Claude/` on Linux). The entry must contain the absolute path to the Python interpreter in your virtual environment and the absolute path to `app_mcp.py` as arguments.
What should I do if the MCP server is not appearing in Claude Desktop?
Verify that the file paths in your configuration are absolute and correct. Check that the virtual environment is activated and dependencies are installed. Test the server manually by running `python app_mcp.py` from the project directory. Ensure your JSON configuration is valid (no trailing commas).
What security measures are included?
The server enforces a 10-second request timeout, validates URLs and parameters, uses modern browser User-Agent headers, and includes configurable result limits to prevent overload. Users are reminded to respect `robots.txt` and practice rate limiting.
More Browser Automation MCP servers
Loadster
loadsterLoadster is a load testing, stress testing, and synthetic monitoring platform. Run Playwright, Browser Bots, and Protocol Bots to test and monitor your websites, web apps, and APIs. Loadster MCP gives your agents the abi
Browser Control MCP
eyalzhMCP server paired with a browser extension that enables AI agents to control the user's browser.
299
playwright-server MCP server
blackwhite084189
Puppeteer MCP Server
merajmehrabiThis MCP server provides browser automation capabilities through Puppeteer, allowing interaction with both new browser instances and existing Chrome windows.
474
Nullprint
nullprintaiLet Claude Code, Claude Desktop, OpenAI Codex, Kimi and any MCP-capable AI agent create, launch, check and drive browser profiles in Nullprint, the AI antidetect browser. Install with one command.
Comments