A lightweight and efficient web scraping MCP server using direct STDIO protocol

🚀 Quick Start

Option 1: Automated Setup

# Clone and setup
git clone https://github.com/navin4078/mcp-web-scraper
cd mcp-web-scraper
chmod +x setup.sh && ./setup.sh

Option 2: Manual Setup

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install minimal dependencies
pip install -r requirements.txt

⚙️ Claude Desktop Configuration

Step 1: Find Your Paths

# Get absolute paths (run this in your project directory)
echo "Python path: $(pwd)/venv/bin/python"
echo "Script path: $(pwd)/app_mcp.py"

Step 2: Configure Claude Desktop

Open your Claude Desktop config file:

macOS:

~/Library/Application Support/Claude/claude_desktop_config.json

Windows:

%APPDATA%\Claude\claude_desktop_config.json

Linux:

~/.config/Claude/claude_desktop_config.json

Step 3: Add Configuration

Add this to your config file:

{
  "mcpServers": {
    "web-scraper": {
      "command": "/full/path/to/your/venv/bin/python",
      "args": ["/full/path/to/your/app_mcp.py"]
    }
  }
}

Example:

{
  "mcpServers": {
    "web-scraper": {
      "command": "/Users/username/Desktop/scrapper/venv/bin/python",
      "args": ["/Users/username/Desktop/scrapper/app_mcp.py"]
    }
  }
}

Step 4: Restart Claude Desktop

Completely close Claude Desktop (Cmd+Q on Mac)
Restart the application
Look for the hammer icon (🔨)
You should see "web-scraper" in your MCP servers

🛠 Available Tools

scrape_website

Extract data from websites with flexible options:

extract_type: text, links, images, table
selector: CSS selector for targeting specific elements
max_results: Limit number of results (1-50)

extract_headlines

Get all headlines (h1, h2, h3) from a webpage with hierarchy and attributes.

extract_metadata

Extract comprehensive metadata:

Basic: title, description, keywords, author
Open Graph: og:title, og:description, og:image
Twitter Cards: twitter:title, twitter:description

get_page_info

Get page structure overview:

Element counts (paragraphs, headings, links, images, tables)
Basic metadata
Page statistics

💡 Usage Examples

Basic Scraping

Scrape the text content from https://example.com

Extract all links from https://news.ycombinator.com

Get headlines from https://www.bbc.com/news

Advanced Examples

Extract all images from https://example.com with their alt text

Scrape text from https://example.com using the CSS selector ".article-content p"

Get metadata and Open Graph tags from https://github.com

What's the page structure of https://stackoverflow.com?

Specific Selectors

Extract text from https://news.ycombinator.com using selector ".titleline a"

Get all table data from https://example.com/data-page

Scrape only paragraph text from articles using selector "article p"

📁 Project Structure

scrapper/
├── app_mcp.py             # Main MCP server (STDIO)
├── requirements.txt       # Minimal dependencies
├── setup.sh              # Simple setup script
├── .gitignore            # Git ignore rules
└── README.md             # This file

🔧 Features

Web Scraping Capabilities

✅ Text extraction with CSS selectors
✅ Link extraction with full attributes
✅ Image extraction with metadata
✅ Table data extraction and formatting
✅ Comprehensive metadata extraction
✅ Headline extraction with hierarchy
✅ Custom CSS selector support
✅ Configurable result limits
✅ Error handling and validation

MCP Integration

✅ Direct STDIO protocol (no HTTP needed)
✅ Native Claude Desktop integration
✅ Automatic server lifecycle management
✅ Schema validation and documentation
✅ Comprehensive error handling
✅ Minimal dependencies

🛡 Security & Best Practices

Respect robots.txt: Always check robots.txt before scraping
Rate limiting: Built-in 10-second request timeout
User-Agent: Uses modern browser headers
Input validation: URL and parameter validation
Error handling: Graceful error handling and reporting
Resource limits: Configurable result limits prevent overload

🐛 Troubleshooting

MCP Server Not Appearing

Check your paths:

# Verify files exist
ls -la /path/to/your/venv/bin/python
ls -la /path/to/your/app_mcp.py

# Test the script manually
/path/to/your/venv/bin/python /path/to/your/app_mcp.py

Validate JSON configuration:

Use a JSON validator to check syntax
Ensure no trailing commas
Use absolute paths (not relative)

Permission Issues

# Make script executable
chmod +x app_mcp.py

# Check virtual environment
source venv/bin/activate
python --version

Import Errors

# Reinstall dependencies
source venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

Testing the MCP Server

You can test if the server works by running it manually:

source venv/bin/activate
python app_mcp.py

The server should start and wait for STDIO input from Claude Desktop.

📚 Dependencies

requests: HTTP library for web requests
beautifulsoup4: HTML/XML parsing
lxml: Fast XML and HTML processor
mcp: Model Context Protocol library

🤝 Contributing

Fork the repository
Create a feature branch
Test thoroughly with Claude Desktop
Submit a pull request

📄 License

This project is open source and available under the MIT License.

🔗 Resources

Simple, efficient web scraping for Claude Desktop! 🕷️✨

Web Scraper MCP