Web Crawler MCP Server

Name: Web Crawler MCP Server
Author: JonathanHsuHH

@JonathanHsuHH

Repository

About Web Crawler MCP Server

A MCP server that provides a web crawling and content extraction tool for AI assistants

Basic information

Config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "web-crawler-mcp-server": {
      "command": "node",
      "args": [
        "build/index.js"
      ]
    }
  }
}

Tools

** Extracts and returns the cleaned text content from a specified URL.

`url` (string, required): The URL to extract content from.

Overview

What is Web Crawler MCP Server?

A Model Context Protocol (MCP) server that provides a web crawling and content extraction tool for AI assistants such as Claude Desktop, Cursor, and other MCP-compatible clients. It extracts and cleans main text content from any public web page.

How to use Web Crawler MCP Server?

Install dependencies with npm install, build with npm run build, then run directly with node build/index.js. Alternatively, configure it as an MCP server in your client (e.g., Claude Desktop) by adding a JSON entry that points command and args to the built server.

Key features of Web Crawler MCP Server

Extracts and cleans main text content from public web pages.
Uses Puppeteer with stealth plugin to bypass anti-bot protections.
Returns readable, whitespace-normalized text for LLM consumption.
Easy integration with Claude Desktop, Cursor, and other MCP clients.
Launches a real browser instance (headless: false) for best compatibility.

Use cases of Web Crawler MCP Server

An AI assistant fetching and reading article text from a news URL.
A developer integrating web page content extraction into an MCP-compatible toolchain.
Enabling LLMs to consume clean, normalized text from any public website.