Web Crawler MCP Server

Name: Web Crawler MCP Server
Author: JonathanHsuHH

@JonathanHsuHH

代码仓库

关于 Web Crawler MCP Server

A MCP server that provides a web crawling and content extraction tool for AI assistants

基本信息

分类

浏览器自动化

许可证

MIT license

运行时

node

传输方式

stdio

发布者

JonathanHsuHH

配置

使用下面的配置,将此服务器添加到你的 MCP 客户端。

{
  "mcpServers": {
    "web-crawler-mcp-server": {
      "command": "node",
      "args": [
        "build/index.js"
      ]
    }
  }
}

工具

** Extracts and returns the cleaned text content from a specified URL.

`url` (string, required): The URL to extract content from.

概览

What is Web Crawler MCP Server?

A Model Context Protocol (MCP) server that provides a web crawling and content extraction tool for AI assistants such as Claude Desktop, Cursor, and other MCP-compatible clients. It extracts and cleans main text content from any public web page.

How to use Web Crawler MCP Server?

Install dependencies with npm install, build with npm run build, then run directly with node build/index.js. Alternatively, configure it as an MCP server in your client (e.g., Claude Desktop) by adding a JSON entry that points command and args to the built server.

Key features of Web Crawler MCP Server

Extracts and cleans main text content from public web pages.
Uses Puppeteer with stealth plugin to bypass anti-bot protections.
Returns readable, whitespace-normalized text for LLM consumption.
Easy integration with Claude Desktop, Cursor, and other MCP clients.
Launches a real browser instance (headless: false) for best compatibility.

Use cases of Web Crawler MCP Server

An AI assistant fetching and reading article text from a news URL.
A developer integrating web page content extraction into an MCP-compatible toolchain.
Enabling LLMs to consume clean, normalized text from any public website.