Web Crawler MCP Server

@JonathanHsuHH

代码仓库

基本信息

分类

浏览器自动化

许可证

MIT license

传输方式

stdio

作者

JonathanHsuHH

关于 Web Crawler MCP Server

A MCP server that provides a web crawling and content extraction tool for AI assistants

Web Crawler MCP Server 配置

使用下面的配置,将此服务器添加到你的 MCP 客户端。

{
  "mcpServers": {
    "web-crawler-mcp-server": {
      "command": "node",
      "args": [
        "build/index.js"
      ]
    }
  }
}

Web Crawler MCP Server 工具

** Extracts and returns the cleaned text content from a specified URL.

`url` (string, required): The URL to extract content from.

概览

What is Web Crawler MCP Server?

A Model Context Protocol (MCP) server that provides a web crawling and content extraction tool for AI assistants such as Claude Desktop, Cursor, and other MCP-compatible clients. It extracts and cleans main text content from any public web page.

How to use Web Crawler MCP Server?

Install dependencies with npm install, build with npm run build, then run directly with node build/index.js. Alternatively, configure it as an MCP server in your client (e.g., Claude Desktop) by adding a JSON entry that points command and args to the built server.

Key features of Web Crawler MCP Server

Extracts and cleans main text content from public web pages.
Uses Puppeteer with stealth plugin to bypass anti-bot protections.
Returns readable, whitespace-normalized text for LLM consumption.
Easy integration with Claude Desktop, Cursor, and other MCP clients.
Launches a real browser instance (headless: false) for best compatibility.

Use cases of Web Crawler MCP Server

An AI assistant fetching and reading article text from a news URL.
A developer integrating web page content extraction into an MCP-compatible toolchain.
Enabling LLMs to consume clean, normalized text from any public website.

常见问题

What dependencies are required to run Web Crawler MCP Server?

Node.js v16 or higher, and an MCP-compatible client. Some Linux environments may require additional Puppeteer dependencies.

What output format does the server return?

The server returns plain, whitespace-normalized text, suitable for LLM input.

How does the server handle anti-bot protections?

It uses Puppeteer with a stealth plugin to bypass common anti-bot measures.

Can I customize the parsing logic?

Yes, for advanced parsing you can modify the Cheerio logic in `src/index.ts`.

What transport or authentication does the server use?

The README does not mention any transport or authentication configuration; it is intended to be configured as a local MCP server via stdin/stdout.

浏览器自动化分类下的更多 MCP 服务器

Gologin MCP Server

gologinapp

MCP server for managing Gologin browser profiles, proxies, fingerprints and cloud browser sessions through AI conversations

QuantumProxies — Residential, Mobile & Datacenter Proxy MCP Server

quantumproxies

Give your AI agent its own proxies. QuantumProxies exposes residential, mobile, datacenter, ISP and IPv6 endpoints as MCP tools: list your plans, generate ready-to-use proxy strings with credentials, target by country, s