Web Crawler MCP Server

@JonathanHsuHH

Basic information

About Web Crawler MCP Server

A MCP server that provides a web crawling and content extraction tool for AI assistants

Web Crawler MCP Server config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "web-crawler-mcp-server": {
      "command": "node",
      "args": [
        "build/index.js"
      ]
    }
  }
}

Web Crawler MCP Server tools

** Extracts and returns the cleaned text content from a specified URL.

`url` (string, required): The URL to extract content from.

Overview

What is Web Crawler MCP Server?

A Model Context Protocol (MCP) server that provides a web crawling and content extraction tool for AI assistants such as Claude Desktop, Cursor, and other MCP-compatible clients. It extracts and cleans main text content from any public web page.

How to use Web Crawler MCP Server?

Install dependencies with npm install, build with npm run build, then run directly with node build/index.js. Alternatively, configure it as an MCP server in your client (e.g., Claude Desktop) by adding a JSON entry that points command and args to the built server.

Key features of Web Crawler MCP Server

Extracts and cleans main text content from public web pages.
Uses Puppeteer with stealth plugin to bypass anti-bot protections.
Returns readable, whitespace-normalized text for LLM consumption.
Easy integration with Claude Desktop, Cursor, and other MCP clients.
Launches a real browser instance (headless: false) for best compatibility.

Use cases of Web Crawler MCP Server

An AI assistant fetching and reading article text from a news URL.
A developer integrating web page content extraction into an MCP-compatible toolchain.
Enabling LLMs to consume clean, normalized text from any public website.

Frequently asked questions

What dependencies are required to run Web Crawler MCP Server?

Node.js v16 or higher, and an MCP-compatible client. Some Linux environments may require additional Puppeteer dependencies.

What output format does the server return?

The server returns plain, whitespace-normalized text, suitable for LLM input.

How does the server handle anti-bot protections?

It uses Puppeteer with a stealth plugin to bypass common anti-bot measures.

Can I customize the parsing logic?

Yes, for advanced parsing you can modify the Cheerio logic in `src/index.ts`.

What transport or authentication does the server use?

The README does not mention any transport or authentication configuration; it is intended to be configured as a local MCP server via stdin/stdout.

Comments

More Browser Automation MCP servers

Browser Control MCP

eyalzh

MCP server paired with a browser extension that enables AI agents to control the user's browser.

299

Loadster

loadster

Loadster is a load testing, stress testing, and synthetic monitoring platform. Run Playwright, Browser Bots, and Protocol Bots to test and monitor your websites, web apps, and APIs. Loadster MCP gives your agents the abi

Nullprint

nullprintai

Let Claude Code, Claude Desktop, OpenAI Codex, Kimi and any MCP-capable AI agent create, launch, check and drive browser profiles in Nullprint, the AI antidetect browser. Install with one command.

Parse

Parse.bot

Parse allows your agent to access thousands of APIs instantly. You can use Parse to build an API for any action on any website and edit any previously existing API in our catalogue.

Open Browser Use

iFurySt

🔮 Platform-neutral Browser Use for AI agents: real Chrome automation with a CLI + SDKs, no lock-in, dead simple.

371