広告掲載
MCP.so
ログイン

Web Crawler MCP Server

@JonathanHsuHH

リポジトリ

基本情報

カテゴリ

ブラウザ自動化

ライセンス

MIT license

トランスポート

stdio

作者

JonathanHsuHH

Web Crawler MCP Server について

A MCP server that provides a web crawling and content extraction tool for AI assistants

Web Crawler MCP Server の設定

以下の設定を使って、このサーバーを MCP 対応クライアントに追加してください。

{
  "mcpServers": {
    "web-crawler-mcp-server": {
      "command": "node",
      "args": [
        "build/index.js"
      ]
    }
  }
}

Web Crawler MCP Server のツール

2

** Extracts and returns the cleaned text content from a specified URL.

`url` (string, required): The URL to extract content from.

概要

What is Web Crawler MCP Server?

A Model Context Protocol (MCP) server that provides a web crawling and content extraction tool for AI assistants such as Claude Desktop, Cursor, and other MCP-compatible clients. It extracts and cleans main text content from any public web page.

How to use Web Crawler MCP Server?

Install dependencies with npm install, build with npm run build, then run directly with node build/index.js. Alternatively, configure it as an MCP server in your client (e.g., Claude Desktop) by adding a JSON entry that points command and args to the built server.

Key features of Web Crawler MCP Server

  • Extracts and cleans main text content from public web pages.
  • Uses Puppeteer with stealth plugin to bypass anti-bot protections.
  • Returns readable, whitespace-normalized text for LLM consumption.
  • Easy integration with Claude Desktop, Cursor, and other MCP clients.
  • Launches a real browser instance (headless: false) for best compatibility.

Use cases of Web Crawler MCP Server

  • An AI assistant fetching and reading article text from a news URL.
  • A developer integrating web page content extraction into an MCP-compatible toolchain.
  • Enabling LLMs to consume clean, normalized text from any public website.

よくある質問

1

What dependencies are required to run Web Crawler MCP Server?

Node.js v16 or higher, and an MCP-compatible client. Some Linux environments may require additional Puppeteer dependencies.

2

What output format does the server return?

The server returns plain, whitespace-normalized text, suitable for LLM input.

3

How does the server handle anti-bot protections?

It uses Puppeteer with a stealth plugin to bypass common anti-bot measures.

4

Can I customize the parsing logic?

Yes, for advanced parsing you can modify the Cheerio logic in `src/index.ts`.

5

What transport or authentication does the server use?

The README does not mention any transport or authentication configuration; it is intended to be configured as a local MCP server via stdin/stdout.

コメント

「ブラウザ自動化」の他のコンテンツ

Firecrawl Mcp Server

mendableai

🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.

6.9K

Parse

Parse.bot

Parse allows your agent to access thousands of APIs instantly. You can use Parse to build an API for any action on any website and edit any previously existing API in our catalogue.

Fastcrw

us

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary.

286

Nullprint

nullprintai

Let Claude Code, Claude Desktop, OpenAI Codex, Kimi and any MCP-capable AI agent create, launch, check and drive browser profiles in Nullprint, the AI antidetect browser. Install with one command.

Playwright MCP Server 🎭

executeautomation

Playwright Model Context Protocol Server - Tool to automate Browsers and APIs in Claude Desktop, Cline, Cursor IDE and More 🔌

5.6K
広告掲載