MCP Smart Crawler

@loo-y

Basic information

About MCP Smart Crawler

MCP Smart Crawler is a Model Context Protocol (MCP) server uses Playwright to crawl web content, extract metadata, and download resources such as videos and images.

MCP Smart Crawler config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "mcp-smart-crawler": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-smart-crawler@latest",
        "--download-folder",
        "C:\\downloads"
      ]
    }
  }
}

MCP Smart Crawler tools

Tool list not available yet

This listing does not currently have a tool list to display. This does not mean the server has no tools; check its documentation for supported capabilities.

Fetch the live tool list by running this server in a temporary sandbox using the button above.

Overview

What is MCP Smart Crawler?

MCP Smart Crawler is a Model Context Protocol (MCP) server that uses Playwright to crawl web content, extract metadata, and download resources such as videos and images. It is designed for users who need to programmatically retrieve and save content from Xiaohongshu (小红书) share links.

How to use MCP Smart Crawler?

Configure your MCP client with the provided JSON, setting the command to npx and adding optional arguments like --download-folder to specify a local download directory. Then send a chat message containing a Xiaohongshu share link to trigger crawling and downloading.

Key features of MCP Smart Crawler

Extracts metadata (title, description, images) from Xiaohongshu posts.
Downloads videos and images from Xiaohongshu share links.
Uses Playwright for browser automation.
Configurable download folder via command-line argument.

Use cases of MCP Smart Crawler

Automatically fetching and saving Xiaohongshu post images for content aggregation.
Downloading videos from shared Xiaohongshu links for offline viewing.
Extracting metadata from Xiaohongshu posts for analysis or archiving.

Frequently asked questions

What does MCP Smart Crawler do?

It crawls web pages using Playwright, extracts metadata like title and images from Xiaohongshu posts, and downloads associated videos and images.

What dependencies does MCP Smart Crawler require?

It requires Playwright for browser automation, and is run via `npx` (Node.js package runner). No other runtime dependencies are specified.

Where are downloaded files saved?

Files are saved to the local folder specified by the `--download-folder` argument. If not provided, the default behavior is not documented.

Does MCP Smart Crawler support authentication or transport security?

No authentication or transport security mechanisms are mentioned in the README.

Can MCP Smart Crawler work with websites other than Xiaohongshu?

The README only describes functionality for Xiaohongshu share links; no other websites are mentioned.

Comments

More Browser Automation MCP servers

Puppeteer MCP Server

merajmehrabi

This MCP server provides browser automation capabilities through Puppeteer, allowing interaction with both new browser instances and existing Chrome windows.

474

BrowserTools MCP

AgentDeskAI

Monitor browser logs directly from Cursor and other MCP compatible IDEs.

7.3K

Playwright Mcp

microsoft

Playwright MCP server

34.8K

TinyFish

tinyfish

Give your AI agent the live web. One MCP server for Search, Fetch, Browser, and Agent: browse real sites, extract structured data, fill forms, and run multi-step workflows. Search and Fetch are free.

Fetch

modelcontextprotocol

Model Context Protocol Servers

88.1K