a year ago
developer-toolsMCP Smart Crawler is a Model Context Protocol (MCP) server uses Playwright to crawl web content, extract metadata, and download resources such as videos and images.
Overview
What is MCP Smart Crawler?
MCP Smart Crawler is a Model Context Protocol (MCP) server that utilizes Playwright to crawl web content, extract metadata, and download resources such as videos and images from platforms like Xiaohongshu (小红书).
How to use MCP Smart Crawler?
To use MCP Smart Crawler, configure your MCP client by adding the provided server settings. Adjust the command and args based on your server script execution.
Example Configuration:
{
"mcpServers": {
"mcp-smart-crawler": {
"command": "npx",
"args": [
"-y",
"mcp-smart-crawler",
"--download-folder", // optional
"c:\\downloads" // optional
],
}
}
}
Key features of MCP Smart Crawler?
- Extracts metadata (title, description, images) from Xiaohongshu posts.
- Downloads videos and images from Xiaohongshu share links.
- Utilizes Playwright for efficient browser automation.
Use cases of MCP Smart Crawler?
- Extracting content and metadata from Xiaohongshu posts for analysis.
- Downloading multimedia resources for offline access.
- Automating web scraping tasks for research and data collection.
FAQ from MCP Smart Crawler?
- Can MCP Smart Crawler extract data from any website?
No, it is specifically designed for Xiaohongshu and may require adjustments for other sites.
- Is there a limit to the number of resources I can download?
The limit depends on the server and network capabilities, but generally, it can handle multiple requests efficiently.
- Is MCP Smart Crawler free to use?
Yes, it is open-source and free to use.
Server Config
{
"mcpServers": {
"mcp-smart-crawler": {
"command": "npx",
"args": [
"-y",
"mcp-smart-crawler@latest",
"--download-folder",
"C:\\downloads"
]
}
}
}