MCP.so
Sign In

TheCrawler MCP Server

@manchittlab

Basic information

Category

Browser Automation

Last updated

License

AGPL-3.0

Author

manchittlab

About TheCrawler MCP Server

Open-source web scraper + LLM-powered structured extraction. PDF/DOCX, markdown, JSON-LD, microdata, commerce data, forms, 16 analytics-tracker detection. Structured errors with retryable flags. Adaptive Cheerio->Playwright. CLI, npm, REST API, and MCP server. AGPL-3.0.

TheCrawler MCP Server config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "TheCrawler": {
      "command": "node",
      "args": [
        "dist/cli.js",
        "crawl",
        "https://example.com",
        "--markdown"
      ]
    }
  }
}

TheCrawler MCP Server tools

Tool list not available yet

This listing does not currently have a tool list to display. This does not mean the server has no tools; check its documentation for supported capabilities.

We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.

Overview

What is TheCrawler?

Open-source web scraper + LLM-powered structured extraction. PDF/DOCX, markdown, JSON-LD, microdata, commerce data, forms, 16 analytics-tracker detection. Structured errors with retryable flags. Adaptive Cheerio->Playwright. CLI, npm, REST API, and MCP server. AGPL-3.0.

How to use TheCrawler?

The README includes setup instructions such as node dist/cli.js crawl https://example.com --markdown.

Key features of TheCrawler

  • Scope: up to 25 public URLs and one target output shape
  • Payment: requested only after the workflow looks like a fit, by one-off $500 payment link or invoice
  • Output: a readiness report with ready, mixed, blocked, or not-worth-automating-yet guidance
  • Credit: if the workflow continues into setup or hosted usage, the $500 is credited toward that next step
  • Crawl mode: $0.005 per page successfully scraped (failed pages don't charge)

Use cases of TheCrawler

  • Connect an MCP-compatible client to this repository's service.
  • Review the README-backed setup before enabling it in production.

Frequently asked questions

1

Where is the source code for TheCrawler?

The source code is linked from the repository URL on this page.

2

Does TheCrawler include a standard MCP config?

If the README contains a parseable MCP configuration block, it is shown in the Config tab.

Comments

More Browser Automation MCP servers