Where is the source code for TheCrawler?
The source code is linked from the repository URL on this page.
@manchittlab
Basic information
Open-source web scraper + LLM-powered structured extraction. PDF/DOCX, markdown, JSON-LD, microdata, commerce data, forms, 16 analytics-tracker detection. Structured errors with retryable flags. Adaptive Cheerio->Playwright. CLI, npm, REST API, and MCP server. AGPL-3.0.
Add this server to your MCP-compatible client using the configuration below.
{
"mcpServers": {
"TheCrawler": {
"command": "node",
"args": [
"dist/cli.js",
"crawl",
"https://example.com",
"--markdown"
]
}
}
}Tool list not available yet
This listing does not currently have a tool list to display. This does not mean the server has no tools; check its documentation for supported capabilities.
We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.
Open-source web scraper + LLM-powered structured extraction. PDF/DOCX, markdown, JSON-LD, microdata, commerce data, forms, 16 analytics-tracker detection. Structured errors with retryable flags. Adaptive Cheerio->Playwright. CLI, npm, REST API, and MCP server. AGPL-3.0.
The README includes setup instructions such as node dist/cli.js crawl https://example.com --markdown.
The source code is linked from the repository URL on this page.
If the README contains a parseable MCP configuration block, it is shown in the Config tab.
Allow LLMs to control a browser with Browserbase and Stagehand
MCP server for fetch web page content using Playwright headless browser.
Give your AI agent the live web. One MCP server for Search, Fetch, Browser, and Agent: browse real sites, extract structured data, fill forms, and run multi-step workflows. Search and Fetch are free.

Web automation for AI assistants. Ask for a task on a website in plain English and Node Mantis carries it out in a real browser, then hands back the result as structured data.
🔮 Platform-neutral Browser Use for AI agents: real Chrome automation with a CLI + SDKs, no lock-in, dead simple.
Comments