Advertise
MCP.so
Sign In

Crawl Readiness MCP ServerVerifiedFeatured

@Grubbomatic

RepositoryHomepageDocumentation

Basic information

Category

Developer Tools

Last updated

License

MIT

Author

Grubbomatic

Submitted by

Grubbomatic

About Crawl Readiness MCP Server

AI SEO tools for MCP clients — audit whether AI crawlers can read a website, validate JSON-LD and robots.txt, and generate llms.txt, AI-crawler-aware robots.txt, and JSON-LD schema without leaving the conversation.

#ai-seo#llms.txt#robots.txt#schema.org#chatgpt#claude#perplexity#crawler#ai-readiness#json-ld

Crawl Readiness MCP Server config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": [
        "-y",
        "crawl-readiness-mcp"
      ]
    }
  }
}

Crawl Readiness MCP Server tools

8

Check whether AI crawlers (ChatGPT, Claude, Perplexity, Google AI, and 50+ others) can access a website. Returns a 0-100 AI readiness score, per-crawler access status, detected AI-specific files (llms.txt, agents.json), structured data presence, meta signals, and a prioritized fix list. The score counts the crawlers that feed AI search and answers: each entry in `robots.results` carries its `purpose` and whether it is `scored`, and training-only crawlers are listed but never counted, so blocking them does not lower the score. Fixes marked `extra: true` (llms.txt, agents.json, Content-Signal, a markdown copy, agent-ready forms) are informational 'Going further' items and do not affect the score; present them as optional. Use this as the first step in any AI SEO audit.

Validate all JSON-LD structured data on a URL. Extracts every <script type="application/ld+json"> block, runs each through a rules engine covering 20+ common types (Article, Organization, Product, LocalBusiness, FAQPage, Recipe, Event, etc.), and reports required-field errors, recommended-field warnings, and type-specific gotchas.

Audit a robots.txt file line-by-line. Detects syntax errors, empty user-agent groups, orphan Allow/Disallow lines, non-slash paths, non-numeric Crawl-delay values, unofficial Noindex usage, and wildcard traps. Also summarizes AI-bot coverage across 50+ known AI crawlers. Provide EITHER url (to fetch and audit) OR text.

Compare what human browsers see vs what AI crawlers see. Fetches a page four times in parallel — as Chrome, GPTBot, ClaudeBot, and PerplexityBot — and reports word-overlap %, title/description differences, and warnings about JS-only shells, cloaking, or edge-based bot blocking.

Generate a properly formatted llms.txt file for a website. Crawls the site's sitemap, groups pages by section, pulls page titles and descriptions, and produces both the llms.txt content and a companion robots.txt snippet. The AI client can then write the returned content to disk in the user's project. Requires an API key.

Generate a complete, ready-to-save AI-crawler-aware robots.txt for a website. Fetches the existing robots.txt (if any) and returns the finished file in `generated.robotsTxt` — the existing rules with an AI-crawler policy section merged in — plus the per-bot allow/block breakdown. Presets: 'allow-all' (every AI crawler allowed), 'recommended' (blocks only the training-only crawlers; everything that feeds AI search and answers stays allowed), 'search-only' (allows only the crawlers that feed AI search and answers, the ones the readiness score counts, and blocks the rest), 'block-all'. A file from 'recommended' or 'search-only' loses no points for crawler access. Write `generated.robotsTxt` to the site's /robots.txt. Requires an API key.

Generate JSON-LD structured data for a website. Detects the site's name, logo, social profiles, contact info, and article metadata, then produces Organization, WebSite, and (when applicable) Article schemas plus starter templates for BreadcrumbList and FAQPage. Returns each schema as a <script type="application/ld+json"> block ready to paste into the site's <head>. Requires an API key.

See whether AI assistants (ChatGPT, Claude, Perplexity, Google AI) actually mention a brand in their answers, and how that share-of-voice is trending versus competitors, week over week. Call with NO argument to list the user's monitored brands with each one's current mention rate and direction; pass a brand name or project id to get that brand's full trend, per-provider breakdown, competitor comparison, average position, and short example answers. Reads data the user's LLM Monitor projects have already collected — it does not trigger new runs. Read-only. Requires an API key.

Overview

AI SEO tools inside Claude, Cursor, Cline, and other MCP-compatible clients.

Ask your AI "is my site AI-readable?" and it actually checks — running a live audit against 50+ AI crawlers (ChatGPT, Claude, Perplexity, Google AI). Then ask it to fix what's broken, and it can generate a proper llms.txt, robots.txt, or JSON-LD schema and write it into your project.

Install (Claude Desktop)

{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": ["-y", "crawl-readiness-mcp"]
    }
  }
}

Same shape works for Cursor, Cline, Windsurf, and Continue.

Tools

Audit (no signup):

  • check_ai_readiness — 0–100 score against 50+ AI crawlers with a fix list
  • validate_schema — JSON-LD audit (20+ types)
  • validate_robots — line-by-line robots.txt audit
  • check_content_parity — human vs AI-crawler view diff

Generate (free API key at crawlreadiness.com/dashboard):

  • generate_llms_txt — auto-crawl a site and produce llms.txt
  • generate_robots_txt — AI-crawler-aware robots.txt with presets
  • generate_schema — Organization / WebSite / Article JSON-LD

MIT licensed. Full docs at crawlreadiness.com/mcp.

Comments

More Developer Tools MCP servers

Mcp-Warden

ermoluk

Local-first credential broker for MCP. AI coding agents (Claude Code, Cursor, Codex…) call APIs, SSH and databases through scoped presets; secrets stay in your OS keychain and never enter the agent's context.

4

offprompt

offprompt

MCP server that lets Claude Code, Cursor or Codex ask you for API keys on a local page and write them to .env. The agent only ever sees key names.

2

Faxbeep

faxbeep

Faxbeep MCP server: test fax numbers for AI agents. Send a test fax or scan-to-email, then read what arrived. Read-only, no account.

Mokxi

Build Arduino, ESP32 and Pico circuits with code from your AI assistant. Mokxi checks the circuit and hands back a link that opens and runs it in the browser simulator.

Vibgrate MCP

Vibgrate

Query your team's drift, vulnerability, and migration data from any AI assistant. Vibgrate MCP connects Cursor, Claude, ChatGPT, Windsurf, or VS Code to Vibgrate Cloud: 51 tools for DriftScores, CVEs and EOL runtimes, up

3
Advertise