广告合作
MCP.so
登录

Crawl Readiness MCP Server已验证已精选

@Grubbomatic

代码仓库主页文档

基本信息

分类

开发工具

许可证

MIT

作者

Grubbomatic

提交者

Grubbomatic

关于 Crawl Readiness MCP Server

面向 MCP 客户端的 AI SEO 工具——审查 AI 爬虫能否读取网站，校验 JSON-LD 和 robots.txt，并生成 llms.txt、面向 AI 爬虫的 robots.txt 和 JSON-LD schema，无需离开对话。

#ai-seo#llms.txt#robots.txt#schema.org#chatgpt#claude#perplexity#crawler#ai-readiness#json-ld

Crawl Readiness MCP Server 配置

使用下面的配置,将此服务器添加到你的 MCP 客户端。

{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": [
        "-y",
        "crawl-readiness-mcp"
      ]
    }
  }
}

Crawl Readiness MCP Server 工具

8

Check whether AI crawlers (ChatGPT, Claude, Perplexity, Google AI, and 50+ others) can access a website. Returns a 0-100 AI readiness score, per-crawler access status, detected AI-specific files (llms.txt, agents.json), structured data presence, meta signals, and a prioritized fix list. The score counts the crawlers that feed AI search and answers: each entry in `robots.results` carries its `purpose` and whether it is `scored`, and training-only crawlers are listed but never counted, so blocking them does not lower the score. Fixes marked `extra: true` (llms.txt, agents.json, Content-Signal, a markdown copy, agent-ready forms) are informational 'Going further' items and do not affect the score; present them as optional. Use this as the first step in any AI SEO audit.

Validate all JSON-LD structured data on a URL. Extracts every <script type="application/ld+json"> block, runs each through a rules engine covering 20+ common types (Article, Organization, Product, LocalBusiness, FAQPage, Recipe, Event, etc.), and reports required-field errors, recommended-field warnings, and type-specific gotchas.

Audit a robots.txt file line-by-line. Detects syntax errors, empty user-agent groups, orphan Allow/Disallow lines, non-slash paths, non-numeric Crawl-delay values, unofficial Noindex usage, and wildcard traps. Also summarizes AI-bot coverage across 50+ known AI crawlers. Provide EITHER url (to fetch and audit) OR text.

Compare what human browsers see vs what AI crawlers see. Fetches a page four times in parallel — as Chrome, GPTBot, ClaudeBot, and PerplexityBot — and reports word-overlap %, title/description differences, and warnings about JS-only shells, cloaking, or edge-based bot blocking.

Generate a properly formatted llms.txt file for a website. Crawls the site's sitemap, groups pages by section, pulls page titles and descriptions, and produces both the llms.txt content and a companion robots.txt snippet. The AI client can then write the returned content to disk in the user's project. Requires an API key.

Generate a complete, ready-to-save AI-crawler-aware robots.txt for a website. Fetches the existing robots.txt (if any) and returns the finished file in `generated.robotsTxt` — the existing rules with an AI-crawler policy section merged in — plus the per-bot allow/block breakdown. Presets: 'allow-all' (every AI crawler allowed), 'recommended' (blocks only the training-only crawlers; everything that feeds AI search and answers stays allowed), 'search-only' (allows only the crawlers that feed AI search and answers, the ones the readiness score counts, and blocks the rest), 'block-all'. A file from 'recommended' or 'search-only' loses no points for crawler access. Write `generated.robotsTxt` to the site's /robots.txt. Requires an API key.

Generate JSON-LD structured data for a website. Detects the site's name, logo, social profiles, contact info, and article metadata, then produces Organization, WebSite, and (when applicable) Article schemas plus starter templates for BreadcrumbList and FAQPage. Returns each schema as a <script type="application/ld+json"> block ready to paste into the site's <head>. Requires an API key.

See whether AI assistants (ChatGPT, Claude, Perplexity, Google AI) actually mention a brand in their answers, and how that share-of-voice is trending versus competitors, week over week. Call with NO argument to list the user's monitored brands with each one's current mention rate and direction; pass a brand name or project id to get that brand's full trend, per-provider breakdown, competitor comparison, average position, and short example answers. Reads data the user's LLM Monitor projects have already collected — it does not trigger new runs. Read-only. Requires an API key.

概览

AI SEO tools inside Claude, Cursor, Cline, and other MCP-compatible clients.

Ask your AI "is my site AI-readable?" and it actually checks — running a live audit against 50+ AI crawlers (ChatGPT, Claude, Perplexity, Google AI). Then ask it to fix what's broken, and it can generate a proper llms.txt, robots.txt, or JSON-LD schema and write it into your project.

Install (Claude Desktop)

{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": ["-y", "crawl-readiness-mcp"]
    }
  }
}

Same shape works for Cursor, Cline, Windsurf, and Continue.

Tools

Audit (no signup):

  • check_ai_readiness — 0–100 score against 50+ AI crawlers with a fix list
  • validate_schema — JSON-LD audit (20+ types)
  • validate_robots — line-by-line robots.txt audit
  • check_content_parity — human vs AI-crawler view diff

Generate (free API key at crawlreadiness.com/dashboard):

  • generate_llms_txt — auto-crawl a site and produce llms.txt
  • generate_robots_txt — AI-crawler-aware robots.txt with presets
  • generate_schema — Organization / WebSite / Article JSON-LD

MIT licensed. Full docs at crawlreadiness.com/mcp.

评论

开发工具 分类下的更多 MCP 服务器

Capgo MCP

Cap-go

Console, Backend and CLI to manage Capgo Instant update and Native build for Capacitor apps Build and upload to app store your mobile app

Banking Assistant (Metabind demo)

Metabind

Interactive MCP App demo built on Metabind: personal-finance answers render as UI cards (net worth, spending, subscriptions) in Claude and ChatGPT, and natively in iOS/Android apps.

LocalCan

LocalCan

Gives AI agents public URLs (tunnels) for localhost, live HTTP traffic inspection, snapshot publishing, and access control. Part of LocalCan, the ngrok alternative for Mac, Windows and Linux. Free plan.

82

Mcp-Warden

ermoluk

Local-first credential broker for MCP. AI coding agents (Claude Code, Cursor, Codex…) call APIs, SSH and databases through scoped presets; secrets stay in your OS keychain and never enter the agent's context.

4

Vibgrate MCP

Vibgrate

Query your team's drift, vulnerability, and migration data from any AI assistant. Vibgrate MCP connects Cursor, Claude, ChatGPT, Windsurf, or VS Code to Vibgrate Cloud: 51 tools for DriftScores, CVEs and EOL runtimes, up

3
广告合作