Overview
What is SEO MCP Tools?
SEO MCP Tools is a Model Context Protocol (MCP) server that provides tools for analyzing robots.txt files. It extracts sitemap URLs from robots.txt files across the web, intended for SEO professionals and developers.
How to use SEO MCP Tools?
Install dependencies with pip install fastmcp advertools requests pandas, then start the server with python mcp_server.py or fastmcp dev mcp_server.py for interactive testing. Use an MCP client to call the tools get_sitemap_url_from_robots_txt or get_all_sitemap_urls_from_robots_txt, providing a robots_txt_url parameter.
Key features of SEO MCP Tools
- Extract the first sitemap URL from a robots.txt file.
- Extract all sitemap URLs from a robots.txt file.
- Uses direct HTTP requests and regex for reliable parsing.
- Supports integration with Claude Desktop via stdio transport.
- Test suite validates against real websites' robots.txt files.
Use cases of SEO MCP Tools
- Quickly find a website’s primary sitemap for crawling or indexing.
- Audit all sitemaps listed in a site’s robots.txt for coverage analysis.
- Integrate sitemap discovery into automated SEO workflows.
- Validate robots.txt sitemap directives across multiple domains.
FAQ from SEO MCP Tools
What tools does SEO MCP Tools provide?
Two tools: get_sitemap_url_from_robots_txt to get the first sitemap URL, and get_all_sitemap_urls_from_robots_txt to get all sitemap URLs from a given robots.txt file.
What are the dependencies to run SEO MCP Tools?
Python 3.8+ and the packages FastMCP, advertools, requests, and pandas.
How do I run SEO MCP Tools?
Start the MCP server with python mcp_server.py or use fastmcp dev mcp_server.py for development/testing.
Can SEO MCP Tools be used with Claude Desktop?
Yes, create a minimal MCP configuration file for stdio transport or run fastmcp install mcp_server.py to install in Claude Desktop.
How does SEO MCP Tools parse robots.txt files?
It uses direct HTTP requests to fetch the robots.txt file, then applies regular expression patterns to extract sitemap directives.
