Mcp Autotest

Name: Mcp Autotest
Author: strowk

@strowk

Repository

About Mcp Autotest

Utility for autotesting MCP servers

Basic information

Config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "mcp-autotest": {
      "command": "npx",
      "args": [
        "mcp-autotest",
        "run",
        "testdata",
        "--",
        "npx",
        "-y",
        "@modelcontextprotocol/server-postgres",
        "localhost:5432"
      ]
    }
  }
}

Tools

No tools detected

We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.

Overview

What is Mcp Autotest?

Mcp Autotest is a simple tool for testing MCP servers in a language-agnostic way. It uses MCP protocol by defining YAML files with requests and expected responses.

How to use Mcp Autotest?

Install via npm (npm install -g mcp-autotest), download prebuilt binaries from GitHub Releases, or build from source with Go. Run mcp-autotest [flags] run path/to/tests/folder [--] command-to-run-server [server-args]. Test case files must be multi-document YAML with _test.yaml suffix. By default stdio transport is used; for Streamable HTTP (POST only) add --url http://host:port/path.

Key features of Mcp Autotest

Language-agnostic MCP server testing via YAML definitions
Supports both stdio and Streamable HTTP transports
Dynamic matching with !!re tag for regular expressions
Embedded regex (!!ere) for partial string matching
Escaping forward slashes in embedded regex strings

Use cases of Mcp Autotest

Validate an MCP server’s responses against expected JSON-RPC messages
Automate regression testing for MCP server implementations
Test servers across different languages without writing test code
Verify dynamic responses (timestamps, UUIDs) using regex matching

FAQ from Mcp Autotest

How do I install Mcp Autotest?

Install globally via npm (npm install -g mcp-autotest), download a prebuilt binary from GitHub Releases, or build from source with go install github.com/strowk/mcp-autotest@latest.

What file naming convention is required for test cases?

Test case files must end with _test.yaml to be recognized.

Can I test an MCP server running over HTTP?

Yes, use the --url flag to specify the server URL. Only the POST method is currently supported for Streamable HTTP; GET support is planned.

How do I match dynamic values like timestamps in responses?

Use the !!re tag to treat a string as a regular expression for full match, or !!ere for embedded regex (parts between slashes / are treated as regex).