Advertise
MCP.so
Sign In

MCP Evals

@mclenhard

RepositoryHomepage

Basic information

Category

Other

License

MIT

Transports

stdio

Author

mclenhard

About MCP Evals

A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring. This helps ensure your MCP server's tools are working correctly and performing well.

#ai#mcp#evals

MCP Evals config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "mcp-evals": {
      "command": "npx",
      "args": [
        "mcp-eval",
        "path/to/your/evals.ts",
        "path/to/your/server.ts"
      ]
    }
  }
}

MCP Evals tools

Tool list not available yet

This listing does not currently have a tool list to display. This does not mean the server has no tools; check its documentation for supported capabilities.

Fetch the live tool list by running this server in a temporary sandbox using the button above.

Overview

What is MCP Evals?

MCP Evals is a Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring with built-in observability support. Designed for developers building MCP servers, it helps ensure tools work correctly, perform well, and are fully observable with integrated monitoring and metrics.

How to use MCP Evals?

Install via npm install mcp-evals and run evaluations using the CLI: npx mcp-evals <evals-file> <server-file>. Alternatively, add the GitHub Action to your workflow with config keys like evals_path, server_path, openai_api_key, and model. Create evaluation files in TypeScript (.ts) or YAML (.yaml/.yml) and set environment variables OPENAI_API_KEY or ANTHROPIC_API_KEY as needed.

Key features of MCP Evals

  • LLM-based scoring on five dimensions (accuracy, completeness, relevance, clarity, reasoning)
  • Built-in observability with metrics and tracing
  • GitHub Action integration for PR-based evaluation
  • Support for TypeScript and YAML evaluation configurations
  • Metrics: tool call counts, errors, and latency distributions
  • Monitoring stack with Prometheus, Grafana, and Jaeger dashboards

Use cases of MCP Evals

  • Automatically evaluate MCP tool responses in a CI/CD pipeline on pull requests
  • Locally test and score tool accuracy, completeness, and clarity during development
  • Monitor production MCP servers for tool errors and latency
  • Compare different model providers (OpenAI, Anthropic) for evaluation consistency

Frequently asked questions

1

What model providers are supported?

OpenAI and Anthropic are supported; set the provider and model name in your YAML config or pass a model instance in TypeScript.

2

How do I run evaluations without a GitHub Action?

Use the CLI: `npx mcp-evals path/to/evals.ts path/to/server.ts` (or `.yaml`). The CLI runs evaluations locally and prints results.

3

What metrics does the monitoring feature provide?

Metrics include tool call counts by tool name, tool error counts by tool name, and tool latency distributions.

4

How do I set up the monitoring stack?

Run `docker-compose up -d` after adding `metrics.initialize(port, options)` to your application. Access Prometheus on port 9090, Grafana on port 3000, and Jaeger on port 16686.

5

Can I use YAML instead of TypeScript for evaluation files?

Yes. Create a `.yaml` or `.yml` file with `model` and `evals` sections. Supported file extensions are `.yaml` and `.yml`.

Comments

More Other MCP servers

P

PickFu MCP

PickFu MCP is a Model Context Protocol (MCP) server that connects AI assistants, developers, marketers, product managers, and AI agents to PickFu's consumer research platform. It enables users to create surveys, conduct

Maestro

mobile-dev-inc

Painless E2E Automation for Mobile and Web

14.8K

Screenpipe

screenpipe

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

22.0K

MaxKB

1Panel-dev

🔥 MaxKB is an open-source platform for building enterprise-grade agents. 强大易用的开源企业级智能体平台。

22.0K

Yuppsy

Yuppsy MCP: build and publish real estate websites from your AI assistant. Yuppsy lets your AI assistant create live websites for real estate agents and individual property listings. You don't need a designer or any code

Advertise