Overview
What is MCP Evals?
MCP Evals is a Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring with built-in observability support. Designed for developers building MCP servers, it helps ensure tools work correctly, perform well, and are fully observable with integrated monitoring and metrics.
How to use MCP Evals?
Install via npm install mcp-evals and run evaluations using the CLI: npx mcp-evals <evals-file> <server-file>. Alternatively, add the GitHub Action to your workflow with config keys like evals_path, server_path, openai_api_key, and model. Create evaluation files in TypeScript (.ts) or YAML (.yaml/.yml) and set environment variables OPENAI_API_KEY or ANTHROPIC_API_KEY as needed.
Key features of MCP Evals
- LLM-based scoring on five dimensions (accuracy, completeness, relevance, clarity, reasoning)
- Built-in observability with metrics and tracing
- GitHub Action integration for PR-based evaluation
- Support for TypeScript and YAML evaluation configurations
- Metrics: tool call counts, errors, and latency distributions
- Monitoring stack with Prometheus, Grafana, and Jaeger dashboards
Use cases of MCP Evals
- Automatically evaluate MCP tool responses in a CI/CD pipeline on pull requests
- Locally test and score tool accuracy, completeness, and clarity during development
- Monitor production MCP servers for tool errors and latency
- Compare different model providers (OpenAI, Anthropic) for evaluation consistency
FAQ from MCP Evals
What model providers are supported?
OpenAI and Anthropic are supported; set the provider and model name in your YAML config or pass a model instance in TypeScript.
How do I run evaluations without a GitHub Action?
Use the CLI: npx mcp-evals path/to/evals.ts path/to/server.ts (or .yaml). The CLI runs evaluations locally and prints results.
What metrics does the monitoring feature provide?
Metrics include tool call counts by tool name, tool error counts by tool name, and tool latency distributions.
How do I set up the monitoring stack?
Run docker-compose up -d after adding metrics.initialize(port, options) to your application. Access Prometheus on port 9090, Grafana on port 3000, and Jaeger on port 16686.
Can I use YAML instead of TypeScript for evaluation files?
Yes. Create a .yaml or .yml file with model and evals sections. Supported file extensions are .yaml and .yml.
