广告合作
MCP.so
登录

MCP Evals

@mclenhard

代码仓库主页

基本信息

分类

其他

许可证

MIT

传输方式

stdio

作者

mclenhard

关于 MCP Evals

A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring. This helps ensure your MCP server's tools are working correctly and performing well.

#ai#mcp#evals

MCP Evals 配置

使用下面的配置,将此服务器添加到你的 MCP 客户端。

{
  "mcpServers": {
    "mcp-evals": {
      "command": "npx",
      "args": [
        "mcp-eval",
        "path/to/your/evals.ts",
        "path/to/your/server.ts"
      ]
    }
  }
}

MCP Evals 工具

工具列表暂未提供

当前收录暂时没有可展示的工具列表，不代表服务没有工具。请查阅项目文档了解支持的能力。

点击上方按钮,在临时沙箱中运行此服务器并实时获取工具列表。

概览

What is MCP Evals?

MCP Evals is a Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring with built-in observability support. Designed for developers building MCP servers, it helps ensure tools work correctly, perform well, and are fully observable with integrated monitoring and metrics.

How to use MCP Evals?

Install via npm install mcp-evals and run evaluations using the CLI: npx mcp-evals <evals-file> <server-file>. Alternatively, add the GitHub Action to your workflow with config keys like evals_path, server_path, openai_api_key, and model. Create evaluation files in TypeScript (.ts) or YAML (.yaml/.yml) and set environment variables OPENAI_API_KEY or ANTHROPIC_API_KEY as needed.

Key features of MCP Evals

  • LLM-based scoring on five dimensions (accuracy, completeness, relevance, clarity, reasoning)
  • Built-in observability with metrics and tracing
  • GitHub Action integration for PR-based evaluation
  • Support for TypeScript and YAML evaluation configurations
  • Metrics: tool call counts, errors, and latency distributions
  • Monitoring stack with Prometheus, Grafana, and Jaeger dashboards

Use cases of MCP Evals

  • Automatically evaluate MCP tool responses in a CI/CD pipeline on pull requests
  • Locally test and score tool accuracy, completeness, and clarity during development
  • Monitor production MCP servers for tool errors and latency
  • Compare different model providers (OpenAI, Anthropic) for evaluation consistency

常见问题

1

What model providers are supported?

OpenAI and Anthropic are supported; set the provider and model name in your YAML config or pass a model instance in TypeScript.

2

How do I run evaluations without a GitHub Action?

Use the CLI: `npx mcp-evals path/to/evals.ts path/to/server.ts` (or `.yaml`). The CLI runs evaluations locally and prints results.

3

What metrics does the monitoring feature provide?

Metrics include tool call counts by tool name, tool error counts by tool name, and tool latency distributions.

4

How do I set up the monitoring stack?

Run `docker-compose up -d` after adding `metrics.initialize(port, options)` to your application. Access Prometheus on port 9090, Grafana on port 3000, and Jaeger on port 16686.

5

Can I use YAML instead of TypeScript for evaluation files?

Yes. Create a `.yaml` or `.yml` file with `model` and `evals` sections. Supported file extensions are `.yaml` and `.yml`.

评论

其他 分类下的更多 MCP 服务器

Klarix Intelligence Engine

Klarix

B2B competitive intelligence, live sales battlecards, 5-axis ICP prospect fit scoring, and tech stack teardowns over MCP and REST with cited public-web evidence.

Blender

ahujasid

Open-source MCP to use Blender with any LLM

23.6K

Yuppsy

Yuppsy MCP: build and publish real estate websites from your AI assistant. Yuppsy lets your AI assistant create live websites for real estate agents and individual property listings. You don't need a designer or any code

Senso MCP for Joomla

sensomedia

Senso MCP turns a Joomla 5 or 6 site into a remote MCP server. It runs inside Joomla itself (no extra server, no Node.js bridge) and is secured with OAuth 2.1 and Joomla's own ACL. The free pack lets an AI assistant crea

MCP Toolbox for Databases

googleapis

MCP Toolbox for Databases is an open source MCP server for databases.

15.9K
广告合作