广告合作
MCP.so
登录

MCP Evals

@mclenhard

代码仓库主页

基本信息

分类

其他

许可证

MIT

传输方式

stdio

作者

mclenhard

关于 MCP Evals

A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring. This helps ensure your MCP server's tools are working correctly and performing well.

#ai#mcp#evals

MCP Evals 配置

使用下面的配置,将此服务器添加到你的 MCP 客户端。

{
  "mcpServers": {
    "mcp-evals": {
      "command": "npx",
      "args": [
        "mcp-eval",
        "path/to/your/evals.ts",
        "path/to/your/server.ts"
      ]
    }
  }
}

MCP Evals 工具

工具列表暂未提供

当前收录暂时没有可展示的工具列表，不代表服务没有工具。请查阅项目文档了解支持的能力。

点击上方按钮,在临时沙箱中运行此服务器并实时获取工具列表。

概览

What is MCP Evals?

MCP Evals is a Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring with built-in observability support. Designed for developers building MCP servers, it helps ensure tools work correctly, perform well, and are fully observable with integrated monitoring and metrics.

How to use MCP Evals?

Install via npm install mcp-evals and run evaluations using the CLI: npx mcp-evals <evals-file> <server-file>. Alternatively, add the GitHub Action to your workflow with config keys like evals_path, server_path, openai_api_key, and model. Create evaluation files in TypeScript (.ts) or YAML (.yaml/.yml) and set environment variables OPENAI_API_KEY or ANTHROPIC_API_KEY as needed.

Key features of MCP Evals

  • LLM-based scoring on five dimensions (accuracy, completeness, relevance, clarity, reasoning)
  • Built-in observability with metrics and tracing
  • GitHub Action integration for PR-based evaluation
  • Support for TypeScript and YAML evaluation configurations
  • Metrics: tool call counts, errors, and latency distributions
  • Monitoring stack with Prometheus, Grafana, and Jaeger dashboards

Use cases of MCP Evals

  • Automatically evaluate MCP tool responses in a CI/CD pipeline on pull requests
  • Locally test and score tool accuracy, completeness, and clarity during development
  • Monitor production MCP servers for tool errors and latency
  • Compare different model providers (OpenAI, Anthropic) for evaluation consistency

常见问题

1

What model providers are supported?

OpenAI and Anthropic are supported; set the provider and model name in your YAML config or pass a model instance in TypeScript.

2

How do I run evaluations without a GitHub Action?

Use the CLI: `npx mcp-evals path/to/evals.ts path/to/server.ts` (or `.yaml`). The CLI runs evaluations locally and prints results.

3

What metrics does the monitoring feature provide?

Metrics include tool call counts by tool name, tool error counts by tool name, and tool latency distributions.

4

How do I set up the monitoring stack?

Run `docker-compose up -d` after adding `metrics.initialize(port, options)` to your application. Access Prometheus on port 9090, Grafana on port 3000, and Jaeger on port 16686.

5

Can I use YAML instead of TypeScript for evaluation files?

Yes. Create a `.yaml` or `.yml` file with `model` and `evals` sections. Supported file extensions are `.yaml` and `.yml`.

评论

其他 分类下的更多 MCP 服务器

Reactive Resume

amruthpillai

A one-of-a-kind resume builder that keeps your privacy in mind. Completely secure, customizable, portable, open-source and free forever. Try it out today!

39.5K

Klarix Intelligence Engine

Klarix

B2B competitive intelligence, live sales battlecards, 5-axis ICP prospect fit scoring, and tech stack teardowns over MCP and REST with cited public-web evidence.

P

PickFu MCP

PickFu MCP is a Model Context Protocol (MCP) server that connects AI assistants, developers, marketers, product managers, and AI agents to PickFu's consumer research platform. It enables users to create surveys, conduct

Sequential Thinking

modelcontextprotocol

Model Context Protocol Servers

88.2K

🚀 Model Context Protocol (MCP) Curriculum for Beginners

microsoft

This open-source curriculum introduces the fundamentals of Model Context Protocol (MCP) through real-world, cross-language examples in .NET, Java, TypeScript, JavaScript, Rust and Python. Designed for developers, it focuses on practical techniques for building modular, scalable,

16.7K
广告合作