広告掲載
MCP.so
ログイン

MCP Evals

@mclenhard

リポジトリホームページ

基本情報

カテゴリ

その他

ライセンス

MIT

トランスポート

stdio

作者

mclenhard

MCP Evals について

A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring. This helps ensure your MCP server's tools are working correctly and performing well.

#ai#mcp#evals

MCP Evals の設定

以下の設定を使って、このサーバーを MCP 対応クライアントに追加してください。

{
  "mcpServers": {
    "mcp-evals": {
      "command": "npx",
      "args": [
        "mcp-eval",
        "path/to/your/evals.ts",
        "path/to/your/server.ts"
      ]
    }
  }
}

MCP Evals のツール

ツール一覧はまだありません

この掲載には表示できるツール一覧がまだありません。サーバーにツールがないという意味ではありません。対応機能はプロジェクトのドキュメントをご確認ください。

上のボタンで一時的なサンドボックスでこのサーバーを実行し、ツール一覧を取得できます。

概要

What is MCP Evals?

MCP Evals is a Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring with built-in observability support. Designed for developers building MCP servers, it helps ensure tools work correctly, perform well, and are fully observable with integrated monitoring and metrics.

How to use MCP Evals?

Install via npm install mcp-evals and run evaluations using the CLI: npx mcp-evals <evals-file> <server-file>. Alternatively, add the GitHub Action to your workflow with config keys like evals_path, server_path, openai_api_key, and model. Create evaluation files in TypeScript (.ts) or YAML (.yaml/.yml) and set environment variables OPENAI_API_KEY or ANTHROPIC_API_KEY as needed.

Key features of MCP Evals

  • LLM-based scoring on five dimensions (accuracy, completeness, relevance, clarity, reasoning)
  • Built-in observability with metrics and tracing
  • GitHub Action integration for PR-based evaluation
  • Support for TypeScript and YAML evaluation configurations
  • Metrics: tool call counts, errors, and latency distributions
  • Monitoring stack with Prometheus, Grafana, and Jaeger dashboards

Use cases of MCP Evals

  • Automatically evaluate MCP tool responses in a CI/CD pipeline on pull requests
  • Locally test and score tool accuracy, completeness, and clarity during development
  • Monitor production MCP servers for tool errors and latency
  • Compare different model providers (OpenAI, Anthropic) for evaluation consistency

よくある質問

1

What model providers are supported?

OpenAI and Anthropic are supported; set the provider and model name in your YAML config or pass a model instance in TypeScript.

2

How do I run evaluations without a GitHub Action?

Use the CLI: `npx mcp-evals path/to/evals.ts path/to/server.ts` (or `.yaml`). The CLI runs evaluations locally and prints results.

3

What metrics does the monitoring feature provide?

Metrics include tool call counts by tool name, tool error counts by tool name, and tool latency distributions.

4

How do I set up the monitoring stack?

Run `docker-compose up -d` after adding `metrics.initialize(port, options)` to your application. Access Prometheus on port 9090, Grafana on port 3000, and Jaeger on port 16686.

5

Can I use YAML instead of TypeScript for evaluation files?

Yes. Create a `.yaml` or `.yml` file with `model` and `evals` sections. Supported file extensions are `.yaml` and `.yml`.

コメント

「その他」の他のコンテンツ

Screenpipe

screenpipe

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

22.0K

MaxKB

1Panel-dev

🔥 MaxKB is an open-source platform for building enterprise-grade agents. 强大易用的开源企业级智能体平台。

22.0K

Blender

ahujasid

Open-source MCP to use Blender with any LLM

23.6K

Volley.sh

Volley

A good sequence diagram can solve almost any communication problem in engineering. Volley lets your agent write them as plain text, with the real request and response payloads on every arrow. It checks its own syntax wi

Handl

Handl speaks the Model Context Protocol. Your whole billing operation — projects, invoices, clients, reports, approvals — as tools any AI assistant can call, inside your autonomy rules.

広告掲載