🦊 MCPBench: A Benchmark for Evaluating MCP Servers

What is 🦊 MCPBench: A Benchmark for Evaluating MCP Servers?

MCPBench is an evaluation framework for MCP servers. It supports evaluating three types of servers (Web Search, Database Query, and GAIA) and is compatible with both local and remote MCP servers. The framework measures task completion accuracy, latency, and token consumption under consistent LLM and Agent configurations.

How to use 🦊 MCPBench: A Benchmark for Evaluating MCP Servers?

Install Python ≥3.11, nodejs, and jq, then run pip install -r requirements.txt. Write a configuration file specifying the MCP server (remote SSE URL or local stdio with run_config). Optionally launch local servers with sh launch_mcps_as_sse.sh <config>. Then evaluate with sh evaluation_websearch.sh <config>, sh evaluation_db.sh <config>, or sh evaluation_gaia.sh <config>. The tool configuration is auto-detected.

Key features of 🦊 MCPBench: A Benchmark for Evaluating MCP Servers

• Evaluates Web Search, Database Query, and GAIA server types.
• Works with both remote (SSE) and local (STDIO) MCP servers.
• Measures accuracy, latency, and token consumption.
• Provides built-in datasets for WebSearch and Database Query.
• Automatically detects server tools and parameters.
• Compatible with multiple MCP servers (e.g., Brave Search, DuckDuckGo).

Use cases of 🦊 MCPBench: A Benchmark for Evaluating MCP Servers

• Benchmark Web Search MCP servers under the same LLM and agent.
• Compare Database Query MCP servers on car-related questions.
• Evaluate GAIA task performance of MCP servers.
• Assess latency and token efficiency of different MCP implementations.
• Validate custom MCP servers using the provided evaluation datasets.

FAQ from 🦊 MCPBench: A Benchmark for Evaluating MCP Servers

What MCP server types are supported?

Web Search, Database Query, and GAIA.

What metrics are measured?

Task completion accuracy, latency, and token consumption.

What are the runtime requirements?

Python ≥3