MCP.so
登录
C

Conkurrence

@AlligatorC0der

代码仓库

关于 Conkurrence

Conkurrence measures whether multiple AI models produce consistent outputs on your evaluation tasks. It tells you which items your AI agrees on and which need human review — using Fleiss' κ, Kendall's W, and bootstrap confidence intervals, the same psychometric methods trusted in

#ai-evaluation#golden-dataset#llm-evaluation#testing#inter-rater-reliability#fleiss-kappa#kendall-w#validating-prompt-output#consensus#multi-model

基本信息

分类

其他

传输方式

stdio

发布者

AlligatorC0der

提交者

joe Etherage

配置

使用下面的配置,将此服务器添加到你的 MCP 客户端。

{
  "mcpServers": {
    "conkurrence": {
      "command": "npx",
      "args": [
        "-y",
        "conkurrence",
        "mcp"
      ]
    }
  }
}

工具

未检测到工具

工具是从 README 中自动提取的。维护者可以在 ## Tools 标题下列出工具,即可填充这部分内容。

概览

What is Conkurrence?

ConKurrence is a statistically validated consensus measurement toolkit for AI evaluation pipelines. It uses multiple AI models as independent raters, measures inter-rater reliability with Fleiss' kappa and bootstrap confidence intervals, and routes contested items to human experts.

How to use Conkurrence?

Install globally with npm install -g conkurrence, then run as an MCP server with npx conkurrence mcp. Add the corresponding configuration to your Claude Desktop or Claude Code plugin settings. Invoke the available MCP tools such as conkurrence_run, conkurrence_report, conkurrence_compare, conkurrence_trend, conkurrence_suggest, conkurrence_validate_schema, or conkurrence_estimate.

Key features of Conkurrence

  • Multi-model evaluation using Bedrock, OpenAI, and Gemini simultaneously
  • Fleiss' kappa with bootstrap confidence intervals and Kendall's W
  • Self-consistency mode (no API keys needed, uses MCP Sampling)
  • AI-powered schema suggestion from your data
  • Trend tracking to detect agreement degradation over time
  • Cost estimation before running an evaluation

Use cases of Conkurrence

  • Evaluate inter-rater reliability when multiple AI models judge the same data
  • Automatically route low-agreement items to human experts for review
  • Track agreement degradation across evaluation runs over time
  • Estimate and control costs before executing a multi-model evaluation
  • Design consistent evaluation schemas with AI-assisted suggestions

FAQ from Conkurrence

What problem does Conkurrence solve?

It provides a statistically validated consensus measurement for AI evaluation pipelines by using multiple models as independent raters.

Does Conkurrence require API keys for every run?

No—the self-consistency mode uses the host model via MCP Sampling and does not require any API keys.

What statistical measures does Conkurrence use?

It uses Fleiss' kappa with bootstrap confidence intervals and Kendall's W for inter-rater reliability and validity.

Can I estimate costs before running an evaluation?

Yes, the conkurrence_estimate tool reports the cost and token usage before execution.

What license does Conkurrence use?

It is licensed under BUSL-1.1 (Business Source License 1.1).

评论

其他 分类下的更多 MCP 服务器

Inbox Zero AI MCP

elie222

The world's best AI personal assistant for email. Open source app to help you reach inbox zero fast.

11.6K

Awesome-MCP-ZH

yzfly

MCP 资源精选， MCP指南，Claude MCP，MCP Servers, MCP Clients

7.4K

MCP Registry

modelcontextprotocol

A community driven registry service for Model Context Protocol (MCP) servers.

7.0K

MCP Go 🚀

mark3labs

A Go implementation of the Model Context Protocol (MCP), enabling seamless integration between LLM applications and external data sources and tools.

8.9K

🚀 Model Context Protocol (MCP) Curriculum for Beginners

microsoft

This open-source curriculum introduces the fundamentals of Model Context Protocol (MCP) through real-world, cross-language examples in .NET, Java, TypeScript, JavaScript, Rust and Python. Designed for developers, it focuses on practical techniques for building modular, scalable,

16.7K