Conkurrence

Name: Conkurrence
Author: AlligatorC0der

@AlligatorC0der

About Conkurrence

Conkurrence measures whether multiple AI models produce consistent outputs on your evaluation tasks. It tells you which items your AI agrees on and which need human review — using Fleiss' κ, Kendall's W, and bootstrap confidence intervals, the same psychometric methods trusted in

#ai-evaluation #golden-dataset #llm-evaluation #testing #inter-rater-reliability #fleiss-kappa #kendall-w #validating-prompt-output #consensus #multi-model

Basic information

Config

Add this server to your MCP-compatible client using the configuration below.

{
  "mcpServers": {
    "conkurrence": {
      "command": "npx",
      "args": [
        "-y",
        "conkurrence",
        "mcp"
      ]
    }
  }
}

Tools

No tools detected

We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.

Overview

What is Conkurrence?

ConKurrence is a statistically validated consensus measurement toolkit for AI evaluation pipelines. It uses multiple AI models as independent raters, measures inter-rater reliability with Fleiss' kappa and bootstrap confidence intervals, and routes contested items to human experts.

How to use Conkurrence?

Install globally with npm install -g conkurrence, then run as an MCP server with npx conkurrence mcp. Add the corresponding configuration to your Claude Desktop or Claude Code plugin settings. Invoke the available MCP tools such as conkurrence_run, conkurrence_report, conkurrence_compare, conkurrence_trend, conkurrence_suggest, conkurrence_validate_schema, or conkurrence_estimate.

Key features of Conkurrence

Multi-model evaluation using Bedrock, OpenAI, and Gemini simultaneously
Fleiss' kappa with bootstrap confidence intervals and Kendall's W
Self-consistency mode (no API keys needed, uses MCP Sampling)
AI-powered schema suggestion from your data
Trend tracking to detect agreement degradation over time
Cost estimation before running an evaluation

Use cases of Conkurrence

Evaluate inter-rater reliability when multiple AI models judge the same data
Automatically route low-agreement items to human experts for review
Track agreement degradation across evaluation runs over time
Estimate and control costs before executing a multi-model evaluation
Design consistent evaluation schemas with AI-assisted suggestions

FAQ from Conkurrence

What problem does Conkurrence solve?

It provides a statistically validated consensus measurement for AI evaluation pipelines by using multiple models as independent raters.

Does Conkurrence require API keys for every run?

No—the self-consistency mode uses the host model via MCP Sampling and does not require any API keys.

What statistical measures does Conkurrence use?

It uses Fleiss' kappa with bootstrap confidence intervals and Kendall's W for inter-rater reliability and validity.

Can I estimate costs before running an evaluation?

Yes, the conkurrence_estimate tool reports the cost and token usage before execution.

What license does Conkurrence use?

It is licensed under BUSL-1.1 (Business Source License 1.1).

Comments

More Other MCP servers

Unity MCP ✨

justinpbarnett

Unity MCP acts as a bridge between AI assistants and your Unity Editor. Give your LLM tools to manage assets, control scenes, edit scripts, and automate tasks within Unity.

12.2K

Activepieces

activepieces

AI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents

23.2K