Conkurrence MCP Server

What is Conkurrence?

ConKurrence is a statistically validated consensus measurement toolkit for AI evaluation pipelines. It uses multiple AI models as independent raters, measures inter-rater reliability with Fleiss' kappa and bootstrap confidence intervals, and routes contested items to human experts.

How to use Conkurrence?

Install globally with npm install -g conkurrence, then run as an MCP server with npx conkurrence mcp. Add the corresponding configuration to your Claude Desktop or Claude Code plugin settings. Invoke the available MCP tools such as conkurrence_run, conkurrence_report, conkurrence_compare, conkurrence_trend, conkurrence_suggest, conkurrence_validate_schema, or conkurrence_estimate.

Key features of Conkurrence

• Multi-model evaluation using Bedrock, OpenAI, and Gemini simultaneously
• Fleiss' kappa with bootstrap confidence intervals and Kendall's W
• Self-consistency mode (no API keys needed, uses MCP Sampling)
• AI-powered schema suggestion from your data
• Trend tracking to detect agreement degradation over time
• Cost estimation before running an evaluation

Use cases of Conkurrence

• Evaluate inter-rater reliability when multiple AI models judge the same data
• Automatically route low-agreement items to human experts for review
• Track agreement degradation across evaluation runs over time
• Estimate and control costs before executing a multi-model evaluation
• Design consistent evaluation schemas with AI-assisted suggestions