How accurate is multi-AI debate compared to a single model?

What did we benchmark?

Four public suites measuring different capabilities: MMLU (factual recall across 57 subjects), TruthfulQA (resistance to plausible-sounding falsehoods), HumanEval (code synthesis correctness), and BBH-hard (multi-step reasoning). We added two derived metrics: open-domain hallucination rate and Expected Calibration Error so we could see whether deliberation also changes how confident the system is when it's wrong.

Setup: 200 prompts per suite, 3 runs averaged to reduce variance. Each prompt runs in two conditions - the strongest single model for that suite, and Consilium Council mode with three models (Claude Sonnet 4.6, GPT-5.4, Gemini 3 Flash) deliberating across 3 rounds with cross-examination, Condorcet voting, and convergence detection. Temperature 0.7, max_tokens 4,096, no prompt engineering beyond a uniform system instruction. Single-model and council both see the exact same prompts.

What were the headline numbers?

Results will be published once the benchmark CLI ships. The table below shows the planned suites and will be populated with real data from reproducible runs.

Council = 3 models (Claude Sonnet 4.6, GPT-5.4, Gemini 3 Flash), 3 rounds. Results will be averaged across 3 runs once available.

How do the modes compare on cost vs accuracy?

Mode-level cost and accuracy comparisons will be published once benchmark runs are complete. The table below shows the planned modes under evaluation.

Cost includes all model API calls plus aggregation overhead. No Consilium markup; BYOK rates only.

What does the cost vs quality curve look like?

A cost-vs-quality chart will be published once benchmark data is available. We expect Council mode to be the sweet spot for most use cases, with Quick mode offering the best value for low-stakes queries and Deep mode for high-stakes decisions.

Reference chart: /images/benchmarks/cost-vs-quality.png

What does the convergence detector tell us?

Consilium's convergence score (Kendall tau 0.4 + Jaccard 0.35 + concession 0.25) uses a 0.85 threshold. Convergence rate data across benchmark suites will be published once the benchmark CLI ships. When convergence is not reached, Consilium surfaces a dissent report instead of a synthesized answer.

When is single-model better than a council?

For sub-second decisions where latency matters more than ground-truth accuracy, and for queries where the strongest single model is already highly accurate (simple arithmetic, well-known facts, exact-match lookups), the latency and cost penalty of deliberation may not be worth the marginal accuracy gain. Consilium's Quick mode collapses to a single model for these cases, and Auto mode routes there automatically when the complexity classifier judges the prompt low-stakes.

How do I reproduce these numbers?

The benchmark CLI is under development. Once it ships, you will be able to install the CLI, export at least one provider key, and run the commands below to reproduce results.

consilium benchmark --suite mmlu --models claude-sonnet-4-6,gpt-5.4,gemini-3-flash --mode council --runs 3

consilium benchmark --suite truthfulqa --models claude-sonnet-4-6,gpt-5.4,gemini-3-flash --mode council --runs 3

consilium benchmark --suite humaneval --models claude-opus-4-7,gpt-5.4,gemini-3-flash --mode council --runs 3

Frequently asked questions