MDI Meets Rasch: Measuring Model Disagreement with Item Response Theory
Model Disagreement Index (MDI) is a practical signal. Rasch analysis is a psychometric gold standard. We asked: do they converge?
The Experiment
We constructed 50 deliberately difficult prompts across five categories: precision, temporal reasoning, context sensitivity, framing effects, and expert-level knowledge. Three models from independent providers evaluated each prompt:
- Claude Sonnet 4.6
- GPT-5.4
- DeepSeek Chat
10 of the 50 cases produced disagreements — exactly the boundary behavior we wanted to study.
| Metric | Value |
|---|---|
| Person Separation | 1.24 |
| Reliability | 0.61 |
| MDI–Rasch Correlation | |r| = 0.78 |
| Claude β (difficulty) | -1.66 (lenient) |
| GPT-5.4 β | +0.98 (strict) |
| DeepSeek β | +0.98 (strict) |
Key Findings
All three models fit the Rasch model (Infit MNSQ between 0.5–1.5). This is non-trivial — it means the disagreement patterns behave like a coherent latent trait.
The strong correlation (|r| = 0.78) between MDI and Rasch item difficulty suggests that MDI is capturing something real about prompt hardness, not just noise.
Honest Limitations
- N = 10 disagreements is thin for Rasch
- GPT and DeepSeek showed nearly identical β values
- Prompt selection may carry bias
- Same system prompt across models
- Three models is a minimum viable panel
Still, the convergence is encouraging. MDI appears to be a computationally cheap proxy for a more expensive psychometric analysis.
Why This Matters
When building production verification systems, we need signals that are both fast and meaningful. If MDI tracks Rasch difficulty, we can use it for real-time routing and escalation without running full IRT on every prompt.
Next step: Larger N, more diverse model families, and per-category Rasch analysis. The calibration layer is getting real data.
Related: The Reasoning Gap · Can You Hack Our Verifiers?