CUSP Quantifies Uncertainty in Multi-Agent Multimodal Reasoning

Summary

Aggregating heterogeneous vision-language models can improve multimodal reasoning, but individual confidence scores do not reliably describe the ensemble as a whole. CUSP, or Collective Uncertainty through Semantic Opinion Pooling, is a training-free framework that maps model responses into a shared semantic space and combines them into a pooled semantic opinion. It reports collective uncertainty, which captures dispersion in that pooled opinion, and Jensen-Shannon divergence, which captures conflict among model-level opinions. The framework exactly decomposes unnormalized collective entropy into the mean individual semantic entropy plus Jensen-Shannon divergence, separating overall uncertainty from disagreement. CUSP requires neither token logits nor calibration labels and can work with open-weight and commercial VLMs. In static ensembles, collective uncertainty achieved 0.764 AUROC for prediction-error detection and 0.889 AUARC for abstention in the small-model setting, outperforming majority voting and naive selection by 4.7 to 15.8 points; its advantage increased with ensemble size. JSD was strongest in the evaluated commercial setting, reaching 0.819 AUROC and 0.910 AUARC, and ranked difficult-answer conflicts with up to 0.982 AUROC. The pooled prediction also improved accuracy over the average individual model by 5.6 to 13.0 points. Across a multi-step, multi-agent trajectory, subagent collective uncertainty detected system failures above chance, with 0.619 AUROC and 0.699 AUARC for abstention ordering.