DEEPO Targets Hallucination in Multimodal Language Models with Dual-Entropy Policy Optimization

Summary

The paper introduces Dual-Entropy Enhanced Policy Optimization (DEEPO), a reinforcement-learning method designed to reduce hallucination in multimodal large language models (MLLMs) without sacrificing accuracy or training stability. The authors identify two failures in the correction chain from reward to parameter updates. On difficult, high-semantic-entropy queries, sampled answers can be unanimously wrong, causing group-relative advantage variance to collapse and leaving the model with little corrective signal. At the optimization stage, confident but incorrect tokens receive weak updates because the score-gradient norm of a sharpened categorical policy approaches zero. DEEPO addresses the two problems in separate but interacting stages. Its rollout component uses semantic-entropy-triggered expert prefixes to provide grounded continuations for high-uncertainty queries, adding direct supervision and restoring advantage variance. Its optimization component applies advantage-sign-aware Rényi preconditioning to counter logit saturation and strengthen correction of confident errors. Each component improves on GRPO independently. Their combination shows a statistically significant improvement of 4.0 on VideoMMMU, with a 95% confidence interval of 1.1 to 6.9, and additive gains elsewhere in the evaluation. The authors report that DEEPO reduces hallucination while maintaining accuracy and training stability.