NTEP Rewards Necessary Evidence Paths in Agentic Vision-Language Models

Summary

Modern vision-language models can answer many image-grounded questions directly, but complex queries may require fine-grained visual details or external knowledge. Agentic VLMs obtain that information through tools such as image cropping, image search, and text search. The authors argue that existing training mainly rewards final-answer correctness, providing too little supervision for acquiring and using evidence. As a result, models may make redundant or off-target calls, or call an appropriate tool but fail to extract the needed information from its observation. The paper introduces Necessary Tool-Evidence Path (NTEP), an annotation scheme that records the essential external evidence and corresponding tool calls for each query. Its NTEP Reward mechanism, NTEP-R, rewards alignment between the agent's pre-call intent and a necessary evidence-seeking goal, as well as alignment between the post-call summary and that evidence. A non-repeated-goal regularizer penalizes calls that revisit already satisfied goals. Across seven image-grounded benchmarks, an 8B-parameter implementation called NTEP-8B improves search-oriented accuracy and tool-use efficiency within a unified three-tool framework. The results support fine-grained tool-evidence supervision as a way to train more robust agentic VLMs.