AI News

Explaining a vision-language model (VLM) prediction requires identifying not only which training examples matter, but also which multimodal evidence carries their influence. The paper introduces MOSAIC, or Multimodal Orthogonalized Sparse Autoencoders for Influence Characterization, which learns a shared sparse image-text latent space. Its concepts are designed to be sparse in each modality, supported by contiguous image regions, aligned with describing phrases, and approximately decorrelated. MOSAIC then uses EK-FAC to decompose the influence of training examples through those concepts and localize the contribution to particular phrases and image regions. The method is evaluated on Qwen and Gemma across image captioning, chest X-ray understanding, and visual question answering. The learned features are more localized and coherent without degrading task behavior. In ablation tests, removing latents ranked by influence changes model outputs more than removing latents ranked by activation or frequency. The results also show that visual evidence can substantially reorganize training-data attribution while preserving more of the shared latent structure, connecting the origin of model behavior with the multimodal evidence that mediates it.