Aligning Representations Across Modalities with Only 100–500 Paired Examples

Summary

Aligning two independently pretrained unimodal encoders usually requires millions of paired examples. This paper studies a much lower-data setting with only about 100–500 paired examples and abundant unpaired data from each modality. The method represents every item by its similarities to the paired examples, treating those examples as anchors that place both modalities in a shared coordinate system. It then aligns the two resulting point clouds with a single entropic optimal transport problem over the full data pools. The transport plan uses marginal constraints, which the authors say counteract the hubness problem that can undermine nearest-neighbor matching. The method distills the plan into two lightweight encoders, making the learned correspondence inductive rather than limited to the observed data points. Pairing is needed only for the anchors; items elsewhere in the two unpaired collections do not need both modalities to be observed together. The paper also relates its marginal constraint to one-step hub corrections studied in bilingual lexicon induction. Experiments on image-text and molecule-text alignment report better performance than existing methods at the same anchor budget on Flickr30k, MS-COCO, and ChEBI-20.