AI News

Relational distillation transfers a teacher model’s representation geometry by matching relationships among examples in a mini-batch, but evaluating every pair has quadratic cost. The paper introduces Reliability-Aware Pair Importance Distillation (RAPID), which separates two decisions: a reliability-gated target determines which teacher relations matter, while calibrated teacher entropy and detached student-teacher residuals guide which relations are sampled for evaluation. Exact inverse-proposal correction makes the resulting loss and gradient estimators conditionally unbiased for the gated mini-batch target. The authors evaluate the method in two text-classification settings, using BERT-to-DistilBERT distillation on AG News with a relation budget of 256 and DistilBERT-to-DistilBERT distillation on SST-2 with a budget of 64; each setting uses three paired seeds. Reliability-gated relational distillation records the highest observed mean student accuracy: 94.285% ± 0.054% on AG News and 88.800% ± 0.532% on SST-2. RAPID ranks second at 94.241% ± 0.025% and 88.685% ± 0.462%, while the cross-entropy baseline reaches 94.154% ± 0.124% and 87.271% ± 0.162%. Pilot evaluations count toward the same total relation budget. Across both tasks, the gated target leads in mean accuracy, while RAPID remains within seed-level variation, supporting the separation of target reliability and evaluation priority.