AI News

Long-context large language model inference is increasingly constrained by prefill, because dense self-attention processes the entire prompt before generation. RBS-Attention is a training-free sparse-prefill method designed to address a failure mode called mean dilution, in which a block centroid can hide a highly relevant token among mostly irrelevant tokens. It combines a centroid-based branch that captures average relevance with a rescue branch that uses the maximum key-block radius and prompt-, layer-, and head-dependent distributions to identify blocks likely to be underestimated. The two branches are thresholded independently and their masks are combined, allowing rescue blocks to be controlled while retaining regular block-sparse FlashAttention execution. On H100 GPUs at 128K context, the method reports a 20.65x standalone prefill-attention speedup, an 11.92x vLLM prefill-attention speedup, and a 5.97x end-to-end time-to-first-token speedup on Qwen3-30B-A3B-Instruct-2507-FP8. On dense Qwen3-32B, it achieves 88.65 overall RULER accuracy, compared with 89.52 for dense attention. LongBench-v2, InfiniteBench, and Video-MME provide additional quality evaluations, while further experiments examine retention, matched-density selector comparisons, block size, thresholds, and memory behavior.