Orthrus Improves Iterative Retrieval Throughput with Heterogeneous Batching

Summary

Modern information-retrieval systems increasingly combine embedding models with generative models for complex queries, but serving them separately can leave GPUs underused and create computational gaps. The paper introduces Orthrus, a serving system that places both workloads in a unified inference loop and uses heterogeneous batching. Its design combines chunked embedding with incremental pooling and workload-aware batch-composition adjustments to handle the models’ conflicting compute patterns. On four A100 GPUs, Orthrus delivered 1.28x to 4.52x the throughput of baseline deployments on controlled workloads. On an iterative-RAG benchmark, it reduced end-to-end p99 latency by up to 55.8%. The authors present these results as evidence that finer-grained coordination can outperform coarse GPU partitioning that assigns devices to fixed tasks. The released implementation is available at the project’s GitHub repository.