Quail Engine Speeds Up AI-SQL Workloads by Up to 14x

Summary

Quail is an open-source inference engine for AI-SQL, where SQL filters and joins invoke language models over large document collections. Its central design combines database query planning with model inference: Quail orders filters and joins, chooses join anchors, sizes forward passes, and manages reusable key-value cache pages according to the physical query plan. It uses vLLM model implementations but adds fused kernels, shared-anchor attention for joins, and an output head restricted to TRUE and FALSE decisions. In a benchmark of 29 queries covering reviews, medical reports, fact-checking claims, legal documents, and software-agent traces, Quail averaged a 1.84x speedup over a stock vLLM baseline at scale factor 0.1, although vLLM was faster on the two agent-trace queries. On BIO-4 at scale factor 1.0, using Qwen3 4B FP8 on one H100, Quail completed the workload in 29.26 minutes versus 6.84 hours for stock vLLM, a 14.04x speedup, and reduced GPU cost from $27.03 to $1.93 per query. The improvement came partly from reducing repeated KV computation and from avoiding CPU scheduling gaps that left the GPU idle. Quail was weaker on AGENT-1, where vLLM’s automatic prefix caching reused matching prefixes across different rows; Quail currently reuses KV when the same document recurs but does not yet provide equivalent cross-row prefix caching. The system currently supports AI filters and joins, three listed models, and H100 GPUs. The authors plan broader model and hardware support, host-memory or SSD KV storage, better kernel utilization, and additional caching and planning techniques. Quail is released under the MIT license.