Hurray Proposes a Common Tensor Interchange Format for AI/ML Systems

Summary

The paper surveys how AI/ML systems describe and move tensors between libraries, processes, accelerators, machines, and storage. It explains that shape and element type are insufficient for many modern workloads: tiled or packed layouts, sparse indexes, quantization parameters, paged KV caches, device memory, synchronization, and sharding can all affect whether a consumer can use buffers directly. Existing systems cover parts of this space. DLPack provides a compact in-memory ABI for strided tensors, Apache Arrow supplies dense and sparse tensor representations plus IPC, SafeTensors and GGUF target persistent model files, Zarr and NetCDF support large arrays, and UCX, NIXL, and NCCL move data without fully describing its meaning. ONNX carries additional tensor and quantization metadata, while MLIR, PJRT, and PyTorch DTensor address related compiler, memory, and distributed-placement concerns. The paper identifies a gap for a reusable descriptor that preserves execution-relevant representation across runtime memory, streams, files, and network transfers. It presents Hurray, an open-source beta, pre-1.0 project with a language-independent descriptor covering logical and storage types, shape, layout, quantization, device and memory placement, buffers, synchronization, and composition. Hurray currently defines twelve layout families and supports descriptions for sparse, paged, composite, and sharded tensors, as well as streaming and indexed file forms. Its protocol is intended to support capability negotiation so a producer and consumer can choose direct use, relocation without reformatting, or explicit conversion. The proposal does not replace CUDA IPC, RDMA, NIXL, UCX, NCCL, allocators, or kernel scheduling; it describes what the buffers mean, while other systems make them accessible and move them. The paper uses disaggregated LLM inference as a concrete case, where paged KV caches may be transferred between prefill and decode workers. It emphasizes that Hurray remains immature: interoperability will require independent implementations, byte-level conformance tests, precise layout and quantization definitions, validation, fuzzing, and evidence that preserving representations removes costly copies or conversions.