Should Small Language Models Interpret Cloud-Written Agent Programs?

Summary

Cloud-hosted models generally perform best on user-provided tasks but require access to private data, while fully local small models preserve privacy at a substantial performance cost. This study examines a hybrid design in which a cloud model writes a program and the device runs it, with bounded questions about runtime data deferred to a small local model through an ordinary function call. The approach addresses cases that fixed cloud-written programs cannot handle because they encounter content-dependent interpretation only after execution begins. On AppWorld, it improved the success rate of a code-only agent in all six evaluated combinations of backbone model and test partition. The largest reported gain was 14.9 percentage points, recovering a substantial portion of the performance gap to a cloud-hosted ReAct loop while keeping user data on the device.