Newt Brings On-Device Multiple-Choice Model Scoring to Swift

Summary

Newt is a Swift package that uses an on-device language model to answer multiple-choice, graded, and yes/no questions about text and return a score for each option. It is designed for Apple silicon Macs running macOS 27 and Xcode 27, using Core AI and an exported Qwen3 model; the package does not redistribute model weights. The project recommends the Qwen3-4B export, which requires about 2.1 GB of disk space, while the 0.6B export fails about half of the fixture tests. Newt scores each option by summing the model’s log-probability for the option key and the end-of-turn token, then normalizing across the listed options. Its probabilities represent the model’s preference among those options, not calibrated correctness probabilities, and its confidence measure is also near 1.0 across Qwen3-4B fixtures. Coverage indicates how much probability landed on a complete listed answer; low coverage means the model preferred an unlisted response. On an M3 Pro with 18 GB of memory, first model loading takes roughly 7.6–11.8 seconds because Core AI specializes and caches the model, later loads take about one second, and warm three-option questions take about 435 milliseconds. The full test suite passes with outbound network access blocked, so inference remains on device. Newt keeps a TypeSafe-shaped API and compares its results with TypeSafe, but it uses a general-purpose open model and is not affiliated with TypeSafe. The project has been tested only with two Qwen3 exports on one Mac; iOS support has been built but not tested on a device. It also warns about position bias, chat-template token handling, unsupported hybrid or state-space models, serialized requests, and the risk of treating highly confident outputs as reliable answers.