A Deterministic Math-Solving Interface for Clinical Language Models

Summary

Large language models can make arithmetic errors in clinical calculators, where one wrong number may change a recommendation. This study evaluates Program-Solve, an interface in which the model writes case-specific Python and a restricted local executor performs the calculation deterministically; the model decides how to use the solver rather than calculating itself. The evaluation uses MedCalc-Bench Verified, comprising 1,100 cases and 55 calculators, and compares direct model arithmetic with both Program-Solve and a hand-written library using Qwen2.5-7B and Qwen2.5-32B-AWQ. Before testing, the authors audited the calculator formulas against current clinical guidelines and flagged 16 calculators for version, usage, or coefficient concerns. With formulas and gold variables supplied and the complete clinical note available, Program-Solve reached 75.31% on the 7B model versus 72.02% for direct arithmetic, a paired gain of 3.29 points whose 95% calculator-cluster interval ranged from -3.49 to 10.38. On the 32B model, it reached 90.53% versus 83.47%, a 7.05-point gain with an interval of 0.47 to 14.60, clearing zero. The hand-written library was exact on its 440 supported cases but abstained on other cases, producing 40.0% overall coverage. The results indicate that an executor can help some open-weight models more than others, but it does not replace verified formulas or dependable variable extraction.