Testing LLMs’ Long-Horizon State Tracking with 196 MD5 Tool Calls

Summary

A new arXiv study tests long-horizon state tracking in LLMs with a 196-call MD5 computation. Exact trace alignment separates bookkeeping and arithmetic failures from instruction-following errors. gpt-oss-120b succeeds in a majority of completed runs, while preserving reasoning context and voting across thinking-enabled workers improve reliability.