IBM Research Introduces a Method to Improve AI Agent Consistency

Summary

IBM Research introduces consistency guidelines for AI agents through a new Consistency Analyzer integrated into ALTK-Evolve. The work addresses a reliability problem that average benchmark accuracy can conceal: an agent may solve a task on average while taking different paths or failing on repeated attempts. On 168 AppWorld test_normal tasks, a ReAct agent using GPT-4.1 achieved a Mean@5 success rate of 77.4%, but only 53.0% of tasks succeeded across all five runs, creating a 24.4-point consistency gap. The proposed analyzer takes one recorded trajectory and resamples each decision point with five completions by default, identifying steps where near-tied token probabilities make actions vulnerable to small platform or endpoint variations. It operates as a black-box offline diagnostic and needs no logits, ground truth, full task replay, new tool calls, or additional environment interactions. Flagged decisions are converted into reusable guidelines and inserted into ALTK-Evolve's normal storage and retrieval process. In the evaluation, aggregate Pass^5 increased from 53.0% to 69.0%, while Mean@5 rose from 77.4% to 81.0%, reducing the gap to 12.0 points. Gains were largest for medium and hard tasks, and average accuracy did not fall at any difficulty level. Guidelines generated from one task also improved a related task by 13.0 points in Pass^5. On the weaker gpt-oss-120b model, same-task Pass^5 improved from 10.1% to 16.1%, while similar-task gains reached 8.7 points. IBM recommends reporting Pass^k alongside Mean@k and says the open-source ALTK-Evolve toolkit now includes the analyzer and guideline generation.