TEAM-Design Targets Evaluation at the Human-Agent Deployment Decision

Summary

The paper frames a practical deployment question: a human-agent workflow should be retained only when it outperforms both a human-only process and an agent-only process. After deployment, those alternatives are not directly observed, so comparing them requires replaying tasks, with each replay consuming expert time or compute. Under a fixed replay budget, the central design problem is deciding which tasks should receive human-only replays and which should receive agent-only replays. The authors argue that standard agent benchmarks do not make this choice, variance-based sampling does not account for which comparison is closer to failing, and Bayesian information methods target parameter learning rather than the deployment decision. They propose TEAM-Design, which assigns each task separate replay probabilities for the two baselines. The rule increases a probability when the corresponding missing outcome is difficult to predict from existing task information and when that comparison is harder to establish, while reducing it when replay is costly. The paper proves that this rule solves the budgeted design problem and that random replay from the assigned probabilities controls the chance of falsely concluding that the workflow beats both alternatives. In a reanalysis of six clinical settings, no human-agent workflow beats both baselines; in a coding benchmark, one workflow does. Synthetic experiments and a semi-synthetic design based on a real chest X-ray reader study show that TEAM-Design performs best when one comparison is clearly harder to settle, but can underperform variance-based allocation when both comparisons are similarly difficult.