ToolGate Builds Auditable Scientific Benchmarks with Tool-Using Agents

Summary

ToolGate is an executable pipeline for building scientific benchmarks that require specialist software. It validates generated items with reproducible scripts, screens out questions solvable without tools, and tests survivors with a tool-using agent. In a FEniCSx experiment, 500 attempts produced 128 unique protocol survivors after GPT-5.5 screening and Codex-based solving. The process makes benchmark acceptance auditable while preserving expert control over domain design and final review.