KC-Bench: A Dynamic Interactive Benchmark for Knowledge Conflicts in LLM Agents

Summary

As large language models increasingly use tools to act, they must reconcile user instructions, knowledge stored in model parameters, and observations from changing environments. KC-Bench is a controlled, multi-turn benchmark designed to measure that behavior across three settings: conflicts with world knowledge, inconsistent inputs, and temporal conflicts involving multiple sources. The benchmark contains 238 manually screened tasks selected from more than 1,000 generated candidates. Each task combines a user simulator, stateful tools, deterministic assertions about the environment, an open-source natural-language evaluator, and human verification of agent trajectories. The authors evaluate nine models, including DeepSeek-V4-Flash, GLM-5.2, and MiniMax-M3. Results vary substantially by domain, and no evaluated model reliably handles factual correction, identity-consistency checking, and temporal-conflict resolution across all settings. In simulated environments, failing to detect a conflict can carry through to tool calls or synthetic protected-data flows. KC-Bench is intended to isolate model-level conflict handling rather than rank complete agent frameworks. The authors present it as a reproducible diagnostic for developing reasoning and execution safeguards that are aware of conflicting information.