Synthetic Ground-Truth Framework for Evaluating Explainable AI

Summary

Evaluating explainable AI (XAI) remains difficult because reliable procedures and ground-truth explanations are generally unavailable. Existing approaches often measure fidelity to a black-box model's predictions, but fidelity only shows how well an explanation reproduces outputs; it does not establish that the explanation reflects the model's actual decision process. As a result, explanations with similar fidelity scores can still offer inconsistent or misleading interpretations. The paper proposes a framework based on controlled interventions that generate synthetic datasets where the importance of input components is determined by design. These interventions make it possible to construct ground-truth explanations aligned directly with the behavior of the model being analyzed. The framework is tested in three domains: binary images, tabular data, and time series. Experiments evaluate nine widely used XAI methods and reveal significant limitations in current techniques. The results support using synthetic, intervention-based benchmarks to assess explanation quality more reliably.