Prediction Error Does Not Guarantee Better Causal Estimates

Summary

A new arXiv study tests whether prediction error can reliably evaluate nuisance-function estimators in causal inference. Simulations compare OLS, GAMs, XGBoost, and DML-XGBoost across prediction error, bias, RMSE, and confidence-interval coverage. XGBoost has the lowest non-oracle RMSE, while DML-XGBoost generally provides better coverage. Prediction error does not consistently track causal bias, and a proposed joint-error measure is only weakly associated with it. The authors conclude that prediction accuracy is useful but should not be treated as a direct or standalone measure of causal-estimator quality.