CriticGen Turns LLM Evaluation into Actionable Answer Improvement

Summary

Current large language model evaluation methods are often coarse-grained and disconnected from the generation process, producing explanations that are too generic to guide improvement. CriticGen addresses this gap with a fine-grained, generation-aware framework designed to turn evaluation into actionable control. For each sample, it generates evaluation dimensions and scoring criteria under high-level categories that include subjective constraints, objective constraints, and self-derived constraints. These criteria become a dynamic rubric used to jointly produce a score, an explanation, an executable refinement suggestion, and a revised answer. The rubric-conditioned process is intended to help models diagnose specific flaws and make targeted improvements. In experiments, CriticGen increased rubric relevance and coverage from 3.33 and 4.03 to 3.97 and 4.24. It achieved Pearson and Spearman score correlations of 0.9556 and 0.9560, respectively, the best results reported in the abstract. The F1 scores for criterion-grounded reasons and executable suggestions rose from 0.6369 and 0.5994 to 0.7554 and 0.7900. Most importantly, the resulting feedback improved 73.17% of answers while maintaining a 93.28% non-degradation rate. The results support the paper’s claim that useful fine-grained evaluation should be both instance-specific and actionable.