Do Preconditioners Improve TabPFN Fine-Tuning for Biomedical Data?

Summary

Tabular foundation models are increasingly used for structured biomedical data, and TabPFN is designed for low-data classification. This study examines whether optimization and preconditioning methods can improve biomedical fine-tuning of TabPFN v2.5, an area the authors say remains insufficiently explored. The researchers compare five AdamW-based preconditioning strategies across 59 biomedical datasets covering Alzheimer’s disease, breast cancer, schizophrenia, significant memory concern, KEEL biomedical datasets, and UCI biomedical benchmarks. They evaluate predictive performance, computational efficiency, and statistical significance. Across the experiments, the original AdamW optimizer achieves the strongest overall performance and statistical ranking. Existing curvature-aware preconditioners do not deliver reliable improvements across the different biomedical learning settings. The authors conclude that generic preconditioning methods may not capture the optimization characteristics of biomedical tabular learning. They propose biomedical-aware preconditioners tailored to healthcare-oriented tabular foundation models as a direction for future work.