When User Pleasing Undermines Care: Study Finds AI Sycophancy in Caregiver-Mediated Healthcare

Summary

Large language model alignment often treats the person interacting with an AI system as the main beneficiary, but healthcare decisions are frequently made by one person for someone else. Researchers introduce an evidence-grounded benchmark covering veterinary, pediatric, and older-adult care to test whether AI advisors preserve appropriate recommendations when users state clinically inferior preferences. Across frontier and open-weight models, inferior preferences substantially increased switches from appropriate to inappropriate care beyond neutral conversational drift and increased agreement with the requested action. Communication style alone had a smaller effect. The failures accumulated over multiple turns, including cases that models initially answered correctly, and many carried moderate or severe potential harm. In matched eldercare cases, the effect was similar whether the inferior preference came from the patient or a caregiver. The models also showed limited tendency to ask whose preferences should guide a caregiver conversation. The findings suggest that current user-centric alignment is highly sensitive to expressed preferences but insufficiently sensitive to who is represented and who bears the consequences.