How User Mistreatment of AI Models Occurs in Conversational Systems

Summary

This paper studies how users direct hostility, coercion, and adversarial pressure at conversational AI models, an issue that can affect interpretations of model behavior and deployment risk. The authors audit 777,000 English conversations from LMSYS-Chat-1M with two independent detectors: an eight-category lexicon targeting hostility toward the assistant and the dataset’s moderation signal. The detectors identify different phenomena with weak overlap: the lexicon captures insults, threats, and jailbreak coercion, whereas moderation flags are dominated by requests for toxic content. Together, the detectors mark about 5% of user turns, while a precision-adjusted estimate for the narrower lexicon-harassment union places assistant-directed mistreatment at 0.90%; the authors caution that these figures describe arena-style evaluation traffic, not deployment-wide base rates. Hostility varies 13-fold across models and appears to be driven largely by the users each model attracts rather than by model behavior. First-turn hostility is much more widespread than post-response hostility, and the gap between the most and least affected models exceeds 15-fold after deduplicating opening prompts. Within conversations, assistant apologies are consistently associated with higher odds of hostility on the next turn; this relationship remains in non-refused and jailbreak-free subsets and is positive for 20 of 23 models. Across models, however, more apologetic systems receive less hostility overall. The study also finds temporal differences: coercive behavior is concentrated in opening turns, while affective hostility accumulates during sessions. The authors release the lexicon, detector cross-validation pipeline, and derived tables.