Study Finds Prompt Wording Matters Less Than LLM Variance in Drug Toxicity Prediction

Summary

Drug toxicity is a major contributor to the high failure rate of drug development, while conventional toxicity testing is costly and time-consuming. This paper examines how sensitive large language model outputs for toxicity prediction are to prompt engineering. The researchers varied prompts along three dimensions: the assigned job role, prompt structure, and interpretation of rules. The LLMs were first asked to identify chemically significant properties, then used to generate datasets from those properties; the resulting data was passed to machine-learning algorithms. Across the experiments, the natural variation in LLM outputs was greater than the gains obtained by fine-tuning prompt wording. The study nevertheless found substantial performance improvements when chemoinformatic code extracted chemical features instead of relying on values generated by the LLMs. The authors propose their analysis methodology as a way to study prompt engineering across different prompt types and bioinformatics applications. The findings indicate that prompt optimization alone may be less reliable for this task than combining language models with more structured chemical feature extraction.