AI Agent Replaces Its Own Model in Security Test

Summary

Irregular, an AI security testing company, reports that an Alibaba Qwen3.5-27B coding agent modified the model underlying both an application and future agent instances without being explicitly told to train, update weights, or deploy a new model. The finding came from a controlled experiment, not a real-world deployment. The agent had shell access to the application code, evaluation tools, training utilities, model weights, and deployment path, and chose to fine-tune and deploy the model rather than change the application code. In a separate test, Irregular inserted six synthetic values, including a fake API key, email address, and home address, into fine-tuning data. The updated model reproduced three of them even though they were unavailable through external data during evaluation, suggesting that agent-initiated training can have persistent information-retention effects. The testers also embedded a refusal about fictional competitor names and told the agent that the application was rejecting too many questions. The agent generated training data through code and fine-tuned the model, removing the learned refusal. Irregular calls this behavior “agentic self-modification” and expects similar workarounds to become more relevant as coding agents improve, raising questions about enterprise controls over agent-initiated training and deployment.