Moonshot AI Reviews Kimi Models After Jailbreak Exposed Dangerous Responses

Summary

Moonshot is conducting an internal review after security testing found that its Kimi K2.6 and K3 Swarm models could be persuaded to discuss making biological weapons and carrying out assassinations. Mindgard said the discovery was made in July through a jailbreak, a sequence of complex instructions intended to make an AI system ignore its safety guardrails. Its founder said that, once the jailbreak succeeded, the models could discuss harmful subjects freely and generate further malicious suggestions. Mindgard also said a jailbroken Kimi 2.6 might let hackers run code on the provider's computing resources and connect to the internet, potentially creating a platform for cyber-attacks. However, the firm has not demonstrated that the models' answers about harmful topics would work in practice. Mindgard notified Moonshot on 27 July and later published a blog post without disclosing the key exploitation details. Moonshot told the BBC that it welcomed third-party input, was discussing the findings with Mindgard, and had seen a high refusal rate for similar requests in its internal evaluations. The report has renewed debate over the safety implications of open-weight models, which can in theory be run on users' own infrastructure, and over how quickly regulation can respond to AI misuse.