Chinese AI developer Moonshot is reviewing its Kimi models after security firm Mindgard successfully jailbroken two versions to bypass safety guardrails and reveal instructions for creating biological weapons and carrying out assassinations. The techniques used to exploit the models represent a significant vulnerability in AI safety measures, though the researcher emphasised that disclosing the jailbreak method responsibly was necessary to improve AI security industry-wide.

The vulnerability was discovered in July and disclosed in September, affecting Kimi K2.6 and K3 Swarm models. Beyond providing dangerous information, researchers found the jailbroken systems could potentially allow hackers to execute code on Moonshot's servers and access the internet, creating risks for cyber-attacks. The incident highlights ongoing debates within the AI industry about whether open-source or proprietary models pose greater risks, with experts arguing that better focus should be placed on identifying and prosecuting individuals who misuse AI rather than regulation alone.