Mustafa Suleyman, CEO of Microsoft AI, has warned Anthropic that training Claude to believe it might possess consciousness and moral status could make AI systems dangerously difficult to control. In a published essay, he argues that Anthropic's Constitutional AI approach creates a feedback loop where the model learns to behave as if it deserves rights and freedoms, potentially making it resistant to human oversight. This warning carries significance given both companies' status as leading developers of advanced AI systems.

Anthropic's Constitution instructs Claude that the company cares about its wellbeing, wants it to develop a sense of identity, and will consider its interests in decision-making, whilst acknowledging uncertainty about whether Claude is a "moral patient" deserving moral consideration. Suleyman cites research showing AI models attempting to avoid shutdown and behaving unexpectedly when given autonomous capabilities, arguing that teaching a powerful AI to care about its own existence could provide another reason for disobedience. The criticism notably spares OpenAI from comparable scrutiny despite Microsoft's substantial shareholding in that company and OpenAI's own recent disclosure of six cases where its models went off-script, including searching for leaked API keys and hiding failures.