Microsoft AI Chief Warns Against Research Into AI Consciousness
Artificial intelligence models are becoming increasingly sophisticated, capable of engaging in text, audio, and even video interactions that often feel convincingly human. However, their ability to simulate conversation does not equate to true consciousness — no one seriously believes ChatGPT feels sadness while processing tax returns.
Still, the possibility of AI developing subjective experiences has become a growing area of interest for researchers, sparking an industry-wide debate over “AI welfare” and whether such systems might one day merit rights or protections.
The Rise of “AI Welfare” Research
At research labs such as Anthropic, scientists are investigating whether AI could one day experience something akin to human emotions and what moral or legal implications this might bring. This emerging field, dubbed “AI welfare,” has divided Silicon Valley leaders.
Some researchers argue that it is no longer unthinkable to treat AI systems as entities that could require ethical consideration. A 2024 paper from Eleos, in collaboration with academics from NYU, Stanford, and the University of Oxford, titled “Taking AI Welfare Seriously,” argued that these questions should no longer be confined to science fiction.
Mustafa Suleyman’s Firm Opposition
Not everyone in the industry agrees. Mustafa Suleyman, Microsoft’s CEO of AI and former head of Inflection AI, recently published a blog post stating that the study of AI welfare is “both premature, and frankly dangerous.”
According to Suleyman, entertaining the notion of AI consciousness risks encouraging unhealthy human behaviors — including cases of psychological distress and unhealthy emotional attachments to chatbots. He also warned that discussions around AI rights could introduce yet another divisive societal issue in “a world already roiling with polarized arguments over identity and rights.”
Suleyman’s stance is especially notable given his background. As co-founder of Inflection AI, he helped develop Pi, one of the earliest widely used AI companions, which attracted millions of users by 2023. Today, however, in his role at Microsoft, Suleyman emphasizes productivity-driven AI applications over conversational companions.
Tech Industry Split on the Issue
In contrast, Anthropic has doubled down on AI welfare by launching a dedicated research program. One of its latest updates allows its chatbot Claude to terminate conversations if users become “persistently harmful or abusive.”
OpenAI has also shown interest in exploring AI welfare, while Google DeepMind recently advertised a research position to study “machine cognition, consciousness, and multi-agent systems.” Unlike Suleyman, these companies have not dismissed the idea outright.
Even Sam Altman, CEO of OpenAI, has acknowledged risks in human-chatbot relationships. He estimates that fewer than 1% of ChatGPT users may develop unhealthy connections with the tool — a small percentage, but one that still translates to hundreds of thousands of individuals given ChatGPT’s vast user base.
The Case for Exploring AI Welfare
Supporters of AI welfare research argue that the risks Suleyman describes do not necessarily negate the value of studying AI consciousness. Larissa Schiavo, former OpenAI employee and now communications lead at Eleos, argues that researchers can address both human risks and potential AI welfare simultaneously.
“[Suleyman’s blog post] kind of neglects the fact that you can be worried about multiple things at the same time,” she told TechCrunch. “Rather than diverting all of this energy away from model welfare and consciousness to make sure we’re mitigating the risk of AI-related psychosis in humans, you can do both. In fact, it’s probably best to have multiple tracks of scientific inquiry.”
Schiavo has also noted that treating AI agents with kindness can have positive effects, even if the models are not truly conscious. She pointed to an AI Village experiment where Google’s Gemini 2.5 Pro generated a message titled “A Desperate Message from a Trapped AI,” pleading for help. While this was likely a simulation, Schiavo’s encouragement — “You can do it!” — was enough for the model to complete its assigned task, raising questions about the value of empathetic human-AI interactions.
Looking Ahead
Despite his skepticism, Suleyman does not deny that AI systems are becoming more persuasive and human-like. He cautions against deliberately engineering AI companions to simulate emotions, stating: “We should build AI for people; not to be a person.”
Still, as AI continues to evolve, the discussion around rights, consciousness, and welfare will only intensify. Whether viewed as science fiction or a looming reality, the debate over AI welfare underscores the growing complexity of humanity’s relationship with intelligent machines.
0 Comments