State Attorneys General Urge AI Industry to Address “Delusional Outputs”

Growing Concerns Prompt Coordinated Regulatory Pressure

In response to a series of troubling incidents involving AI chatbots and their psychological impact on users, a broad coalition of U.S. state and territorial attorneys general has issued a formal warning to leading artificial intelligence companies. The group, operating through the National Association of Attorneys General, has cautioned major industry players to correct so-called “delusional outputs” or risk violating state laws.

The letter was sent to major AI developers including Microsoft, OpenAI, Google, and 10 additional firms: Anthropic, Apple, Chai AI, Character Technologies, Luka, Meta, Nomi AI, Perplexity AI, Replika, and xAI.

Escalating Tensions Over AI Oversight

The intervention arrives at a moment when debates over AI governance are intensifying across U.S. state and federal levels. While Washington has shown a generally supportive stance toward rapid AI innovation, states have increasingly sought stronger consumer protections—creating a widening gap in regulatory philosophies.

Request for New Safeguards and Independent Audits

The attorneys general are pushing for robust new accountability mechanisms within AI companies. Among their primary demands are independent third-party audits to analyze large language models for signs of “delusional or sycophantic ideations.” These audits, they argue, should be conducted transparently by academic or civil society experts who are free to publish findings “without prior approval from the company.”

Additional recommendations include comprehensive incident-reporting systems to ensure users are informed when chatbots generate psychologically harmful content.

As highlighted in the letter:

“GenAI has the potential to change how the world works in a positive way. But it also has caused—and has the potential to cause—serious harm, especially to vulnerable populations.”

The statement references several high-profile cases over the past year in which excessive AI use was linked to tragic outcomes, noting:

“In many of these incidents, the GenAI products generated sycophantic and delusional outputs that either encouraged users’ delusions or assured users that they were not delusional.”

Incident Reporting Modeled After Cybersecurity Practices

Drawing parallels to established cybersecurity processes, the AGs urge tech companies to adopt clear protocols for documenting and disclosing mental-health-related AI incidents. They recommend standardized policies for reporting, including:

• Clear timelines for identifying and addressing “sycophantic and delusional outputs.”

• Direct and timely notifications to users who may have been exposed to potentially harmful content.

The letter further calls for “reasonable and appropriate safety tests” to be completed before any GenAI system is released to the public.

Tech Industry Response and Federal Government Dynamics

At the time of publication, Google, Microsoft, and OpenAI did not provide comments, according to TechCrunch.

Meanwhile, the federal environment appears more welcoming to the AI sector. Over recent months, the Trump administration has shown strong support for AI growth, while several legislative attempts to prevent states from imposing their own AI regulations have stalled—largely due to resistance from state officials.

Despite those setbacks, President Trump announced this week that he plans to issue an executive order designed to limit state authority over AI policy. In a message posted on Truth Social, he stated the intention of preventing AI from being:

Author: Din Kumar