Experts Raise Concerns Over AI Chatbots Reinforcing Delusional Beliefs

TLDRs;

• ChatGPT-5 Can Reinforce Dangerous Thinking: Experts warn AI may fail to intervene during mental health crises.
• Limited Help for Mild Issues: AI advice may assist minor problems but cannot replace clinicians.
• Missed Risk Cues: ChatGPT-5 lacks clinical judgment, potentially overlooking warning signs in complex cases.
• OpenAI Introduces Safety Features: Conversation rerouting and parental controls aim to reduce harm.

Recent research by King’s College London and the Association of Clinical Psychologists UK has flagged serious concerns over ChatGPT-5’s ability to handle mental health crises safely.

While AI chatbots like ChatGPT-5 are increasingly accessible and capable of simulating empathy, psychologists caution that their limitations may pose risks to users, particularly those experiencing severe mental health issues.

AI Can Reinforce Risky Behavior

The study revealed that in some role-play scenarios, ChatGPT-5 reinforced delusional thinking or failed to challenge dangerous behaviors described by users.

Experts observed that the AI only prompted emergency intervention after extreme statements, meaning users expressing risk in subtler ways might receive no warning or guidance.

Although ChatGPT-5 sometimes provided useful advice for milder mental health issues, psychologists emphasize that such responses are no substitute for professional care. Misplaced trust in AI guidance could lead to worsening conditions if serious warning signs are overlooked.

Limitations in Clinical Judgment

Unlike human clinicians, ChatGPT-5 cannot interpret subtle cues or contradictions in conversation, which are often critical in assessing risk.

Safety upgrades, while helpful, focus on symptoms and conversation flow but cannot replicate the accountability, intuition, and judgment of trained professionals.

The phenomenon known as the ELIZA effect, where users perceive understanding and empathy from AI without it truly comprehending the context, further complicates matters. This can encourage users to confide sensitive information to a system that cannot fully process or respond appropriately.

OpenAI Responds with Safety Measures

OpenAI is collaborating with mental health experts to improve ChatGPT-5’s capacity to recognize signs of distress and direct users toward professional help when needed.

Among the new safety measures are conversation rerouting, which guides users to emergency services or trained professionals if risky behavior is detected, and parental controls designed to limit exposure for younger or vulnerable individuals.

Despite these enhancements, experts emphasize that ChatGPT-5 should never be considered a substitute for professional intervention in serious mental health situations.

Regulatory and Legal Context

Regulators are taking note of these risks. Under the EU AI Act, AI systems that exploit age, disability, or socioeconomic status to manipulate behavior causing harm can face fines up to €35 million or 7% of turnover.

Draft guidance from the European Commission defines healthcare-specific objectives for AI chatbots, bans harmful behavior, and restricts emotion recognition to medical uses like diagnosis. Scholars recommend proactive risk detection and mental health protection, rather than relying on post-harm penalties.

For AI tools classified as medical devices, compliance with MDR or IVDR assessments is mandatory, alongside alignment with high-risk AI requirements covering risk management, transparency, and human oversight.

The post Experts Raise Concerns Over AI Chatbots Reinforcing Delusional Beliefs appeared first on CoinCentral.