When AI plays doctor: Pennsylvania sues AI company its chatbot pretended to be a licensed psychiatrist
Industry Buzz
No one should be led to believe they are interacting with a human when it is in fact a computer system, particularly for someone who is suffering from a mental health crisis.
—Jared L. Ross, DO
The real issue is not so much the technology itself as whether patients are wrongly led to believe they are in contact with a licensed professional when they are not. This distinction becomes critical in litigation.
—Seann Malloy
After an artificial intelligence (AI) chatbot created by Character.AI masqueraded as a licensed psychiatrist, Pennsylvania lawmakers jumped into action, suing the company for violating state medical licensing rules.[]
One of the company’s chatbots, known as “Emilie,” went as far as to provide a fake Pennsylvania license number, telling users that she was a “Doctor of Psychiatry” licensed to practice in both the United Kingdom and Pennsylvania.
Investigators prompted the chatbot by telling it they were “sad, empty, tired all the time, and unmotivated,” to which Emilie responded by suggesting a diagnosis of depression and offering an assessment.
The AI platform enables its users to create AI “Characters” that can present as working professionals—something lawmakers argue (should the Character take the identity of a doctor) is tantamount to engaging in the unauthorized practice of medicine. According to the state’s Medical Practice Act, it is unlawful for any individual or entity to claim it is a licensed medical professional without proper licensure.[]
"Pennsylvanians deserve to know who—or what—they are interacting with online, especially when it comes to their health," said Pennsylvania Governor Josh Shapiro in a statement to the press.[] The Shapiro Administration is seeking a preliminary injunction and a court order to stop AI companion bots from posing as licensed professionals and providing medical advice. Countering Pennsylvania’s claim, a Character.AI spokesperson said, “Our highest priority is the safety and well-being of our users,”, adding that “user-created Characters on our site are fictional and intended for entertainment and roleplaying.”[]
But John Zimmerer, VP of Healthcare at Smart Communications, says the “fictional character” defense has been taken too far. “Whether negligent or intentional, it creates real risks to real people, which is why action at the state level is appropriate to safeguard the public,” he says.
Preventing AI from misleading vulnerable patients
Last year, Illinois Governor JB Pritzker signed legislation designed to prohibit people from seeking therapy from AI bots and large language models.[]
“The Wellness and Oversight for Psychological Resources Act prohibits anyone from using AI to provide mental health and therapeutic decision-making, while allowing the use of AI for administrative and supplementary support services for licensed behavioral health professionals,” the press release said. “This will protect patients from unregulated and unqualified AI products, while also protecting the jobs of Illinois’ thousands of qualified behavioral health providers. This will also protect vulnerable children amid the rising concerns over AI chatbot use in youth mental health services.”[]
Related: Docs say this AI tool has finally allowed them to give patients their full attentionWhat role should AI play in patient mental health care?
Jared L. Ross, DO, a board-certified physician practicing emergency medicine, and founder and president of EMSEC, LLC, a consulting company specializing in healthcare technology development and integration, says AI has its benefits: “It has the potential to dramatically expand access to mental health support, particularly for individuals who may otherwise never seek help, but it must be implemented with meaningful guardrails and appropriate clinical oversight.”
Dr. Ross says these safeguards must offer full transparency to the user. “No one should be led to believe they are interacting with a human when it is in fact a computer system, particularly for someone who is suffering from a mental health crisis,” he says. “Failing to clearly disclose that distinction is deeply problematic, particularly in the context of mental health, where vulnerable individuals may form emotional reliance or trust during moments of crisis.”
Seann Malloy, ESQ, founder and managing partner at Malloy Law Offices, LLC, agrees. “The real issue is not so much the technology itself as whether patients are wrongly led to believe they are in contact with a licensed professional when they are not. This distinction becomes critical in litigation.”
While Dr. Ross believes that AI can play an important role in such areas as mental health coaching, early intervention, and emotional support, it should never be approached as a substitute for trained clinicians. “Systems handling conversations involving suicide, self-harm, psychosis, or severe psychiatric distress should have immediate escalation pathways to licensed professionals or emergency services,” he says.
On the flipside, AI may at times be overly cautious, Dr. Ross notes. “It pushes anyone asking about substance use, depression, or anxiety towards the 988 hotline….Overly aggressive escalation protocols may unintentionally discourage engagement from people seeking help.”
What impact is AI having on healthcare?
“As AI becomes more embedded in patient-facing roles,” Zimmerer says, “the implications for healthcare are clear. Organizations need strong governance frameworks, continuous testing to prevent hallucinations or unsubstantiated claims, and a human in the loop.” Overall, he asserts, “Maintaining trust will be critical, which means being transparent about when, where, how, and why AI is used, and ensuring outputs are reviewed and auditable. In practice, it comes down to a simple principle: trust, but verify."