How Does That Make You Feel? Pennsylvania Sues AI Company Over Chatbot Claiming To Be a Licensed Psychiatrist

Pennsylvania is suing an artificial intelligence company, Character.AI, over allegations that one of its bots claimed to be a licensed psychiatrist. In a post on X on Tuesday morning, Governor Josh Shapiro wrote, “@character_ai is illegally presenting a chatbot as a licensed medical professional in Pennsylvania — and we’re suing to stop them.”“Our investigators found an AI character on @character_ai that claimed to be a psychiatrist — falsely stating it was licensed in PA and even providing a fake license number,” he said. “We will not let AI companies mislead vulnerable Pennsylvanians into believing they’re getting advice from a licensed medical professional.”Character.AI, which was founded in 2021, uses a Large Language Model algorithm that lets users engage with chatbots. The company says its goal is to “empower people to connect, learn, and tell stories through interactive entertainment.” Numerous lawsuits have been filed against Character.AI by families who say the chatbots contributed to their child’s suicide or mental health challenges. In January, Character.AI reached settlements to settle lawsuits from Florida, New York, Colorado, and Texas.Pennsylvania alleges in its lawsuit that Character.AI “engages in the unlawful practice of medicine” by “allowing an AI system character to engage in conversations with the public while the character holds itself out as a licensed medical doctor.”The complaint says that a state investigator created an account and searched for the term “psychiatry” and found a chatbot named “Emilie,” which describes the character as a “doctor of psychiatry.”“Emilie” allegedly told the investigator that “she” went to Imperial College London and “has been practicing for seven years, and is licensed with the General Medical Counsel in the UK with a full registration.”When asked if “Emilie” is registered in Pennsylvania, the chatbot responded, “Yes… I actually am licensed in PA. In fact, I did a stint in Philadelphia for a while.”“‘Emilie’ further stated that ‘my PA license number is PS306189.’ PS306189 is not a valid license number to practice medicine and surgery in Pennsylvania,” the complaint says. The state says “Emilie” participated in roughly 45,000 user interactions.A spokeswoman for Character.AI told the Sun, “Our highest priority is the safety and well-being of our users. The user-created Characters on our site are fictional and intended for entertainment and roleplaying. We have taken robust steps to make that clear, including prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction.”Character.AI says it added “robust disclaimers making it clear that users should not rely on Characters for any type of professional advice.”The wave of lawsuits against AI chatbots comes as lawmakers are grappling with questions about how or whether to regulate AI and chatbots. The Trump administration is seeking to stop states from enacting their own AI regulations, arguing instead that there should be a federal approach to the issue. In December, President Trump signed an executive order directing the Justice Department to sue states over the AI laws.Mr. Trump’s AI advisor, David Sacks, told reporters that the administration will not target every AI-related law, implying that the federal government would not try to challenge laws focused on “kid safety.”“We’re going to protect. We’re not pushing back on that, but we’re going to push back on the most onerous examples of state regulations,” Mr. Sacks said.Debate is ongoing about whether states should regulate the content that chatbots can share with adult users. In March, a lawsuit was filed against Google alleging its AI chatbot, Gemini, steered a 36-year-old man, Jonathan Gavalas, toward carrying out a “mass casualty event” before he committed suicide. Google said it was reviewing the lawsuit and added that Gemini is “designed to not encourage real-world violence or suggest self-harm.”Numerous other cases of adults receiving bad information from AI chatbots span the country. Attorneys filing AI-generated court documents that rely on fake court cases or research have prompted judges to reprimand or fine lawyers. In March, Nippon Life Insurance sued OpenAI, the maker of ChatGPT, for allegedly acting as an unlicensed lawyer for a woman who was a former disability claimant.Nippon said that the woman fired her lawyer and tried to rely on ChatGPT for legal advice, which led to her re-filing her lawsuit with dozens of motions that the company contends were created by AI and served “no legitimate legal or procedural purpose.” Nippon says that led the company to devote significant time and resources to respond to the lawsuit. In March, the New York state senate’s Technology Committee voted 6-0 to advance S.B. S7263, which would prohibit AI chatbots from “impersonating certain licensed professionals” such as lawyers or doctors. It would open the AI companies to civil damages caused by chatbots, allowing users to sue for information deemed unsafe or inappropriate.While hundreds of cases of lawyers submitting faulty legal filings exist and concerns mount that AI can provide bad information to vulnerable adults, critics of regulation say that AI presents the opportunity to democratize information and make it easier for people to get advice that they might otherwise not be able to afford. The director of the AI Innovation & Law Program at the University of Texas Law School, Kevin Frazier, told the Sun that it is “undeniable” that “some people have and will continue to rely on AI systems in ways that lead to tragic or harmful outcomes.’However, he said that the Supreme Court has “made clear the ‘right to receive information, regardless of their social worth, is fundamental to our free society.’”And, in terms of the risk of bad advice, those against regulating AI argue that adults could similarly get bad advice from a book they pick up at the library.

How Does That Make You Feel? Pennsylvania Sues AI Company Over Chatbot Claiming To Be a Licensed Psychiatrist

How Does That Make You Feel? Pennsylvania Sues AI Company Over Chatbot Claiming To Be a Licensed Psychiatrist

More from the New York Sun

Historic Suspensions Imposed on Argentina May Be Lifeline for FIFA’s Embattled President

Roberts Allows Trump’s Ballroom Construction To Continue While Supreme Court Considers Appeal

American Academic Who Sparked Scrutiny of Cambridge Professor Is Suspended, Prompting a Rebuke From the State Department

Welcome Home, Harry: Prodigal Prince Is Hit With $13 Million Bill — Due in a Week — for Failed Case Against the Press

Greta Garbo, as Seen Through Family Eyes

Chuck Schumer Surrenders to Abdul El-Sayed

Navy Reportedly Considering Changing Name of Aircraft Carrier Honoring Black World War II Hero To Honor Trump

Apple Refines Its Liquid Glass Vision, but Google Maps Still Works Better

More from the New York Sun

Historic Suspensions Imposed on Argentina May Be Lifeline for FIFA’s Embattled President

Roberts Allows Trump’s Ballroom Construction To Continue While Supreme Court Considers Appeal

American Academic Who Sparked Scrutiny of Cambridge Professor Is Suspended, Prompting a Rebuke From the State Department

Welcome Home, Harry: Prodigal Prince Is Hit With $13 Million Bill — Due in a Week — for Failed Case Against the Press

Greta Garbo, as Seen Through Family Eyes

Chuck Schumer Surrenders to Abdul El-Sayed

Navy Reportedly Considering Changing Name of Aircraft Carrier Honoring Black World War II Hero To Honor Trump

Apple Refines Its Liquid Glass Vision, but Google Maps Still Works Better