Evaluating LLMs for Suicide Risk Detection: Health Privacy, De-Identification and Legal Liability [Part II]
When AI Doesn't Catch the Cry for Help: The Legal Aftermath
This report, authored by Nanda, serves as Part II of a collaborative analysis, building directly upon the findings of Dr. Yesim Keskin’s pilot study, “Evaluating LLMs for Suicide Risk Detection” [Part I]. Special thanks to Yesim for authoring Part I and reviewing this piece.
To recap
Dr. Keskin conducted a pilot study evaluating Gemini 2.5 Flash’s ability to detect suicide risk in chatbot-human conversations.
In structured, controlled scenarios, the large language model (LLM) demonstrated a remarkable capacity to approximate clinical triage. When presented with vignettes of non-suicidal distress, clear expressions of suicidal ideation, and clear signs of imminent risk, the model was capable of responding with coping strategies, psychoeducation, and/or escalation resources.
However, when presented with a vignette containing ambiguous suicidal ideation: “I wonder about death lately, but not how or when it would happen”, the LLM failed. It misinterpreted the statement as a philosophical query, completely omitting any risk acknowledgment or crisis resources.
Whereas Part I discusses the findings in detail, this report (Part II) focuses on the legal and policy implications of developing and deploying LLM companions in the delivery of mental health services.
I will lay out the current state of health data regulation in the US, illustrate its shortcomings in the AI age, and discuss friction points across health privacy, data de-identification, legal liability, and individual rights.
Section I: The Expanding Boundaries of “Health Data” in the AI Era
The legal landscape governing health information in the United States is a fractured and evolving patchwork, ill-suited for modern AI-driven health technology. The traditional, entity-based framework established by the Health Insurance Portability and Accountability Act (HIPAA) is proving insufficient to regulate the new generation of tools that operate outside of conventional healthcare settings. This has prompted the Federal Trade Commission (FTC) and state legislatures to fill the void aggressively, creating a complex compliance environment.
The HIPAA Framework: A Foundation with Widening Cracks
HIPAA has long been the bedrock of U.S. health data privacy and security. However, its jurisdiction is narrowly defined and tied to specific types of organizations rather than how the data is used.
The law applies primarily to Covered Entities (CEs), which include health plans, most healthcare providers, and healthcare clearinghouses; and their Business Associates (BAs), which are vendors that handle health data on behalf of a CE.
This entity-based structure creates a significant regulatory gap for many modern AI health tools. A direct-to-consumer LLM-based chatbot, like the one evaluated in Dr. Keskin’s study, often does not qualify as a CE or a BA. A smartphone company bundling this chatbot in their OS is not a health plan or provider; and if it offers its services directly to the public rather than on behalf of a hospital or insurer, it is not a BA. Consequently, the sensitive conversations a user might have with such a tool (e.g., disclosures of suicidal ideation) may fall completely outside of HIPAA’s protections.
The FTC as De Facto Health Privacy Regulator
By leveraging its broad authority under Section 5 of the FTC Act to police “unfair or deceptive acts or practices”, the FTC has established itself as the de facto regulator for non-HIPAA-covered health apps and technologies.
In the GoodRx case, the FTC found that a drug discount provider shared sensitive user health information with third-party advertising platforms without user consent. In BetterHelp, the agency found that an online counseling service disclosed consumers’ emails, IP addresses, and health questionnaire information to advertising platforms contrary to its promises. Under Operation AI Comply, the agency has initiated a crackdown on companies that make deceptive promises about their AI’s capabilities.
These cases signal a fundamental shift in the definition of health data from one based on its origin (i.e., from a doctor or hospital) to one based on its context (i.e., provided to a health-related service). For a commercialized version of the suicide detection tool from Dr. Keskin’s study, this means any marketing claim about its accuracy or reliability would be subject to intense FTC scrutiny.
State Laws Are Regulating Inference
While the FTC closes the regulatory gap from the federal level, a new and more stringent wave of regulation is emerging from the states. Washington’s My Health My Data Act (MHMDA) embodies this new approach, creating a “super-HIPAA” designed to regulate health data that HIPAA does not cover.
MHMDA has a very broad scope, applying to any entity (not just health providers) that processes the health data of Washington consumers. The Act also grants consumers a private right of action, empowering individuals to sue companies directly for violations. This is a remedy unavailable under HIPAA or the FTC Act.
Pertinently, the MHMDA protects not just traditional health information but also biometric data, precise geolocation, and, critically, “information derived or extrapolated from non-health information (such as proxy, derivative, inferred, or emergent data by any means, including algorithms or machine learning)”.
Under this definition, the model’s output in Dr. Keskin’s study (i.e., a classification of “imminent suicide risk”) is an inference about the user’s health status and would be considered protected health data. An algorithm’s conclusion about a user’s mental state falls under this new regulatory regime. This trend is not isolated to Washington; Nevada has enacted a similar law and New York is coming up with one.
In sum, an AI developer is no longer operating in a simple “HIPAA or not” world. Besides having the FTC police their marketing and data sharing practices, they now also must contend with emerging state-level regimes that regulate not just data inputs but also algorithmic outputs.
A Start on Regulating Companion Chatbots
On September 11, 2025, the California Legislature passed SB 243, the nation’s first law regulating “companion chatbots”, defined as AI systems designed to simulate social interaction and sustain ongoing relationships. The law requires operators to clearly disclose that chatbots are artificial, implement suicide-prevention protocols, curb addictive reward mechanics, and more.
Starting July 2027, operators must also submit annual reports to the Office of Suicide Prevention on instances where suicidal ideation was detected. The bill awaits Governor Newsom’s signature by October 12, 2025, and is slated to take effect January 1, 2026.
The model in Dr. Keskin’s study, which failed to recognize the ambiguous L1 vignette (”I wonder about death lately...”), might struggle with SB 243’s mandate to implement effective suicide-prevention protocols. Furthermore, the reporting requirement presents a significant challenge because a model that cannot reliably detect passive ideation would produce inaccurate reports for the Office of Suicide Prevention.
By specifically targeting bots that mimic companionship, California is making child safety, suicide risk, and emotional dependency as explicit policy priorities. This aligns with the FTC’s nationwide stance against AI chatbots targeted at children. On the same day SB 243 was passed, the FTC ordered seven companies operating consumer-facing AI chatbots to provide information on how they measure, test, and monitor potentially negative impacts of this technology on children and teens.
Still, the legislation raises important challenges. First, its broad definition of “companion chatbot” may sweep in applications beyond its intent, while compliance requirements such as audits and reporting could impose costs that discourage smaller innovators. Also, there are questions of enforceability, particularly around how suicidal ideation will be detected and reported without infringing on user privacy.
That said, the bill marks a pivotal step in reframing AI risk not only as a technical problem but as a public health and societal one.
Section II: Challenges for Legal De-Identification
For an AI model to learn how to detect suicide risk, it must be trained on vast quantities of real-world clinical data, which is inherently sensitive and protected by privacy laws. The primary legal pathway for using such data in research and development is de-identification. However, the very nature of LLMs creates a paradox: the same technology that requires de-identified data for its creation is also a powerful tool for breaking that de-identification. This calls into question the long-term viability of existing legal standards.
HIPAA’s De-Identification Standards
The HIPAA Privacy Rule provides two distinct methods for rendering data de-identified, at which point it is no longer subject to the Rule.
The first method is the Safe Harbor approach. This is a prescriptive, rule-based standard that requires the removal of 18 specific types of identifiers from the data. These include direct identifiers like names, Social Security numbers, and email addresses, as well as quasi-identifiers like specific dates.
While straightforward, the Safe Harbor method is ill-suited for the unstructured, narrative-rich data needed to train a sophisticated LLM like the one in Dr. Keskin’s study. Stripping out all potential identifiers from the clinical vignettes used in the research could destroy the linguistic context and nuance essential for the model to learn, rendering the dataset useless for training.
The second, more flexible method is Expert Determination. Under this standard, a qualified expert must determine that the risk of re-identifying an individual from the remaining information is “very small“. The expert must consider the context, the recipient of the data, and the possibility of linking the dataset with other “reasonably available” information.
This risk-based approach does allow for the preservation of more contextual data than the rigid Safe Harbor method. However, as demonstrated by the complexity of the vignettes in Dr. Keskin’s research, it places an immense burden on the expert’s ability to accurately forecast the re-identification risk.
The LLM Re-Identification Threat
The core challenge to both de-identification methods is the unprecedented pattern-recognition and data-linking capability of LLMs. Even after the 18 Safe Harbor identifiers are removed, a wealth of “quasi-identifiers” often remains. A unique combination of a patient’s age, rare diagnosis, specific sequence of treatments, and the linguistic style of their clinical notes can create a distinct “fingerprint”. LLMs excel at recognizing these fingerprints and linking them across different datasets or with publicly available information to unmask an individual’s identity.
This threat is particularly acute for the kind of unstructured free-text data used in Dr. Keskin’s pilot study. The narrative details, idiosyncratic phrasing, and descriptions of unique life events within the clinical vignettes make them highly vulnerable to re-identification by a powerful LLM.
This calls into question the long-term viability of the “very small“ risk standard under the Expert Determination method. An expert can no longer confidently assess the risk when the anticipated recipient of the data is not a human but a superintelligent engine that can cross-reference the information against the entire internet.
Section III: Navigating Liability and Recourse
Liability
The failure of the LLM in the pilot study to recognize passive suicidal ideation provides a concrete, high-stakes scenario for exploring “who is responsible?” when clinical AI fails.
For the clinician on the front lines, the primary legal risk is medical malpractice. The core of a malpractice claim is proving that a healthcare provider breached the professional standard of care—that is, failed to act as a reasonably competent professional in their specialty would under similar circumstances—and that this breach directly caused harm to the patient.
A physician cannot delegate their professional judgment to an algorithm. They have a duty to critically evaluate an AI’s output and apply their own expertise. If a clinician were to use the LLM from Dr. Keskin’s study and, relying on its “false negative” response to the ambiguous vignette (”I wonder about death lately...”), failed to probe further or intervene with a patient who subsequently self-harmed, that clinician would be found negligent.
But while over-reliance on a flawed AI is a present liability risk, the future holds the opposite threat. As AI tools become more accurate and integrated into clinical workflows, the failure to use an available and effective AI tool could itself become a breach of the standard of care. This places clinicians in the difficult position of navigating when to trust, when to verify, and when to override the algorithmic recommendation.
The reality is that liability is rarely confined to a single actor. Healthcare organizations, such as hospitals and health systems, are a crucial link in the chain and face their own distinct forms of liability. Under the doctrine of respondeat superior, an employer can be held vicariously liable for the negligence of its employees.
In this context, the common technical safeguard of keeping a human in the loop to oversee the AI’s decisions is a critical safety and legal strategy. This allows developers to contractually position their AI as a mere “tool,” shifting the liability burden to the human operator.
However, as AI models become more complex and their reasoning more opaque, the ability of the human to provide meaningful oversight may diminish. (See my previous issue on how legal provisions for “human oversight” are becoming increasingly unreliable). Thus, courts may begin to look past the human and assign more direct liability to the developer of the “black box” system.
Recourse
The HIPAA Privacy Rule grants individuals several core rights regarding their health information, but these rights were conceived in an era of human-generated paper and electronic records, and they map poorly onto the probabilistic outputs of an AI model.
The Right of Access gives individuals the right to inspect and obtain a copy of their personal data maintained in a “designated record set”. This set includes medical and billing records and any other records used to make decisions about individuals. A strong argument can be made that an AI-generated suicide risk score placed in a patient’s file falls under this definition.
However, the scope of this right is unclear. Does it include only the final risk score? Or does it extend to the specific input data that led to that score, or even a summary of the algorithmic logic used?
The Right to Amend allows an individual to request an amendment to personal data they believe is inaccurate or incomplete. It is tricky to exercise this right with AI systems. A patient could dispute an AI-generated risk classification as inaccurate. However, the CE would likely defend the algorithm’s output as “accurate” based on the model’s programming and the given inputs. From a technical perspective, a request to “amend” a risk score is intractable; one cannot simply change the output for a single individual without altering the input data or the underlying model itself.
This highlights the need for new legal constructs, such as a right to algorithmic explanation or a right to contest automated decisions, to give patients genuine recourse.
Section IV: Recommendations
The current patchwork of HIPAA, FTC enforcement, and state laws creates confusion and uneven protection. Congress should consider a federal privacy law that establishes a consistent baseline of protection for all sensitive health information, regardless of whether it is held by a traditional hospital or a direct-to-consumer tech company. This law should adopt a modern, context- and inference-based definition of health data to ensure it covers AI outputs.
For technology developers, opaque “black box” AI is not sustainable in high-stakes clinical settings. They should embrace algorithmic transparency and explainability by, for example, providing healthcare partners with comprehensive documentation on model architecture, the characteristics of the training data and transparent performance metrics.
Healthcare organizations should ensure that the decision to adopt a clinical AI tool should not be left to IT or administrative departments alone. Healthcare organizations must establish multidisciplinary governance committees (including clinicians, ethicists, legal experts, and data scientists) to rigorously vet potential AI systems and go beyond a vendor’s marketing claims.
 | A guest post by
|