What may go in, what may come out, and who signs off

Data entry is the first gate. Under HIPAA, a cloud AI vendor that creates, receives, maintains, or transmits electronic protected health information (ePHI) on a covered entity's behalf is generally a business associate and needs a signed business associate agreement (BAA) before any ePHI reaches it, even if the data is encrypted and the vendor never holds the decryption key. A vendor is not automatically a business associate just for selling software; the deciding question is whether it accesses, maintains, or processes PHI while providing the service. A consumer-facing or public generative-AI tool without a signed BAA and defined data-retention and training-use terms is not an approved destination for PHI, proprietary methods, credentials, or confidential event data, full stop, regardless of how the prompt is worded.

Removing PHI, specimen-identifying detail, proprietary procedure text, and credentials from a prompt is necessary but not sufficient. The destination system's contract terms, whether a BAA is signed, what the retention period is, whether the vendor trains its model on submitted data, still determine whether entry is approved at all. 45 CFR 164.312 requires technical safeguards, including access control, audit controls, integrity controls, authentication, and transmission security, for any system handling ePHI, and a BAA under 45 CFR 164.504(e) must define permitted uses, require safeguards, require breach notification, and require return or destruction of PHI at termination.

Regulatory status is the second gate, and it depends on what the output does, not on the underlying technology. Under FDA's Clinical Decision Support guidance, a software function is excluded from the medical-device definition ("Non-Device CDS") only if it meets all four statutory criteria in FD&C Act section 520(o)(1)(E), including letting a healthcare professional independently review the basis for the recommendation rather than primarily relying on the output.

A generative-AI function that gives a specific diagnostic conclusion, a specific treatment plan, or a time-critical alert generally does not qualify as Non-Device CDS and may be regulated as a device software function. FDA also maintains a public, non-exhaustive list of AI-enabled medical devices cleared or approved through 510(k), De Novo, or PMA; check a tool's status on that list rather than assuming it either way. When a use case is genuinely ambiguous, FDA's own recommendation is to file a Q-Submission rather than guess.

Verification and change control are the third gate. Under 42 CFR 493.1253, a laboratory verifies a manufacturer's performance specifications for an unmodified FDA-cleared test system, or establishes its own performance specifications when a method is modified, laboratory-developed, or otherwise lacks manufacturer specifications; that verify-or-validate logic is the closest CLIA analog for a laboratory-adjacent software tool that touches results or workflow. CAP's AI Committee states that even an FDA-cleared, vendor-purchased AI tool needs full local validation before clinical use, because standard assay validation alone does not address clinical impact, edge cases, bias, or population drift at a specific site.

CAP requires validating autoverification logic initially and revalidating it whenever a software, algorithm, or data-element change could affect it, per CLSI AUTO10-A and AUTO15-Ed1:2019; no CAP checklist item names "large language model" directly, so this requirement is presented here as the closest available analog, not a direct requirement. Under 42 CFR 493.1407 and 493.1445, the laboratory director may delegate specific duties but keeps ultimate responsibility for confirming methodology is appropriate, performance is validated, and personnel are trained; that responsibility extends to any AI-assisted workflow the laboratory adopts.

FDA's 2025 final guidance on Predetermined Change Control Plans (PCCP) lets a manufacturer pre-specify bounded, evidence-based future changes to an AI-enabled device function without a new submission for each change, but a PCCP is not authorization for unrestricted retraining; it must define the boundary of allowed change, a modification and validation protocol, and an impact assessment.

Human review is the fourth gate, and it is not automatic just because a person looks at the screen. A systematic review of automation bias in healthcare decision support found that erroneous advice raised the odds of an incorrect clinician decision by roughly 26%, and clinicians overturned an already-correct decision to follow erroneous advice in about 6% of cases; those figures come from pre-LLM clinical decision-support studies, so their exact size for conversational LLM assistance is inferred, not directly measured, but the direction of the effect is a reasonable caution.

Mitigations that reduced automation bias in that review included requiring an independent initial assessment before viewing the AI output, showing the output's uncertainty or supporting evidence, and training reviewers with deliberately incorrect examples so they practice catching an error. ISO/DIS 24051-1, a draft international standard on AI in medical laboratories, remains unpublished as of this writing (ballot stage 40.92, referred back to committee); it is not an accreditation requirement and should be treated as emerging reference only, not cited as binding.

A lower-risk task, an approved destination with a signed BAA, and a documented human check are three separate yes/no questions, and a task only clears for use when all three answers are yes.