A confident paragraph with a fake citation

A new-hire medical laboratory scientist (MLS) student is asked to draft a short in-service note on why a potassium delta check might fire. The student's laboratory has one internally hosted, IT-approved large language model (LLM) tool running under a signed business associate agreement (BAA), so no patient data leaves the building through that tool. The student types a plain request: draft a short note explaining delta-check flagging, using the lab's own policy and citing the relevant CLSI guidance. No names, accession numbers, or reagent lot numbers go into the prompt.

The tool returns a clean, well-organized paragraph about hemolysis, intravenous-line contamination, and pseudohyperkalemia as causes of a delta-check flag. It also returns a specific citation, CLSI H21-A6, and a specific numeric threshold, 20% or 1.0 mmol/L, stated as if both were simply true. The prose reads like something a colleague would write.

Nothing about the output looks wrong. That is the problem: an LLM predicts the next plausible word given the prompt and its training; it does not check a fact against a source before it writes a sentence. A citation-shaped string and a number that sounds precise are not evidence that either one is real. The bench question is not whether AI assistance is allowed, it usually is for bounded tasks, but which tasks it may help with, what may be typed into it, and how the output must be checked before anyone relies on it.

Fluent, well-formatted text is not the same thing as verified text, and the difference only shows up when someone checks.