Required section · Section 2 of 6
What an LLM actually does, and what governance looks like
An LLM is a text-prediction system. Given a prompt, it generates the statistically likely continuation, token by token, based on patterns in its training data. It has no built-in mechanism to check a fact against a source, run a calculation and verify the result, or know the current content of a specific controlled document. When it produces a citation, a number, or a procedural step, that output is a plausible continuation of the prompt, not a lookup result, unless the tool is explicitly connected to a retrieval system that grounds its answer in named documents.
The National Institute of Standards and Technology (NIST) names the risk of confidently generated false or ungrounded content confabulation in its Generative AI Profile (NIST AI 600-1), and lists it alongside information integrity, information security, data privacy, and human-AI configuration, including automation over-reliance, as risks that generative AI use in an organization must be governed, mapped, measured, and managed. NIST's earlier AI Risk Management Framework (AI RMF 1.0) organizes that work into four functions: Govern (set policy and accountability), Map (identify context and risk), Measure (test and monitor performance), and Manage (respond, and correct course).
A 2026 policy brief from the Association for Diagnostics & Laboratory Medicine (ADLM) recommends applying laboratory quality-system thinking to healthcare AI governance: a defined intended use, independent validation before deployment, continuous monitoring for drift and bias after deployment, and a named human who is accountable for the tool's use. That framing should feel familiar; it is close to how a laboratory already treats a new analyzer or a new method, extended to a tool that predicts text instead of measuring analyte concentration.
The process map below lays out the four-function governance cycle at the level a bench scientist needs: know what the tool is approved for, know what you may put into it, check what comes out of it, and know who is accountable if it drifts or fails. Treat an LLM's output the way you would treat an unverified result from any new tool, plausible until confirmed.
Illustrative drawing — this picture was drawn rather than captured.
NIST AI RMF 1.0's four functions, applied to a laboratory's use of an approved generative-AI tool.
Govern
The laboratory or its parent organization decides which AI tools are approved, under what contract terms, and who owns that approved list; this is documented local policy, not a fact fixed across every lab.
Map
Before a task is handed to the tool, classify it: what data would the prompt need to contain, and where does the task sit on the risk ladder from drafting an outline to interpreting a specific patient's result?
Measure
Every output is checked against a primary source or a local controlled document before use; for a tool integrated into a workflow, performance is monitored at intervals against a data set with known-correct answers, because performance can drift after deployment.
Manage
A named person, often the laboratory director or a delegated informatics or quality role, keeps responsibility for the tool's use, holds a rollback plan, and runs an incident-response path when an output disagrees with a controlled document.
Knowledge checks
This section has no knowledge checks.
Section status
Finish this section
Reading and checks are open. Sign in only to save.
The module finishes after every required section is marked done and every check in those sections is correct.