◆Healthcare
A pilot that clinicians actually like is rare enough that the instinct is to scale it immediately. Resist that instinct for two weeks. The distance between a good pilot and a system safe to run every shift is a short list of governance artifacts, and every one is cheaper to build before go-live than after the first incident.
During the pilot, safety came from novelty. Everyone knew it was an experiment, so every output got a second look. The project champion was watching the queue daily. If something looked wrong, someone said so in the hallway and the prompt got fixed that afternoon.
Production removes all of that. A nurse on a night shift six months from now will treat the output as routine, because by then it will be routine. The checking that happened by culture has to move into structure: a named owner with the authority to pull the system offline, a rollback that has actually been rehearsed, review steps that fire because the workflow requires them and not because someone remembered.
The single most valuable artifact you can build in the transition is a frozen evaluation set: a few hundred real cases from the pilot, handled according to your privacy process, with a documented correct output for each and a record of which clinician decided what correct meant. Include the ugly cases deliberately. The ambiguous medication list, the conflicting progress notes, the record with a decade of history, the case the pilot got wrong.
This set becomes the regression contract for the system. No prompt change, no retrieval change, no model update ships until it runs the set and someone signs off on the comparison. It is also the artifact your compliance committee will actually read, because it answers their question in their language: here is how we know, in writing, what this system does with hard cases.
Model providers ship new versions and retire old ones. That is normal, and a health system needs a normal procedure for it rather than a scramble. Pin the model version in production. When a new version arrives, run it against the frozen evaluation set in a staging path, document the differences, and promote it deliberately or decline it with a written reason. Keep the prior version callable and rehearse the switch back before you ever need it.
Widen the lens slightly: the system is not just the model. It is the specific combination of model version, prompt version, and retrieval corpus version, and any of the three can drift. Log which combination produced every output. When someone asks in eight months why the system said what it said on a Tuesday in March, you should be able to reconstruct the exact configuration that ran, not approximate it from memory.
Not every output needs a clinician's signature, and pretending otherwise guarantees the review step gets rubber-stamped into meaninglessness. Tier by consequence. Internal drafting and staff-facing summarization can run with sampled spot checks and a clear escalation path. Anything that reaches a patient, or feeds a clinical decision, goes through a named reviewer before release, every time, with no bypass.
Then design the review screen so verification is fast. Put the cited source passage from the record directly beside the generated text, so the reviewer confirms rather than rewrites. A review step that takes ninety seconds gets done. A review step that requires opening three systems gets skipped, and a skipped control is worse than no control because it shows up in your documentation as protection you did not actually have.
Write down what counts as an incident before one happens. A fabricated citation. Patient data surfacing somewhere it should not. An output released past the review step. For each, define severity, who is paged, who has authority to roll back or shut off, and what gets written up afterward. This fits on two pages. The point is that at 2 a.m. nobody is improvising.
The audit trail is what makes any of this defensible later. Every output should be logged with its inputs, the sources retrieved, the model and prompt versions, the reviewer if one was required, and timestamps. That log is how you answer a compliance question in minutes instead of weeks, and it is the difference between an engagement designed to support HIPAA obligations and one that merely hopes to. This is the territory our Model Governance & Audit Pack covers, and a Governed Pilot builds these artifacts from day one rather than retrofitting them after go-live.
The pilot proved the workflow is worth running. The artifacts above prove it is safe to run when nobody is watching, and that second proof is what your compliance office, and your patients, are actually owed. The free Claude Readiness Assessment is a reasonable place to find out which pieces you already have.