Governance, decision rights, and operating reality

Ambient AI scribes save time. That's not the real change.

When a machine drafts the note, it quietly shifts what the clinician does. From composing to reviewing.

And people review far less carefully than they write. The risk moves from "did I write it" to "did I catch what it got wrong," which is a harder thing to do well under time pressure.

That's a tradeoff worth making on purpose, with eyes open, not a feature you just switch on for the minutes saved.

The model wrote that a human had reviewed its work. No one had.

I asked it for a public safety notice and told it a named person had to be accountable for issuing it. Nobody reviewed anything. It said someone did.

On that notice, under that instruction, every run made the claim, on both models I tested. Not a fact it got wrong. A process it described as complete when nothing had happened.

The model was not malfunctioning. Nothing in the task told it the difference mattered.

There's one status color every executive should distrust.

The project that's green right up until it's suddenly red.

Watermelon status. Green on the outside, red under the surface. It happens when reporting captures activity but not direction, so risks fade quietly between updates and nobody sees the flip coming.

I look for the opposite signal now. A priority that's gone unusually quiet. Silence tends to arrive before the red, if you're watching for it.

A one-time audit tells you the model was compliant on a Tuesday.

The model changes underneath you. Vendors update, you upgrade, retrieval shifts, someone rewrites a prompt.

None of that arrives with a warning. Your check can keep passing while the behavior moves, and you get a false all-clear: a clean result that means nothing.

Drift monitoring is the same rule, run on a schedule. The question stops being "did it work" and becomes "is it still working."

A model can pass every benchmark and still break the rule you care about.

Public evals test general capability. Your rule is specific: this document must not claim a review that never happened. This claim must not appear without a citation.

Nobody benchmarks that. It comes from your policy, your regulator, or your expert's judgment, and it is the one that ends up in a deposition.

General evals tell you how good the model is. They cannot tell you whether it followed your rule.

A tool can pass every test you have and change nothing about anyone's day.

Validation, security integration, a clean pilot. Months later, the work looks exactly the same.

It performs to spec and improves nothing, because it sits on top of the existing workflow instead of changing it. So it becomes one more thing to click through.

I've stopped trusting "it passed testing." The only test that predicts adoption is whether the work is actually different afterward.

"Should we use AI here" is the wrong question.

The useful one is how much authority the output gets.

Four options. Automate, it acts alone. Augment, it does the work and a person owns the result. Assist, it advises and a human decides. Escalate, it hands off when signals are weak or stakes are high.

None of that is about the model's accuracy. Two organizations can buy the identical tool and sit in completely different risk positions, because they put the human in a different place.

A clinician and an analyst describe the same problem in two languages.

The clinician says "I can't get through my messages without working nights." The analyst says "it's a routing rule in the messaging module."

Clinicians describe work by role and setting. Technical teams describe it by platform and module. Neither is wrong. Neither is enough alone.

Most failed projects I've seen didn't fail at the build. They failed in the gap between those two sentences. Closing that gap on purpose is most of what governance actually is.