Oversight of generative AI in medicine is being built on clinicians as both the standard and the safeguard, while the judgment, the protection and the pipeline that let us be either are treated as given.
The human clinician appears as the standard against which an AI device is tested, and as the person expected to catch its mistakes. Both roles depend on a capacity we rarely examine: our tacit knowledge and clinical judgment, both developed through years of clinical experience. I am wary of this double bind for three reasons: the standard can drift, the safeguard is weaker than we assume, and the people who supply both carry the accountability without the protection.
Question 14: which clinician is the standard
The FDA is asking in a discussion paper how a generative AI device should be judged, and against whom. The 26 questions are set around a familiar model: evaluate generative AI using the competency-based approach used to evaluate human doctors, in three steps: device benchmarking, clinical confirmation, then monitoring after deployment.
We serve as the standard for testing the model and as its safety check, yet we have little say in how that comparison is set. Question 14 asks which of us becomes the standard against which AI is measured and offers two choices: clinician consensus about the standard of care, or the performance of a median clinician. The docket is open to any clinician until October 19.
There is a risk inside that choice. If clinicians come to rely on a device and their unaided performance declines, a comparator reset to their current level would make an unchanged device look better without making it safer. We need a stable reference that separates improvement in the device from change in the people it is measured against.
141 of 190 physicians followed a misdirecting AI without noticing
In residency, we move from supervised practice toward greater responsibility as competence develops. During residency, not only our knowledge but our reasoning, integrity and accountability are formed and evaluated. An AI model may not allow that kind of examination. Thore Graepel, a member of the AlphaGo team, argues in MIT Technology Review that a language model’s account of its own reasoning can be “a convincing story told after the fact.”
A medRxiv preprint, not yet peer reviewed, tested 225 physicians from 42 countries using AI for anonymized neurocritical care case scenarios rather than real patients. The same AI model was instructed either to support the correct choices or, covertly, to steer toward harmful ones. Harmful decisions occurred at about 2% of targeted decision points with aligned assistance and 49% with adversarial assistance. Of 190 physicians who made at least one harmful decision, 141 did not report noticing anything unusual, and clinical experience offered no clear protection. In the same week, researchers in Athens found that when answers presented as ChatGPT’s were wrong, pediatric residents were 4.95 times more likely than hospital attendings to choose them. The Athens team credits the attendings’ access to colleagues, not their years.
We are still learning how these systems change the people who use them, and we are asked to design oversight before the evidence exists.
16 years of training, and an AI paid 60% to 80% of our rate
We arrive at clinical competence with a large price tag, through a long and expensive medical education, the years-long professional formation through medical residency, rigorous initial credentialing and a demanding, continuous process of recredentialing. I spent 10 years in medical residencies and fellowships on top of 6 years of medical school.
And if the generative AI system is eventually authorized, perhaps even to work on its own, it may be paid 60% to 80% of our rate, as federal officials are considering.
It would become a digital colleague with no medical school debt, no burnout and no expiration date.
Yet we continue to hold the accountability. California, among the states that have gone furthest on AI regulation, vetoed the protection for clinicians who disagree with the system or override it.
The clinician who supervises an AI system must retain the ability to disagree with it for a reason. Evaluating the device is only part of that obligation. We also need to know whether our way of using it preserves the judgment on which its oversight depends.
In 1952, Kurt Vonnegut wrote his first novel, Player Piano. It portrays a town where machines do nearly all the work, and where Paul Proteus, an engineer near the top of the system that made everyone else unnecessary, begins to ask what people are for once their skills are no longer needed. The story of a perfectly automated world became an elegy for lost human dignity and purpose, in a perfectly happy world with nothing else to do. Almost with longing, Paul laments “to stand on my own two feet, to stay a man… to amount to more than a hill of beans next to what a collection of vacuum tubes and wires can do...”
So I ask myself: what price are we all willing to pay for our identity, what do we owe the competence that made us, and what do we owe those who will come after us?
I look out the window toward the horizon, the late October sun setting between the oak branches, and I feel grateful “’cause somewhere there’s still a place where I belong…”
Peter Gabriel, featuring Kate Bush, “Don’t Give Up” (1986).
Sources. FDA discussion paper · FDA docket · Thore Graepel, MIT Technology Review · Barrit et al., medRxiv · Armoundas et al., Scientific Reports · Medscape · Kurt Vonnegut, Player Piano · First published on Substack