On September 24, in JAMA Health Forum, Joshua Sharfstein took up AI agents built to order tests, change prescriptions and, paired with robots, perform surgery. A former principal deputy commissioner of the FDA, he gives the regulator’s answer on oversight.

It depends, he writes, and his 6 first principles say on what.

His first two principles answer proposals to license these agents as if they were clinicians. An agent, he writes, is still “a technology product sold for profit,” and it can affect the health of more people in a single day than any licensed clinician.

So check promotional claims independently and optimize their conditions of use.

For clinicians, the principle that matters most concerns conditions of use. An agent that adjusts blood pressure medication, recommended by a physician, may earn a patient’s trust. The same technology, made broadly available outside an established patient-physician relationship, may carry greater risk. Nothing in the software changes, but the conditions of use do.

In parallel, under SB 903, a bill on Governor Newsom’s desk, an AI in California may score a standardized mental health screening questionnaire, but it may not decide what the score means. He has until September 30 to sign or veto it. Sharfstein never mentions the bill, but his principles make a strong case for it.

Clinicians and professional organizations, he writes, should work with developers to define the patients an agent serves, its relationship with human clinicians, and the limits of patient access.

The bill confines AI in psychotherapy to administrative and supplementary support. It names 5 things an AI may not do without a licensed professional’s review and approval: make therapeutic decisions; talk with patients in therapeutic communication; generate assessments, diagnoses, recommendations or treatment plans; detect emotions or mental states; perform triage or screening.

An AI may talk with a patient on its own if the FDA has authorized it for that use and it complies with HIPAA, but it still may not make the therapeutic decision. And it may score a standardized questionnaire and send the result to a licensed professional, but it may not use the score to judge the person’s condition or how urgently they need care.

The machine may talk, if the FDA says so. It may score, but it may not decide what the score means. This is Sharfstein’s principle of shared responsibility in legislative text: the FDA answers for the product, a licensed professional for the patient. The FDA sees where the line will be tested. Its discussion paper on generative AI devices, open for comment until October 19, notes that a conversational device may begin by informing and end by directing.

SB 903 does not license the machine but instead positions a licensed professional as one of the machine’s conditions of use. TechNet, citing a shortage of behavioral health workers, said in August, before the final amendments, that the bill puts “a clinician bottleneck” in front of intake and screening tools.

Sharfstein’s own example explains what the bottleneck is for and why it is needed. Engaging with a counseling agent, he writes, may go well in moderation and badly at more than 12 hours a day. The software cannot tell those two apart, but a clinician can.

Last Tuesday’s Standing Order read AB 2575, on the same desk with the same deadline. SB 903 makes the same move: when an employer requires the tool, the employer answers for how it is deployed.

“No individual or thing,” Sharfstein writes, “not even the most advanced AI model,” can predict the future. But he calls it a safe bet that agentic AI will not reach its potential unsupervised, on unsupported claims, or without attention to the conditions of its use.

So: who will set the “rules of engagement” for AI agents in your specialty? Will it be your professional society, a legislature, or a vendor’s terms of use?

Sources. Joshua M. Sharfstein, “Overseeing Agentic AI in Medicine From First Principles,” JAMA Health Forum, September 24, 2026 · SB 903 (Padilla), enrolled text · Bill history · FDA, “Considerations for the Regulation of Generative AI-Enabled Medical Devices,” discussion paper, docket FDA-2026-N-7874 · Ana B. Ibarra, CalMatters, August 6, 2026