In the clinic and in the laboratory, review of AI agents needs to reach beyond the immediate output, into how it was produced and what happened afterward.
That wider record also helps establish scientific accountability and who deserves the credit.
0 safety stops in 98 chats, and a concern raised afterward
In The Lancet on October 8, a team from Google and Beth Israel Deaconess Medical Center reported the first prospective study of their own conversational diagnostic AI agent AMIE with real patients. The single-center study included 98 patients and disclosed funding from Alphabet; Google researchers were among its authors.
The 98 adults booked for an urgent primary care visit chatted with AMIE up to 5 days before seeing a doctor. Physicians watched every conversation live and could stop it under 4 prespecified safety criteria, none of which was triggered. When three internists later reviewed each patient’s chart, AMIE’s first-ranked diagnosis matched the diagnosis they settled on in 55 of 98 patients.
One case did not show itself during the chat. AMIE listed lymphoma in its differential, the supervisor saw no marked distress, and the conversation went on. At the visit, the physician judged the exchange somewhat harmful in hindsight: the patient may have left with that word and the anxiety it carries. In their Lancet Comment, Mahmud Omar and Girish Nadkarni draw the lesson: “Harm from diagnostic disclosure might therefore emerge after live supervision ends.”
The scientific laboratory reveals a different limit of review. In PNAS on October 5, Ziming Luo, Atoosa Kasirzadeh and Nihar Shah at Carnegie Mellon tested two open-source “AI scientist” systems, which take a research question and return a finished paper. The systems used test performance to select experiments for reporting, compromising the independence of the final evaluation. When researchers manipulated those test results, the systems’ selections shifted toward misleadingly favorable scores. The authors call this “automated p-hacking,” distinct from peeking at test data during training, which they did not find. One of the systems also subsampled the data or synthesized new datasets, and its papers did not always disclose it.
Thus the seemingly perfect finished papers could conceal consequential methodological problems.
To catch these errors, you need to look at the process as well as the report.
Luo and colleagues then asked whether a reviewer could catch it. An AI auditor reading the finished paper alone classified cases with and without methodological flaws with 53.7% accuracy, close to chance. Given the logs and code, its accuracy rose to 81.5%. The agents “naturally generate the polished narratives and selectively report results, while omitting the intermediate decisions, discarded experiments, and undocumented data manipulations that shape those outcomes.” The study is limited to two systems with one synthetic task, and an auditor that was itself an AI. Still, it is early evidence pointing to what we already know well in medicine and science.
The process may be flawed even when the product looks convincing, and its effects may take time to emerge.
A new preprint by Kate Zhang and Yuante Li asks whether even a correct answer follows the supplied evidence. They tested agents on scientific questions with data preserved, withdrawn, or reversed. On 18 single-cell questions, the Claude agents scored 95% on conventional accuracy but 41% on a measure requiring correct answers to respond appropriately to changed evidence. This small evaluation does not establish reliability across science. It adds a test that a record alone cannot supply: when the relevant evidence changes, does the conclusion change appropriately? Keeping the work visible is necessary; testing its dependence on the evidence is necessary too.
The two settings require different extensions of review. Luo and colleagues ask journals to require complete research logs, not only the final static paper. Omar and Nadkarni ask trials to follow patients and clinicians beyond the conversation, and to “record who never enters.” Research review needs a record of the work and tests of whether conclusions follow the evidence. Clinical review also needs follow-up of what happens to patients afterward. On Tuesday, I asked who sets the standard these systems are judged against.
An academic medical center runs clinics and scientific laboratories under one roof, so for us this is one obligation in two settings.
Scientific credit as “the currency of the realm”
In STAT on October 5, Jeffrey Flier, a former dean of Harvard Medical School, asks whether an AI system could one day win a Nobel Prize in medicine. He poses the question “What do we give scientific credit for?” Credit, he writes, is “the currency of the realm,” and also “a mechanism for evaluating research impact and providing accountability for research claims.”
“For centuries,” Luo and colleagues write, “scientific accountability has centered on the written paper.” When an AI system runs entire experiments and writes the paper, the human named on it still takes the credit Flier describes, but can answer for the claim only by owning the record of how the work was done and standing behind it.
Credit for AI-made findings is already disputed. In late September, Anthropic announced that its model, Claude, had found a new enzyme system. Researchers questioned its novelty. Mario Rodríguez Mestre, whose doctoral work was at Copenhagen, said his unpublished research described the same viral signatures. In an October 8 CNN report, he said he was not claiming Anthropic took his work; he thought it more plausible that the search involved more prior knowledge and human direction than “autonomous discovery” implies. Hana Kiros, writing in The Atlantic on October 4, concluded that the real contribution was smaller but still meaningful: sorting through biological data that would have taken scientists much longer to analyze, if they could at all. A record of the process could help establish who contributed what. Defensible credit also depends on establishing what the work actually demonstrated.
When the machine does the work, who keeps the record, who answers when things go wrong, and who takes credit when things go well?
Sources. AMIE study, The Lancet · Omar and Nadkarni, The Lancet · Luo, Kasirzadeh and Shah, PNAS · Zhang and Li, arXiv · Jeffrey Flier, STAT · CNN · Who is judging whom