On October 1, Lee and colleagues from Seoul National University Hospital, Mayo Clinic, Weill Cornell, and NYU published VentPilot in the Journal of Intensive Care. The system uses offline reinforcement learning to recommend ventilator settings, learning from recorded patient trajectories and incorporating intensivists’ preferences.
The study included 4,296 adults: 3,002 to develop the system, 283 for internal validation at Seoul National University Hospital, and 1,011 for external retrospective validation at Mayo Clinic.
In a weighted retrospective comparison, patients whose care more closely resembled VentPilot’s recommendations had 5.0 more ventilator-free days in the internal validation cohort and 3.1 more in the external cohort. The larger figure comes from the smaller cohort. These were associations between observed care and outcomes. Patients had not been assigned to treatment guided by VentPilot.
The authors also used fitted Q-evaluation to estimate the return a policy would achieve from recorded data. That analysis favored VentPilot over observed clinician behavior. It is a separate result, expressed in the study’s reward measure, rather than the source of the 5.0-day difference.
A day earlier, Li and colleagues published a narrative review in Cureus examining digital twins and offline reinforcement learning for mechanical circulatory support. Their concern was the distance between estimating physiology and establishing that a treatment decision improves it.
They identify 3 recurring problems: prediction does not establish treatment benefit; evaluating a policy inside the simulator against which it was optimized cannot provide independent confirmation; and a retrospective record may contain information unavailable when the clinical decision was made.
Li’s review concerns another intervention and does not evaluate VentPilot. Fitted Q-evaluation is also not interchangeable with running a policy inside a physiological simulator.
The connection between the papers is the question they require us to ask about evidence.
Recorded care contains the clinician’s actions, the patient’s response, and only part of the reasoning connecting them. A setting may have been chosen because of a change the dataset captured poorly. Agreement with an algorithm may be more common in patients whose course was easier to manage. Statistical adjustment addresses measured differences; it cannot guarantee that every clinically consequential difference was recorded.
VentPilot’s authors acknowledge the retrospective limits and call for further clinical evaluation of safety, usability, and impact. The promise is substantial enough to deserve that next step.
My own group works on these methods. Learning from the decisions we already make is an important scientific opportunity, particularly in critical care, where treatment changes repeatedly as physiology changes.
The difficulty is knowing when we have learned a better decision and when we have learned the circumstances surrounding one.
Monday’s newsletter asked what an FDA-cleared test adds to care. Here, the same question arrives earlier, while a recommendation system is still being evaluated: what happens to patients when its advice changes what clinicians do?
So: what evidence would turn those associated ventilator-free days into a benefit we could reasonably expect from using the system?
Sources. Lee and colleagues, VentPilot · Li and colleagues, digital twins and offline reinforcement learning