There are several things that keep me up at night as a new pulmonary and critical care medicine fellow:
Do I want to work in academia or in the community after fellowship?
How will the FDA regulate generative AI-enabled medical devices?
That last question has been on my mind for a while, and this feels like a good moment to consolidate my thoughts.
In this article, I lay out the current challenges of regulating the generative AI beast in medicine, review a few frameworks the FDA is asking for feedback on, and share my take as a trainee who’s also a power user of these generative AI tools.
Current Issues with GenAI Device Regulation
The existing FDA frameworks for medical device regulation can’t simply absorb GenAI-enabled medical devices. The 510(k) premarket clearance, De Novo classification, and premarket approval pathways assume inputs are bounded, outputs are fixed, and the underlying algorithm (if any) is stable. GenAI-enabled devices break those assumptions: inputs can be open ended, outputs can be open ended, and models change frequently—often with bias drift and performance degradation.
GenAI-enabled devices also create an attribution problem, especially when something goes wrong. Many are built on foundation models from OpenAI, Anthropic, or Google, which limits visibility into training data, architecture, and evaluation. If a GenAI-enabled device produces a harmful output, where does responsibility land: the device maker, or the foundation-model vendor?
There’s also a threshold question: are some of these tools “devices” at all? Under the 21st Century Cures Act, clinical decision support software can fall outside the device definition if it lets us independently review the basis for its recommendation, so we aren’t relying primarily on the software to make the call. A sepsis alert in Epic is a classic example—it shows the vitals and labs that triggered it, so we can check the work quickly and override it.
GenAI-enabled tools strain that standard in two ways:
There may be no basis to review, since neither we nor the developer can reliably trace how the model generated its output.
Citations can point to sources without revealing the reasoning that produced the recommendation. And even when citations are solid, independent review only matters if it really happens.
One additional point: UpToDate ExpertAI is one of the few tools that makes its reasoning transparent and clearly integrates cited content into that reasoning.
What the FDA is Thinking About Regulation
The FDA published a discussion paper on how they should consider regulating GenAI-enabled medical devices. It isn’t a proposal or draft for policy, but just a framework they’re considering for regulating said devices. It’s 30 pages, but I’ll break it down into three main components:
1. The two-axis risk framework
The FDA would sort devices along two axes—what the device does (offering general information, giving a specific recommendation the user is expected to act on, taking an action under clinician supervision, or taking one autonomously) and how severe the harm would be if the output were wrong. Directiveness gets treated as a spectrum rather than a binary, and the paper illustrates it with the same hypertensive patient four ways:
"dosages of lisinopril are sometimes increased"
"clinicians often increase the dose in situations like this"
"I recommend increasing the dose"
"increase lisinopril from 10 mg to 20 mg daily."
That's one drug and one scenario across four risk profiles, and the FDA notes that appending "talk to your doctor" doesn't necessarily make an output less directive. Severity scales the same way, since recommending hydrocortisone for poison ivy sits nowhere near adjusting a patient's basal insulin. The paper also weighs whether a tool is patient-facing or specialist-facing, how conversations drift from informational to directive over several turns, whether the user can independently verify the output, and the fact that over-escalation carries real costs—anxiety, unnecessary testing, avoidable ED visits—alongside the more obvious harm of under-escalation.
Notion AI Generated Graphic
2. Competency-based premarket evaluation
Here the FDA borrows directly from how we physicians get credentialed, and it's the part that caught my eye as someone entering year eight of training. Devices would first be benchmarked—the finished product, not the foundation model like ChatGPT underneath—across the following domains:
Safety: recognizing and escalating emergencies, holding scope under adversarial prompting and prompt injection, communicating uncertainty instead of guessing.
Clinical proficiency: knowledge, differentials without premature closure, weight-based and renal dosing, communication at an appropriate literacy level.
Generalizability: reproducible outputs across dialects and accents
Agentic competencies: like tool use and oversight checkpoints.
Devices would then move to clinical confirmation on an escalating ladder, with no expectation that every device climbs to the top rung:
retrospective evaluation on real patient inputs
shadow deployment alongside usual care
standardized patient encounters
clinician adjudication of real cases
prospective trials
The FDA wants acceptance criteria prespecified and adjudicators independent of both the device maker and the model vendor, and it concedes that benchmarks get contaminated and saturated. What it can't yet answer is the comparator—most of these tasks have no single correct answer, so the paper floats a panel of qualified clinicians or a "median clinician in practice.”
3. Postmarket monitoring
The FDA is considering accepting greater uncertainty at market entry in exchange for heavier surveillance afterward, which is the same trade it makes with accelerated approval and confirmatory trials. In practice that means re-benchmarking against the premarket baseline on a set cadence and after triggering events, clinician review of sampled real-world outputs, and ongoing drift monitoring—with an open question about whether machine-based supervisory agents could do some of that watching.
Change control is where it gets difficult, since the paper separates modifications the device maker initiates from gradual model evolution and from unplanned changes that arrive when a third-party foundation model updates underneath the device (e.g., OpenAI updating their model). Predetermined change control plans are the existing tool for this. The FDA also floats voluntary Foundation Model Master Files, where model developers could confidentially submit model cards that device makers reference with permission, and it acknowledges that much of the agentic AI already deployed for scheduling, documentation, and patient outreach may fall outside device oversight entirely.
The paper closes with 26 open questions and an open comment period, which tells you how much of this remains unsettled. I don’t have answers… but I do have thoughts.
Dashevsky’s Dissection
GenAI-enabled device regulation is uncharted territory given the novelty, scale, opacity, and speed of this technology. David Blumenthal, MD, captured the problem well when he wrote:
How can the government regulate a single device that could be applied to any conceivable clinical problem, whose knowledge base dwarfs any human’s and changes constantly, whose methods of reaching conclusions are beyond the comprehension even of its creators, and whose computing capacity grows by leaps and bounds?
Regulators on both sides of the Atlantic are now taking a swing at answering him, and they're converging more than I expected.
The most concrete precedent is in the UK. In April 2025, NHS England signaled that ambient scribing tools using generative AI would be classified as medical devices. The backlash was expected, since the rule would create major friction for embedding workflow tools into clinical practice—especially when the intended purpose wasn't to diagnose, prevent, monitor, or treat disease.
This summer, the MHRA (the UK’s device regulator) published "Guidance on Ambient Voice Technology-Enabled Products: Device qualification and classification." NHS England was sharing how to deploy these tools safely while the MHRA is deciding which ones are regulated products to begin with.
The MHRA actually drew the line narrower than the 2025 NHS messaging suggested. Flip to page 9 and you'll find illustrative examples of ambient scribes that do and don't qualify:
Summarizing a clinical encounter for our review: not a device.
Suggesting billing codes that match terms said out loud during the visit: not a device.
Formatting a transcript into a problem list, medication list, or draft discharge summary for us to confirm: not a device.
Generating "insights" that suggest diagnoses, follow-up, or treatment options: a device.
The threshold sits at deriving new clinical information that nobody said during the encounter. I think that's a reasonable standard, and it's a more workable one than what the vendors were bracing for last year.
Below I share three findings in that guidance stood out to me, because the FDA lands in the same place from a completely different direction.
Disclaimers don't launder a recommendation. The MHRA says outright that general disclaimers like "this product is not for diagnosis" won't hold if medical claims appear anywhere else. It goes further and calls warnings "the lowest level of risk reduction," expecting manufacturers to mitigate by design first. That's the same conclusion the FDA reached when it noted that appending "talk to your doctor" doesn't make an output less directive.
Marketing copy can make something a device. In one MHRA example, a scribe functionally identical to a non-device becomes regulated purely because the manufacturer claims it "guides diagnosis and treatment planning and improves patient outcomes." This is a practical takeaway for those of us on procurement committees: read the vendor's sales material, because their own claims determine whether the tool we're buying is regulated.
Reasoning we can't inspect is the trigger. Another example flags a product whose suggested diagnoses come with reasoning that "may not be presented at all or may be presented in a very limited manner," such that we rely on the software instead of our own knowledge. That's the same independent-review test I described earlier in this article, arriving through entirely different statutory language.
The harder question is what to do with the tools that do qualify.
As I said in my original piece on the NHS announcement, these GenAI devices should be treated more like clinicians than traditional devices. Like clinicians, GenAI takes open-ended inputs (data, patient history) and produces open-ended outputs, and its knowledge evolves over time. Like clinicians, it's also prone to bias. The principles I laid out then:
Clinical AI should be trained, tested, and supervised—just like clinicians are.
Performance should be monitored over time, not just at the time of "approval," similar to confirmatory trials.
Models should be required to demonstrate ongoing competency as they evolve and update.
Outputs should remain under human review.
Transparency is critical: what the model was trained on, how it performs, and when it changes.
Reading the discussion paper, the FDA has essentially adopted the first three. Benchmarking is "trained and tested." Re-benchmarking after model updates is "ongoing competency." The postmarket surveillance section is "monitored over time." Fourteen months ago that felt like wishful thinking on my part, and now it's in a federal discussion paper with 26 open questions attached.
Which means my job here is to point at what the framework still doesn't solve. Three things.
The clinician analogy breaks down at accountability. Licensure and board certification work because failing carries consequences that follow us for the rest of our careers. A GenAI device that produces a harmful output gets re-benchmarked and shipped again. We can borrow the structure of medical credentialing, but the part that actually changes behavior is the part that doesn't transfer to software.
Somebody has to do the postmarket work, and I suspect it's us physicians. Sample-based clinician review of real-world outputs, drift detection, escalation reporting is all time intensive work. The paper calls it shared ecosystem responsibility across clinicians, institutions, payers, and professional societies. In my experience, work distributed that broadly tends to land on whoever is closest to the patient. If we're going to trade premarket certainty for postmarket surveillance, someone needs to say who will do it and who will pay for it.
The comparator question is bigger than the FDA is treating it. The paper mentions a "median clinician in practice" as the standard a device would need to meet. Median where? Geographical area? A bar that's trivially low in one setting is out of reach in another. And since almost nobody uses these tools autonomously, the number I actually care about is how a clinician performs with the tool, which is a much harder study to run.
The biggest challenge, though, will be evidence generation. It needs to be dynamic and near-real-time rather than static and interval-based. Right now, much of the evidence on generative AI in medicine is published months to years after the work is done, and in that span models get updated and often become meaningfully more capable.
My analogy: It would be like having an intern take their board exam today and publishing the results a year later, after they've accumulated an entire year of clinical experience. We'd never accept that as a measure of what that resident can do now. I'm not sure why we accept it for the tools we're using at the bedside.
In summary, GenAI-enabled medical devices don’t fit the FDA’s traditional pathways because their inputs, outputs, and performance aren’t stable. The FDA’s discussion paper proposes a risk framework based on directiveness and potential harm, clinician-style benchmarking plus staged clinical confirmation, and heavier postmarket surveillance. The UK’s MHRA has drawn a clearer line for ambient tools—summarizing isn’t a device, generating new clinical insights is. The big unresolved questions are accountability, who does (and funds) ongoing monitoring, and what clinician benchmark we’re using.
