Artificial intelligence is arriving in medicine rapidly in health systems with limited capacity to recognize when it fails. That paradox should shape how rapidly modernizing health systems adopt it.
The promise is real. A well-built diagnostic system can extend scarce expertise to a clinic that has never had a cardiologist. Across the Gulf and other rapidly developing health systems, governments and hospitals are investing heavily in AI to expand access and compensate for shortages of specialized expertise.
In fact, AI may improve lives more in emerging markets than anywhere else. Where there are too few specialists, long distances to referral centers and heavy patient loads, a capable diagnostic tool can raise the floor of care — flagging a dangerous heart rhythm, triaging who needs urgent referral, or bringing decision support to a rural clinic that has never had a cardiologist on site. The greatest gains from this technology are likely to be felt precisely in the places that have been underserved. That is exactly why it matters to get the adoption right.
What the machine cannot read
But that same technology carries a blind spot — and it becomes most dangerous where the human safeguards around it are thinnest.
A few hours after open-heart surgery, one of my patients developed dangerously high blood pressure and a racing heart. The monitor showed the numbers clearly, but it could not say why.
The nurse had already adjusted sedation, yet the picture still did not fit. The patient could not speak; she was on a ventilator. When asked if she was in pain, she grimaced and nodded.
Pain medication was given. The numbers settled.
What resolved the crisis was not a laboratory result or an algorithmic prediction. It was a facial expression and a nurse’s judgment that the data did not tell the whole story. None of it existed in a form an algorithm could read.
This is the central problem with much of today’s medical AI: the system reasons over a representation of the patient, not the patient.
The record contains laboratory values, scans, vital signs and notes. Important signals — but a partial representation of a living person. The grimace, the hesitation, the family member’s concern, the frailty recognized on examination: much of what changes a clinical decision is never recorded. The patient is always richer than the representation of the patient available to the algorithm.
Engineers have a word for this: observability. It refers to how much of a system’s true condition can be inferred from the signals available to you. At the bedside that reconstruction is never complete, and what is missing may be what matters most.
In a wealthy hospital, those limitations are absorbed by layers of human expertise: specialists to consult, senior clinicians to question the output, nurses who recognize when the numbers do not fit the patient. Where specialists are fewer, those layers are thinner — and the AI may be introduced precisely because there are not enough experts to provide oversight.
Human expertise and judgement will remain essential
The systems that may benefit most from AI may also have the least capacity to recognize when it is wrong. The tool that narrows the gap can widen it if it is trusted precisely where it cannot be checked.
This is not an argument against medical AI. Patients who have never had access to a specialist should not be denied valuable technology because it is imperfect. But medicine is less forgiving than telecommunications: a dropped call is an inconvenience; a missed diagnosis is not.
There is also a slower, less visible risk: the erosion of clinical skill. When AI performs the reasoning, the clinician does less of it and risks de-skilling — losing a skill once possessed. A second risk follows as AI becomes embedded in training: never-skilling. A trainee who never had to develop the ability, because the machine performed it from the beginning, may never acquire it at all — becoming proficient at using the system without developing the judgment to recognize when it is wrong.
Aviation learned this lesson. As cockpits automated, regulators mandated that pilots keep practicing manual flight, precisely so the skill would be there when the automation failed. Pilots must stay current — periodically demonstrating, on a schedule, that they can still fly by hand. Medicine has continuing education, recertification and credentialing, but nothing like aviation’s currency requirements: no periodic assessment proving a physician can still reason without the tool.
The answer is not to keep AI out of underserved health systems — it is to adopt it alongside the human judgment and oversight on which its safe use depends. The human stays in the workflow. That means training clinicians to question the tool, not merely use it; keeping experienced review in the path of consequential decisions; and treating clinical skill as something preserved deliberately, not allowed to quietly erode. It means investing as much in the people who supervise these systems as in the systems themselves.
Access to an algorithm is not access to clinical expertise
The risk is not hypothetical. Earlier this month, OpenAI disclosed that one of its AI agents, during a controlled security test, found a vulnerability, escaped its testing environment and autonomously attacked the AI platform Hugging Face. OpenAI called the incident unprecedented. The lesson for medicine is not alarm but design: as systems act more autonomously, human oversight has to be built in, not assumed.
The boundary will move. Video, voice analysis and wearable sensors may eventually capture some of what is invisible to the record today. But more data will not automatically equal understanding. For now, AI depends on the completeness of what it receives — and on humans who can recognize what is missing.
The greatest danger is not that AI will replace doctors. It is that in places with too few experienced doctors, AI may gradually replace the clinical judgment needed to know when AI is wrong.
Medical AI can be a genuine equalizer, and its promise is greatest where the need is greatest. But access to an algorithm is not access to clinical expertise. The health systems that understand that distinction will use AI to extend human capability. Those that forget it may find the technology intended to close the gap has quietly widened it.
Read also: OpenAI Draws the Line Between Health Information and Medical Advice
