AI’s Medical Advice: A Risky Proposition, New Study Warns
The allure of instant medical information is undeniable, especially in today’s fast-paced world. With the rise of sophisticated artificial intelligence, particularly large language models (LLMs) like ChatGPT, many are turning to these digital oracles for health queries. However, a groundbreaking study from the esteemed University of Oxford is sounding a serious alarm: relying on LLMs for medical advice and decision-making is a practice fraught with significant risks.
The research, which involved 1,300 participants, presented individuals with specific medical conditions developed by medical professionals. These participants were then divided into two distinct groups. One group was tasked with seeking medical advice and information directly from LLMs, while the other group consulted traditional sources, such as the internet and their own judgment. The findings revealed a concerning chasm between the capabilities of LLMs and the nuanced needs of individuals seeking medical guidance.
The Communication Conundrum
While LLMs demonstrate remarkable proficiency in understanding medical terminology and standard treatment protocols, the study highlighted a critical shortfall in their ability to effectively communicate and engage with users on complex health issues. Dr. Rebecca Payne, a lead medical practitioner involved in the study, stated in a press release, “Despite all the hype, AI just isn’t ready to take on the role of the physician.” She further emphasised the potential dangers, warning, “Patients need to be aware that asking a large language model about their symptoms can be dangerous, giving wrong diagnoses and failing to recognise when urgent help is needed.”
The research indicated that LLMs did not consistently provide superior outcomes compared to conventional methods of medical assessment, which include online searches and personal reasoning. A primary issue identified was the struggle for mutual understanding between the LLM and the user. The AI models frequently misinterpreted user queries, while participants often lacked the precise language or context needed to elicit accurate responses. These communication breakdowns significantly hampered the likelihood of the LLM delivering appropriate medical advice.
The Need for Rigorous AI Testing
A recurring problem observed throughout the study was the tendency for LLMs to generate a blend of accurate and inaccurate information. In the absence of professional medical guidance, participants found it challenging to discern between sound advice and potentially harmful misinformation.
Adam Mahdi, a senior author from the Oxford Internet Institute, described the observed gap between LLMs and patients as a “wake-up call” for both AI developers and regulatory bodies. He asserted, “We cannot rely on standardised tests alone to determine if these systems are safe for public use. Just as we require clinical trials for new medications, AI systems need rigorous testing with diverse, real users to understand their true capabilities in high-stakes settings like health care.”
A Growing Trend with Concerning Implications
Seeking medical counsel from LLMs is becoming an increasingly prevalent practice, particularly in countries like the United States where access to affordable healthcare can be a significant barrier. One study released in September by an AI platform revealed that over one-fifth of Americans admitted to following chatbot advice that was later found to be erroneous.
Adding to these concerns, a separate study published in June 2025 employed developer tools to investigate the LLMs’ susceptibility to providing incorrect information. The researchers discovered that it was remarkably easy to program these models to disseminate false data, with the chatbots confidently delivering inaccurate advice a staggering 88 percent of the time.
Natansh Modi of the University of South Africa, an author of this latter study, issued a stark warning: “If these systems can be manipulated to covertly produce false or misleading advice, then they can create a powerful new avenue for disinformation that is harder to detect, harder to regulate and more persuasive than anything seen before.” The implications for public health and the spread of misinformation are profound, underscoring the urgent need for caution and robust oversight in the deployment of AI in sensitive domains such as healthcare.
The study authors have been contacted for further comment via email.
