By Ruhi Raghavan
The integration of artificial intelligence with medicine and diagnostics is often praised for its potential to revolutionise healthcare, promising diagnostic accuracy and efficient resource allocation. AI applications are expanding in nearly every clinical setting. This rapid technological shift brings a complex web of ethical, legal, and social risks. As we move towards a future where AI is present across all systems, we must acknowledge the potential patient harm, algorithmic bias, and lack of transparency if we hope to strike a balance between technological optimism and the fundamental human right to autonomy and dignity.
Transparency, The ‘Black Box’ and Informed Consent
One of the most significant ethical hurdles in medical AI is the “black box” phenomenon. The complex reasoning of machine learning (ML) algorithms, such as neural networks, remains impossible to understand. When a physician relies on a recommendation they cannot fully explain, it treads the edge of informed consent. Consent is based on the patient’s right to decide among alternatives on an informed basis. If a physician uses AI to recommend a certain treatment without disclosing their involvement, they risk flouting the patient’s right to self-determination. Informed consent demands that a patient understand the risks, benefits, and limitations of the actions taken. A major challenge is determining how much granularity the explanations require: whether patients need to understand the underlying engineering or if knowing about the strict regulations and empirical support is sufficient. One possible solution is a framework that employs physicians as “technological interpreters” who explain the principles behind AI to build trust with patients, without detailing the specific mathematical processes.
Fairness and Bias In the Training of AI
AI models are only as good as the data used to train them. The reality is that medical data is notoriously incomplete, noisy, and imbalanced. Human biases relating to sex, age, race, and socioeconomic status tend to find their way into AI models, perpetuating or even amplifying existing inequities in healthcare. For example, an algorithm used in the US to track population health turned out to discriminate against black patients because it associated healthcare costs with need, failing to account for racial disparities in spending. Several analyses have demonstrated that AI solutions targeted for the general population often fail marginalised populations. For example, diagnostic models that have demonstrated high overall accuracy tend to perform worse for black patients and women. Along the same lines, AI models for skin cancer detection are largely composed of fair-skinned samples. This makes them less effective for people of colour. To mitigate this issue, it has been suggested to integrate subgroup-level evaluation and document training data distribution to identify people or groups for whom the system risks underperforming.
Accountability for Machine Errors
The question of accountability for the algorithms is the most pressing legal concern in modern medicine. In cases where the system fails to diagnose a life-threatening condition or generates a false positive, the allocation of liability is less than ambiguous. This becomes a multi-actor problem; it is difficult to identify who should bear the burden when developers, data managers, hospitals, and clinicians have all participated in the diagnostic process.
Several theories of liability are currently under debate:
- Medical Malpractice: The physician might be held accountable if they used the AI improperly or ignored their own clinical judgment when warranted.
- Product Liability: The developer could be held strictly liable if the AI tool has a defect in its configuration or design.
- Vicarious Liability: Hospitals could be held responsible for the malpractice of their employees or failure to review an AI tool’s reliability.
- AI Personhood: Some suggest conferring legal personhood to AI devices, allowing patients to sue the system directly.
Ultimately, the consensus favours a “common enterprise model” where physicians, manufacturers, and hospitals are held jointly liable, creating a shared sense of responsibility that incentivises all involved to prioritise safety.
Protecting Human Autonomy
The World Health Organisation (WHO) stresses that human autonomy must remain at the heart of healthcare. There is a risk that “techno-optimism” could lead to decisions being transferred entirely to machines, displacing humans from the centre of knowledge production. Autonomy in this context means that humans should remain in full control of healthcare systems and medical decisions. Furthermore, AI comes between the trust-based doctor-patient relationship. While AI is expected to handle mundane administrative tasks, freeing up time for doctors to listen to patients, there is a risk that clinicians become overly dependent on automated outputs and lose their skills. We must work to ensure that AI remains an assistive tool rather than a substitute for human empathy.
Data Privacy and Cybersecurity
AI’s reliance on big data risks privacy and security. High-profile incidents, such as the transfer of 1.6 million patient records from the NHS to Google DeepMind without explicit consent, highlight the vulnerability of sensitive health information. Beyond privacy breaches, medical AI systems are susceptible to malicious cyberattacks. Ransomware attacks on hospitals can interfere with patient records, and researchers have even demonstrated that AI-powered insulin pumps could hypothetically be hacked to flood a patient’s body with excessive insulin. These risks necessitate robust data protection laws, such as the EU’s GDPR, and continuous research into secure, privacy-preserving AI.
The Next Steps for Ethical Frameworks and Global Governance
To make medical AI safe and beneficial, we need clearer ethical rules and stronger global cooperation. Developers must work closely with doctors, patients, and ethicists, use standard checklists to test fairness and safety, and track how AI systems perform over time. Medical training must also evolve to ensure healthcare professionals understand how to use AI responsibly. Most importantly, countries must work together to prevent weak regulations and protect human rights, especially in the context of telemedicine. Only then can medical AI truly serve patients and society.
References
- Cohen, I. Glenn, and Andrew Slottje. “Artificial Intelligence and the Law of Informed Consent.” Research Handbook on Health, AI and the Law, edited by Barry Solaiman and I. Glenn Cohen, Edward Elgar Publishing Ltd, 16 July 2024, Chapter 10.
- Lekadir, Karim, et al. Artificial Intelligence in Healthcare: Applications, Risks, and Ethical and Societal Impacts. European Parliamentary Research Service (EPRS), Scientific Foresight Unit (STOA), June 2022.
- Cestonaro, Clara, et al. “Defining Medical Liability When Artificial Intelligence Is Applied to Diagnostic Algorithms: A Systematic Review.” Frontiers in Medicine, vol. 10, 27 Nov. 2023, doi:10.3389/fmed.2023.1305756.
- World Health Organisation. Ethics and Governance of Artificial Intelligence for Health: WHO Guidance. World Health Organisation, 2021.
Roller, Roland, et al. “One Size Fits None: Rethinking Fairness in Medical AI.” arXiv, 2024.
