Bias in Clinical AI: Urgent Calls for Robust Governance, Transparency, and Patient Safeguards Amid Mounting Evidence of Disparities

Today's LOG Standards Daily Briefing focuses on the escalating concerns surrounding bias, safety, and governance in clinical artificial intelligence, with a clear consensus emerging on the urgent need for robust regulatory and ethical frameworks. Multiple recent studies provide compelling evidence that AI-powered clinical decision support tools, particularly large language models, are not only mirroring but actively amplifying existing healthcare disparities. Research from the Icahn School of Medicine at Mount Sinai, published in Nature Medicine, found "extremely universal" bias in clinical recommendations, where AI models provided systematically different advice based solely on patient race, income, housing status, and LGBTQIA+ identity, despite identical clinical information. For instance, Black, unhoused, and LGBTQIA+ patients were more frequently recommended urgent care or mental health evaluations, while higher-income patients were disproportionately advised advanced imaging like CT and MRI. This multi-model evaluation, echoed by Reuters, demonstrates how AI can entrench existing biases, threatening patient safety and equity by influencing critical diagnostic and treatment pathways. Compounding these findings, a JAMA study highlighted by EurekAlert revealed that clinicians' diagnostic accuracy dropped by over 11 percentage points when relying on biased AI models. Crucially, post-hoc explanations provided by the AI, even when pointing to irrelevant features, did not restore performance, indicating that explainability alone is insufficient to safeguard patient safety. A viewpoint in the Journal of Medical Internet Research, summarized by Telehealth.org, further warns that while generative AI could enhance patient-centered care, significant work is needed to address safety, bias, and transparency risks, advocating for independent testing, algorithmic drift monitoring, and clear patient consent policies. In response to these pervasive issues, there is a growing call for enhanced governance and regulation. Expert recommendations in the Journal of the American Medical Informatics Association propose a comprehensive governance framework for AI-enabled clinical decision support, emphasizing transparency, verification, certification, ongoing monitoring, and adverse event reporting. Key concepts include Unique AI Identifier labels and formal infrastructure for oversight to mitigate biased and unsafe recommendations. The World Health Organization has also released new guidance on the ethics and governance of large multimodal medical AI, offering 40 recommendations for responsible deployment, risk management, and oversight across various clinical and administrative uses. Regulatory bodies and states are beginning to act. The FDA, through its evolving guidance, continues to refine its risk-based framework for clinical AI, emphasizing labeling, post-market controls, and managing software updates, while also clarifying exemptions for some AI-enabled clinical decision support tools. Concurrently, states like California are implementing stricter AI governance, requiring human final review for AI-assisted utilization decisions and disclosure for generative AI patient communications. These state-level initiatives, as noted in healthcare compliance and governance briefings, aim to prevent AI from usurping licensed clinician judgment and to ensure greater transparency for patients. From the LOG Standards perspective, these developments underscore the critical importance of robust accreditation, rigorous independent validation, and continuous post-market surveillance for all clinical AI systems. The evidence of systemic bias, even in widely used models, necessitates that fairness, accountability, and transparency are not merely aspirational goals but foundational requirements for deployment. Healthcare stakeholders, including developers, providers, and regulators, must collaborate to establish enforceable standards, disclose training data sources, and ensure that AI integration genuinely enhances patient care without compromising equity or safety. The emerging hybrid model of patient care, where individuals are increasingly using AI chatbots for mental health support, further complicates the landscape, raising questions about clinician preparedness, liability, and the safe integration of consumer AI tools into standard practice, particularly given studies linking AI advice to increased depression and anxiety symptoms for some users.