The clinical artificial intelligence (AI) landscape is currently defined by a dual focus: the undeniable potential for innovation and the critical imperative to address inherent risks, particularly concerning bias and patient safety. Recent analyses consistently highlight how biases can accumulate throughout the AI lifecycle, from data collection and annotation to model development and deployment, directly impacting clinical decisions and patient outcomes. Real-world scenarios demonstrate that biased models can deliver substandard care to specific populations, widening existing healthcare disparities. Strategies to combat this include the use of more globally representative datasets, the involvement of multidisciplinary development teams, and the implementation of dashboards that reveal model limitations and potential inequities to clinicians. Regulatory bodies are actively responding to these challenges. The U.S. Food and Drug Administration (FDA) continues to refine its oversight, with a focus on a Total Product Lifecycle model for AI/ML-based Software as a Medical Device (SaMD). This includes new guidance on Good Machine Learning Practice principles, generative AI considerations, and updated expectations for premarket review and post-market monitoring. The FDA has also issued a revised final guidance on Clinical Decision Support (CDS), clarifying which AI-driven tools qualify as medical devices and extending enforcement discretion to certain single-output recommendation tools. These frameworks aim to ensure that AI tools meet statutory safety and effectiveness standards, though clearance does not equate to an endorsement of clinical benefit. Despite these efforts, significant gaps in regulatory oversight persist. Policy experts and reports, including one to the FDA, critique current approaches, citing deficiencies in safety evaluation, post-market surveillance, and ethical safeguards. A key concern is that many clinical decision-support tools currently avoid stringent FDA oversight, effectively turning the marketplace into a large-scale testing ground for medical AI. This decentralized oversight, coupled with a lack of mandated transparency or equity auditing, allows potentially biased tools to enter practice, compromising patient safety and exacerbating healthcare disparities. Calls for adaptive regulation, mandatory post-market performance monitoring, disclosure of training data sources, and enforceable standards for fairness and accountability are growing louder. International bodies are also stepping in to provide guidance. The World Health Organization (WHO) has published new guidance on ethics and governance for large multimodal models (LMMs) in healthcare, offering 40 recommendations to ensure safe, equitable, and accountable deployment of powerful AI systems. Similarly, major radiology societies have issued joint guidance on the safe clinical use of AI, emphasizing safety monitoring, multidisciplinary collaboration, and ethical integration into clinical workflows, particularly as AI adoption accelerates in imaging. These initiatives underscore a global commitment to establishing robust governance structures. While regulatory and ethical challenges are being addressed, the clinical utility of AI continues to advance. Researchers at Children's National Hospital have developed an AI-powered tool for early detection of rheumatic heart disease, potentially preventing severe outcomes. Furthermore, a Harvard-led study reported that an advanced large language model achieved higher diagnostic accuracy than emergency physicians in hospital triage, demonstrating AI's potential to improve diagnostic precision and even aid in complex treatment planning. However, the rapid proliferation of AI chatbots for mental health support presents a new frontier of concern. While some studies show promise for symptom reduction in controlled conditions, experts warn that the limited evidence, unclear regulation, and potential for companies to prioritize engagement over user safety pose significant risks. The American Psychological Association distinguishes between approved mental health tools and unregulated consumer chatbots, noting that reliance on AI during crises can worsen delusions, suicidality, and emotional dependence. This highlights the critical need for rigorous, prospective evaluation and clear regulatory pathways to ensure that AI-driven solutions genuinely enhance patient safety and well-being without introducing new harms.
