10155860CLIO: Clinical LLM Integration and OversightClosedCollaborative R&DInnovate UKThe rapid growth of large language models (LLMs) in healthcare brings transformative opportunities to improve patient care and efficiency. These advanced AI systems can assist with tasks like generating clinical notes, supporting decision-making, and interacting with patients. However, their use in healthcare creates unique safety challenges that need to be addressed. These challenges include: **Unclear governance:** LLMs are often relied upon in ways that can be both beneficial and risky. For example, while they can reduce administrative burdens for clinicians, over-reliance might lead to errors or a loss of critical thinking. On the other hand, under-reliance may mean the opportunity they present is not optimised. Currently, there's no clear guidance on how much to depend on these tools or how to monitor their safe use, leaving room for unintended consequences. **Risky outputs:** LLMs sometimes produce outputs that seem correct but are factually wrong, incomplete, or biased. In healthcare, such errors can have serious consequences, like overlooking a patient's allergy and leading to incorrect treatment being prescribed. Unfortunately, there's no reliable way to check the accuracy or appropriateness of these outputs on a large scale, nor a clear system to manage errors. **Compounding risks in complex systems:** When LLMs are integrated with other systems like electronic health records or diagnostic tools, small errors can potentially escalate into bigger problems. For instance, a misleading suggestion from an LLM could lead to improper allocation of resources or delays in urgent care. These ripple effects are hard to track and manage because healthcare providers don't yet have tools to evaluate the interactions between these systems, human users, and clinical processes. **Lack of monitoring tools:** Currently, there's no standardised way to monitor how LLMs perform once deployed in real-world settings. This includes tracking who uses them, how they are used, and whether their performance changes over time. Without such tools, it's difficult to detect and address problems like biased outcomes or failures in safety-critical areas like emergency care. To tackle these issues, we must create systems that combine monitoring, governance, and safety measures tailored specifically for healthcare. This approach aims to ensure that LLMs are used responsibly, maintaining trust in AI systems while enhancing patient outcomes and reducing risks.