10086253Optimising Industry Dynamics: A Performance-Persistent AI System Adaptable to Rapid Data and Model ChangesClosedCollaborative R&DInnovate UKIn today's fast-evolving AI landscape, models ranging from Convolutional Networks (ConvNets) to Large Language Models (LLMs) increasingly rely on high-quality relevant data. The significance of efficient data management in AI systems, particularly in maintaining system performance and ensuring versatile sources of challenging unstructured data can be efficiently processed into high quality knowledge bases, is undeniably crucial. In response to this challenge, we propose a systematic approach that addresses the critical missing link and feedback loop for AI systems and unstructured data sources. Our unified, adaptive AI system will feature innovative LLM-based evaluation and routing mechanisms to streamline data, model and pipeline utilisation. Our approach incorporates four core mechanisms, each designed to make a significant contribution to the AI and Data industry: 1. Adaptive Routing Mechanism: This will decompose and evaluate system inputs to automatically route them to the most appropriate pipelines and models, ensuring real-time adaptability and predictions that are attuned to evolving real-world conditions. 2. Multimodal Processing Pipelines: We will implement customisable unstructured data ETL pipelines and models to relate text, images, audio, video, and web data to ensure high-performance predictions regardless of the data modality or file complexity. 3. Tracking Performance Degradation & Optimisation: We will use 'golden' metrics to measure pipeline performance, detect any decline, and invoke necessary optimisation. 4. On-the-fly Calibration with Memory and Reasoning Units: We will ensure accurate and precise predictions on complex tasks by incorporating a memory system and Large Reasoning Models (LRMs) to facilitate well-aligned multi-step problem solving. Instill AI, with its expertise in developing versatile ETL data pipelines, will focus on developing multimodal processing capabilities, a sophisticated AI agent system and pipeline optimization features. Concurrently, InfuseAI will enhance its advanced data-centric code review tools to handle unstructured data evaluations to enable robust performance tracking, contributing to the system's adaptability. By working together, we are not only pooling resources but also addressing industry challenges head-on. This collaboration will enable us to navigate the dynamic AI market confidently and lead the evolution of MLOps systems.