📣 Try out the NEW Gateway to Research and let us know what you think.

We're looking for users to test the new service during August and September and share their feedback. Express your interest by completing this short form.

MUSE: Multi-Modal Software Evolution

Lead Research Organisation: University of Surrey
Department Name: Computing Science

Abstract

Software systems are heterogeneous, combining components developed by independent teams. Software developers rely on third-party libraries to cut development time and cost. The synergy between these components is crucial for the overall maintainability and health of the software system. Unfortunately, popular libraries are typically fast-moving and grow rapidly in size while catering to a diversity of client software. As libraries evolve and grow in size, developers tend to defer upgrades despite clear upgrade directives from the libraries, citing the cost of upgrade in both time and money.

To build large and sustainable software systems, it is crucial that independently evolving software systems are synchronised automatically. Multi-Modal Software Evolution (MUSE) is a transformative step towards autonomous software maintenance where directives in software documentation for human developers will guide automated software upgrade. In MUSE, we will develop a novel approach to software upgrade that integrates upgrade directives for human developers into formal frameworks for program synthesis, generation and repair. We will include directives in documentation for libraries as first class objects in frameworks for reasoning and transformation of software. We will produce hybrid statistical-formal reasoning frameworks which will make human-to-human communication the main driver in automatic program transformation.

Working closely with stakeholders through engagement events, we will develop both the theory and the tooling for automatic software upgrade to use newer versions of libraries. We will demonstrate the tools by upgrading client software that relies on fast-moving libraries and distribute the tools that we develop in multiple forms for developers at all skills levels, from enthusiasts to experienced developers, making our outputs widely accessible.

Related Projects

Project Reference Relationship Related To Start End Award Value
EP/W015927/1 30/09/2022 25/03/2024 £421,797
EP/W015927/2 Transfer EP/W015927/1 26/03/2024 31/10/2026 £311,024
 
Description We found that interfaces for software libraries are often partially documented. This has an impact on how developers use the library, which impacts quality of software that we use. Consequently, there is a need for tools that can identify missing information and work with partial documentation. We have also developed a tool that can auto-process software documentation to guide transformation of software. This tool can help software developers adopt newer versions of software libraries, making software systems resilient in the long run.
Exploitation Route Our research can be used for automatic software maintenance, allowing developers to respond to changes in the software libraries they use. We expect that this finding can also inform tools for identifying and auto-generating documentation in future. Such tools can be used by library developers to update their documentation.
Sectors Digital/Communication/Information Technologies (including Software)
 
Title Deterministic Oracle for Detecting Hallucinations in LLM Generated Code 
Description LLMs enable automation in software engineering tasks such as API migration. We identified a critical failure mode in LLM-driven API migration, Scaffolding Hallucination. While LLMs successfully capture the semantic intent of an API migration, they frequently fail to generate the correct syntactic scaffolding - the specific builders, constants, and helper methods required for valid instantiation. We demonstrated that standard metrics like CodeBLEU and probabilistic LLM judges are unreliable for this task, as they often validate structurally plausible but functionally broken code. A Deterministic Fact-Checking Framework To bridge this gap, we developed a lightweight, deterministic evaluation framework. Unlike probabilistic approaches, our system grounds verification in an irrefutable API Oracle, constructed directly from official library specifications (e.g., the Android SDK). This moves the evaluation standard beyond simple plausibility to strict specification compliance. Novel Taxonomy & Algorithms Our approach is driven by a new detection-oriented taxonomy that categorises failures into two verifiable tiers: 1. Atomic Hallucinations: The invention of non-existent symbols (e.g., phantom constants). 2. Context-Dependent Hallucinations: The invocation of valid symbols on an invalid receiver type. To detect Context-dependant errors, we propose a novel Abstract Type Propagation algorithm. This static analysis technique resolves variable types and method chains within partial code snippets, enabling the detection of invalid call chains without requiring a full build environment. Validation We validated this framework on a curated dataset of Android API migrations. Our results confirm that the system successfully flags hidden hallucinations that bypass standard metrics, offering a reliable, automated guardrail for enterprise-grade API migration pipelines. 
Type Of Material Improvements to research infrastructure 
Year Produced 2026 
Provided To Others? No  
Impact Challenging Evaluation Standards: The development of this framework has exposed critical flaws in current evaluation methodologies. We demonstrated that standard metrics (e.g., CodeBLEU) and "LLM-as-a-Judge" systems are unreliable: they suffer from optimism bias when evaluating broken code and over-correction bias when analysing valid implementations. Our work establishes that deterministic verification is a necessary baseline for API migration tasks, forcing a re-evaluation of how the field measures success. Validating Lightweight Industrial Architectures: Our research validates the shift towards lightweight, syntax-aware tools over complex agentic loops. This architecture, using AST parsing and deterministic Oracles, aligns with recent industrial findings (e.g., Bytedance), which demonstrate that streamlined, rule-based verification workflows significantly outperform heavy, multi-turn agent systems in terms of scalability and latency for large-scale code repair. Enabling Reinforcement Learning (RL) Loops: Beyond evaluation, this tool provides a high-fidelity signal that can serve as a reward function for model training. As evidenced by deployment in large-scale industrial settings, using static analysis feedback (like linting or our API Oracle) as a reward signal for RL effectively suppresses hallucinations and redundant edits, offering a clear path to training more precise repair models. 
 
Title Grammar-based taxonomy for API Migration 
Description Context & Problem API evolution and deprecation management are critical software engineering challenges, yet current automated approaches often lack a granular assessment of migration complexity. Existing methods typically treat all API updates, whether simple method renames or complex architectural changes, as equivalent tasks. This lack of differentiation obscures the underlying difficulty of automated code repair, leading to inefficient resource allocation and high failure rates in automated migration tools. Approach To address this, we developed a novel taxonomy and automated analysis framework that categorises API migrations based on their structural requirements. The method utilises a formal grammar to classify deprecations into distinct taxa, ranging from atomic replacements to complex structural adaptations involving fresh variables or logic wrappers. Building on this taxonomy, we formulated a quantifiable complexity metric. The metric's weights were calibrated based on a dataset assigning higher costs to operations that disrupt local semantic flow. We implemented a documentation analysis tool that classify API deprecations into our taxonomy. Evaluation Methodology The method was validated through a two-stage empirical study. First, we conducted a manual audit of the classification pipeline on a sample of real-world Android API migrations. Second, we assessed the correlation between our complexity metric and the success rates of Large Language Model (LLM) based repairs. The evaluation confirmed that the metric serves as a reliable proxy for migration hardness , enabling the development of automated abstention policies that significantly improve repair reliability by filtering out infeasible tasks. 
Type Of Material Improvements to research infrastructure 
Year Produced 2025 
Provided To Others? No  
Impact A New Standard for Measuring Complexity. This work established a formal framework for quantifying the difficulty of software updates. Previously, most approaches treated all API changes as roughly equal in difficulty. Our research proved that structural changes impose a much higher cost than simple replacements. This theoretical contribution provides the wider software engineering community with a better way to predict and manage the difficulty of maintaining evolving software systems. Strategic Planning for Software Evolution. This framework enables better planning for both the creators of software libraries and the developers who use them. Library owners can now evaluate the complexity of their proposed changes before releasing them, allowing them to choose designs that minimise the effort required for upgrades. Furthermore, including this complexity information in release notes allows developers to accurately estimate the time and resources needed to update their systems, transforming migration from a reactive fix into a proactive strategy. Improved Reliability for Automated Repair. The practical impact of this research is a significant increase in the reliability of automated code repair tools. By developing a method to identify and filter out tasks that are too complex for current models to handle, the system's overall success rate improved substantially. This helps prevent the generation of incorrect code, reducing the workload for developers who would otherwise need to review and reject these failures. Computational Efficiency and Cost Savings. The method demonstrated that a large portion of computational resources is often wasted on tasks that are destined to fail. By implementing our complexity-based filtering, we achieved a major reduction in processing costs. This efficiency allows automated systems to operate more economically, ensuring that expensive computing resources are focused only on tasks where a successful solution is feasible. 
 
Title API migration dataset 
Description Dataset of 102 pairs of code snippets with deprecated usage and ground-truth migration. 
Type Of Material Database/Collection of data 
Year Produced 2025 
Provided To Others? No  
Impact The dataset serves as benchamark for automated migration tools. The ground-truth provides an example of how deprecated usage should be migrated. Researchers can use this dataset to compare automatic repair systems. The dataset will be released with our upcoming publication. 
 
Title Annotated API Migration Benchmark 
Description A dataset of automated API migrations containing code snippets generated by large language models, annotated with verified ground truth, manual evaluations, and identified hallucinations. We constructed a specialised dataset of automated API migrations, focusing on the complex syntactic scaffolding required to bridge legacy and modern Android APIs. This collection comprises code snippets generated by Large Language Models, paired with verified ground truth implementations and detailed manual evaluations. The dataset is annotated with specific failure modes, including hallucinations. This resource serves as a high-fidelity benchmark for evaluating the precision of automated repair tools and hallucination detection frameworks, addressing the industry need for granular failure analysis in generative code. 
Type Of Material Database/Collection of data 
Year Produced 2026 
Provided To Others? No  
Impact Benchmarking Precision: This dataset addresses a critical gap in current API migration benchmarks, which lack detailed annotations. This dataset can serve as a benchmark for detecting hallucinations in API usage, and it will be released to the public with our upcoming publication. Fuelling Reinforcement Learning: High-quality, verified data is essential for advanced model training. As demonstrated in recent industrial applications, using verified repair examples as feedback for Reinforcement Learning significantly reduces redundancy and hallucination. This dataset provides the necessary ground truth to train next-generation models that are natively resistant to scaffolding errors. Calibrating Automated Judges: Recent studies reveal that LLM-based judges frequently suffer from optimism and over-correction bias, misclassifying valid code as defective due to a lack of precise context. This annotated dataset serves as a crucial calibration resource, allowing researchers to fine-tune automated evaluators to distinguish between genuine API misuse and valid, complex scaffolding. 
 
Description Collaboration with Institute of Science, Tokyo 
Organisation Science Tokyo
Country Japan 
Sector Academic/University 
PI Contribution This is a continuing partnership. Our collaborator moved from National Institute of Informatics to Institute of Science.
Collaborator Contribution This is a continuing partnership. Our collaborator moved from National Institute of Informatics to Institute of Science.
Impact This is a continuing partnership. Our collaborator moved from National Institute of Informatics to Institute of Science.
Start Year 2025
 
Description Collaboration with National Institute of Informatics, Japan 
Organisation National Institute of Informatics (NII)
Country Japan 
Sector Public 
PI Contribution In this project, we extracted Android documentation to identify APIs that have been deprecated or in simple terms, marked for removal. A documentation message in these deprecations serve as a hint to developers that use Android's API on how they can use newer version of the API.
Collaborator Contribution Our collaborators helped us categorise these deprecation messages so that the category information can be used by an automated tool to update software that uses these APIs, replacing uses of the deprecated API with its replacement.
Impact We are currently writing a joint paper on the usefulness of these categorisations, describing how it can improve outcomes for automated code rewriting to use newer versions of the API.
Start Year 2024
 
Title Fact-checking API hallucinations 
Description The software is a Python application that takes code snippets as input and consults an oracle created from API documentation to detect hallucinated API usage in LLM-generated code. We developed a lightweight, static analysis tool designed to verify the correctness of LLM-generated API usage. The software parses partial code snippets into Abstract Syntax Trees and cross-references extracted symbols against a verified API Oracle constructed from official SDK specifications. By employing a novel Abstract Type Propagation algorithm, the tool resolves method chains and variable types to detect Scaffolding Hallucinations, specifically distinguishing between non-existent symbols and valid symbols used in invalid contexts, without requiring a compile-ready environment. 
Type Of Technology Software 
Year Produced 2026 
Open Source License? Yes  
Impact Exposing Evaluation Blind Spots: The deployment of this tool demonstrated that standard metrics (e.g., CodeBLEU) and LLM judge systems are prone to miss scaffolding hallucinations in API migration tasks. Our software reveals that these methods suffer from optimism bias (approving broken code) and over-correction bias (rejecting valid code), establishing that deterministic, syntax-aware verification is a necessary baseline for reliable automated repair. Validating Lightweight Verification: Our tool validates the efficacy of lightweight, AST-based static analysis over complex, multi-turn agentic workflows. Enabling Feedback Loops for Training: Beyond evaluation, the software provides the machinery to collect data for model training. The software will be released as part of the artefact of our upcoming publication. 
 
Title Tool for Automatic Taxon Classification from API Documentation 
Description We developed an automated analysis framework designed to characterise the complexity of API deprecations in the Android ecosystem. The software is a Python application that parses official documentation to classify deprecated methods into a refined taxonomy of edit operations, distinguishing between simple replacements and complex structural changes. Based on this classification, the tool calculates a numeric complexity score using calibrated weights. This score serves as a predictive metric for migration difficulty, enabling automated repair systems to distinguish between feasible updates and those requiring significant architectural intervention. 
Type Of Technology Software 
Year Produced 2026 
Open Source License? Yes  
Impact Strategic Planning for Software Evolution: This software enables better planning for both software library creators and developers who use them. Library owners can now evaluate the complexity of their proposed changes before releasing them, allowing them to choose designs that minimise the effort required for upgrades. Furthermore, including this complexity information in release notes allows developers to accurately estimate the time and resources needed to update their systems, transforming migration from a reactive fix into a proactive strategy. - Resource Optimisation: The tool enabled significant computational efficiency by identifying "infeasible" migrations before any repair attempt was made. Repair system can abstaint or assign more resources to complex migration, e.g. more powerful models. - The tool can be used to characterise different ecosystems. - The tool will be released later this year as part of the artefact of our upcoming publication. 
 
Description Research Institute on Verified Trustworthy Software Systems 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact This was a specialist meeting to discuss maturity of cyber security tools for software development. The event had about 20 attendees. We presented our work and it led to discussions for collaborative work with some of participants.
Year(s) Of Engagement Activity 2025