EA-MDE: An Empirical Assessment of the Efficacy of Model Driven Engineering

The complexity and pervasiveness of software in society is growing exponentially. It is generally agreed that theonly realistic way to manage this complexity, and to continue to provide software benefits to the public at large, is todevelop software using appropriate methods of abstraction. Today, the state-of-the-art in software abstraction ismodel-driven engineering (MDE) - that is, the systematic use of models as primary artefacts during a softwareengineering process. MDE includes various model-driven approaches to software development, including model-drivenarchitecture, domain-specific modelling, and model-integrated computing.Although MDE claims many potential benefits - chiefly, gains in productivity, portability, maintainability andinteroperability - it has been developed largely without the support of empirical data to test these claims. As a result,decisions whether or not to use MDE are based mainly on expert opinion rather than hard empirical data; and theseopinions often diverge. The lack of empirical results on MDE is a problem for two reasons.Firstly, industry invests millions in the development and application of MDE tools. Without empiricalevidence of the efficacy of these tools, there is a danger that resources are being wasted. Whether or not the currentbrand of MDE tools succeeds, the notion of abstract models is crucial to the future of software. Empirical evaluationsare needed to ensure that future software tools will match the way that software developers think.Secondly, academia also invests significantly in MDE in the form of PhD theses and research papers. This research israrely informed by empirical evidence, which means that it is difficult for funding bodies to properly assess theusefulness of research results. One issue is that early-career researchers lack the multidisciplinary knowledge that isinherently required - that is, knowledge both of MDE and the psychological know-how of conducting experiments.Another problem is simply that empirical evaluation in MDE is hard. Rigorous evaluations ought to engage industryover lengthy periods of time, but industry is often reluctant to get involved because it cannot see immediate benefits of anovel research technique.The proposed research will address these two problems by developing a framework for empirical evaluation of MDEthat is informed by current industry practice and needs and is available to researchers to use and adapt it as necessary.The overall goal of the research is to address fundamental questions as to how empirical evaluations can best beintegrated with MDE research and practice as well as to provide steps toward a scientific foundation for MDEevaluation and adoption.This is a 12 month pilot project. This is for the following reason. The challenges of empiricallyevaluating MDE are fundamentally hard. Since there has been very little research in this area, a feasibility study wouldbe beneficial before investing further resources. We will use this pilot phase to map out the MDE evaluation landscapeand to develop an initial evaluation framework. We view the pilot as phase I of a two phase effort to provide a scientificfoundation for MDE evaluation. We plan to use the results of the pilot to support a follow-on phase II effort that willpropose: a theory of software modellers' cognitive processes; a more detailed evaluation framework; and a newgeneration of MDE tools.