DIverse DatabasE ReplicatiOn - Performance Comparison (DIDERO-PC)

This proposal is about experimental evaluation of diverse database replication protocols. It aims at comparing the performance of several database replication protocols, which protect against non-crash failures and are based on use of diverse, i.e. different by design, database management systems (DBMSs). These DBMSs are developed by different vendors, and can be either commercial (e.g. Microsoft SQL Server, Oracle), or open-source products (e.g. MySQL, PostgreSQL). The experimental evaluation will be conducted using two approaches: a model-based approach and a direct experimentation on a testbed with "real" systems.
Diverse database replication protocols have a potential of providing higher dependability assurance than traditional approaches that are based on the use of the DBMSs from the same vendor (research from the Centre for Software Reliability (CSR) at City University London has contributed significantly to that finding). However, besides achieving appropriate dependability level, system designers and other DBMS stakeholders are interested in the performance characteristics offered by alternative solutions. For many applications performance is, in fact, the dominant criterion in the selection process.
A fair and thorough performance comparison between database replication protocols, including the ones studied in this proposal, is of great benefit to system designers when making decisions about which one is most suitable to integrate in their system. This is the main motivation of the proposal.
The project aims at comparing the performance of three database replication protocols which guard against non-crash failures and are based on use of diverse DBMSs: our DivRep (the reference point), SES and Byzantium. There are two main objectives of the particular performance evaluation to be achieved:
- Objective 1 - evaluation based on using "real" systems on an experimental testbed, i.e. using respective implementations of the replication protocols together with the real DBMSs.
- Objective 2 - model-based evaluation, whereby each replication protocol, as well as DBMS resources, are represented with a suitable model (implemented in a particular tool of choice) and the protocol performance measured by simulation of the respective model.
Performance comparison of DBMSs and related replication protocols is too often based only on the comparison that uses (synthetic) benchmarks, which in most cases do not reflect the operational profile of the system under test. In addition, there are limits to what one can achieve with performance comparison based only on experimental evaluation with "real" systems: one can feasibly explore only a limited range of parameter values (such as number of replicas, ratio between read and write operations, etc). In contrast, model-based approach offers greater flexibility by providing means for evaluation with a wide range of configuration parameters in a practical manner. Therefore, the proposal aims at satisfying the following objective:
- Objective 3 - provision of a general-purpose methodology for model-based performance evaluation of database replication protocols.

A database replication protocol based on the use of diverse Database Management Systems (DBMSs) has been developed by the researchers at the Centre for Software Reliability at City University London. This protocol is referred to as DivRep. The findings of the project could confirm our intuition that DivRep is characterized with good performance. That result will be a significant justification towards building a fully functional diverse database replication solution (referred colloquially in CSR as DivSQL), which requires several other components apart from the replication protocol DivRep. Instigating the development of DivSQL and turning it into a hi-tech software product could, in turn, have far-reaching effects: it will open new employment opportunities for highly skilled personnel (particularly in software development field) and, more generally, will improve the competitiveness of the UK economy. As a result, the society as a whole will benefit financially from the success of the project and so will the institution (City University London).
CSR has invested considerably, mainly through funding from the City University London, in setting an Innovation vision and corresponding strategy. The main pilot project considered as part of the Innovation initiative at CSR is commercialisation of DivSQL. CSR team has been negotiating different commercialisation alternatives with the University's Technology Transfer team; the most viable of which is a set-up of a spin-off company. Our DivRep replication protocol is at the heart of DivSQL, and the project results would feed back directly to the plans for DivSQL development; e.g., they would help us assess DivSQL performance and identify necessary changes to be made. Nevertheless, the results obtained in the project are not meant to be self-serving: any company interested in developing solutions for dependable database replication will benefit from the measurement results and model-based methodology for database replication assessment provided by this project.
The increasing reliance on IT systems by various businesses, of different sizes and from virtually all industrial sectors, is exemplified by the pervasive use of database software. Despite contributing to improved business productivity, these complex software packages do fail, occasionally with serious consequences, e.g. a corruption in Oracle database has left JPMorgan Chase online and mobile banking non-operational for more than 24 hours (http://www.banktech.com/articles/227700032). A part of the failures will manifest themselves as crashes and will be acted upon by system operators, but many will not be self-evident and will remain unnoticed, possibly for long periods of time, causing unknown damage to the system. It is the latter, more subtle, type of failures that diverse database protocols, including our DivRep, protect against and as such aid business continuity. Early adoption of this disruptive technology can be a valuable asset of UK businesses in a long run. If the adoption materializes, the results from this project will help system designers in making informed decision if, and when, to deploy diversity: the detailed analysis of the cost of diverse redundancy for database replication protocols (in terms of performance overhead) will be a valuable contribution in that respect.