Toward an Industrial Strength Automatic Differentiation Product

Lead Research Organisation: University of Hertfordshire
Department Name: Science and Technology RI


Numerical simulation of complex real-world phenomena is a major challenge for almost all activities in science and engineering, particularly when it is desired not merely to model a process (such as airflow across the surface of a car or a wind turbine) but to optimise the process (for example, to adjust the shape of the surface so as to minimize the drag).

To do this efficiently for large problems, it is essential for the model to have access to numerical derivatives (sensitivities of the outputs with respect to inputs) that are accurate (free from coding, rounding and trunction errors) and computationally cheap. At present most users either provide hand-coded derivative routines, which are tedious to program and update, and prone to coding errors; or finite differences, which are expensive to compute and numerically inaccurate.

Automatic Differentiation (AD) is a set of techniques for transforming numerical modelling code mechanically so that it calculates the numerical sensitivities of the model values as well as the values themselves, at the same order of speed as hand-coded derivatives, and to the same precision. This is done by using the chain rule from calculus, but applied directly to floating point numerical values, rather than to symbolic expressions.

The so-called reverse, or adjoint, mode of AD can produce a complete set of sensitivities for a computer program evaluating a model, at a computational cost of order five times that of a single evaluation of the program itself - even if there are millions of input variables and sensitivities. However, achieving this requires the ability to make the program run backwards, recovering all the intermediate numerical values in the process, and it is this which requires the development and use of sophisticated software tools.

AD techniques are still not widely used in modelling and optimisation, due in large part to a lack of suitable user-friendly general-purpose commercial-strength AD tools.

Current AD tools work either by operator overloading or by source pre-processing. Operator overloading is reliable, but slow, and do not provide support for calls to specialist numerical library functions, such as linear algebra routines. Source pre-processing tools require a high user awareness of AD, produce code that is not easy for humans to understand, and have memory management issues.

The research compiler developed by the CompAD team with EPSRC support is an enhancement of the NAG Fortran95 compiler, and is still the world's only industrial strength compiler to have built-in support for AD. Embedding an overloading approach to AD within an existing high performance compiler gives the best of both worlds: the convenience of the overloading approach and the speed of source pre-processing. However the current compiler is a research tool, and requires expert assistance to configure and integrate with application code. Consequently the research version of the CompAD compiler is unsuitable in its present form for the majority of potential beneficiaries.

This follow-on proposal will extract the AD functionality from the CompAD compiler, integrate it with the NAGWare Fortran Library, and make the resulting prototype widely available. As well as providing an immediate benefit to many users, this prototype will be suitable for systematic market testing and development. The prototype will be used to capture user requirements for, and to underpin subsequent development of, a commercially viable software AD product that will work "out-of-the-box" on problems of moderate size.

Planned Impact

The economic impact of this project ultimately arises from the fact that increasing the accuracy and speed of modelling and optimisation leads to the more efficient use of material resources. Examples of relevant client sectors likely to benefit include engine design, industrial process optimisation, aerospace applications, energy management, multidisciplinary design, forecasting, medical imaging, pharmaceutical industry, and many more.

Numerical simulation and optimisation of complex real-world phenomena requires the model to have access to numerical derivatives (sensitivities). Automatic Differentiation (AD) has the potential to revolutionise the provision of such sensitivities. Previous EPSRC funded research projects have produced a compiler-based tool with the required capability. The primary objective of this project is to take this research tool and modify it into a prototype of an industrial-strength, easy-to-use product.

The first impact to be felt is that resulting from an increased uptake of Automatic Differentiation (AD) by NAG's existing clients. This will provide an immediate competitive benefit to those clients with modelling and optimisation problems of moderately large size. [NAG currently has 3,000 clients, with Fortran products accounting for about stg 2M annual turnover.] The ability to offer fully integrated AD capability will also make the NAG Fortran compiler and libraries more attractive to a wider user base.

The availability of a commercial product will also increase the take-up of AD by those with very large problems at the edge of feasibility, by enabling them to have a positive initial experience of AD acquired at low entry cost prior to investing in bespoke technology.

The next impact is the knowledge-transfer effect of this commercialisation project upon NAG. Fortran is not the only language with a requirement to support AD. For example, many Tier 1 financial institutions rely heavily on computationally intensive modelling in C++ that uses high-order derivatives (Greeks). Sixty percent of the Tier 1 banks already hold global licences for NAG software. The experience gained from this project by the NAG compiler team would enable a similar level of AD support to be deployed for C++ within a five-year impact cycle. Tier 1s have repeatedly indicated that they are prepared to invest heavily in such technology. Better and more reliable financial modelling gives banks a competitive edge in the short term, but the effect of improved financial modelling on the management of the economy will ultimately benefit everybody.

There is also a knowledge transfer effect on the CompAD team. This will be their first commercialisation of an AD compiler, but they do not intend it to be their last. Their experience will also be disseminated through various technical fora, as well as consultancy, and (given their reputation within the AD community) is likely to exercise a significant influence upon other compiler providers. This will in turn impact on the market for AD tools.

A further phase of impact will be the effect on the various modelling communities themselves of the widespread availability of commercial strength AD. Preliminary indications are that further development of AD has the potential to supplement, and in some cases replace, computationally expensive modelling techniques such as Monte Carlo methods. [For example, instrumented use of the experimental CompAD compiler with a model adapted from the German Waterways Authority took less than twenty minutes to calculate a silt density distribution that required over fifteen hours using Monte Carlo.]

The ultimate impact of this project is potentially vast, particularly if it succeeds in trail-blazing a phase-change in the way in which numerical models are currently tuned in engineering and science. But even the incremental impact of the first stage on existing clients of NAG software will generate a high-multiple return on investment.


10 25 50

publication icon
Gremse F (2016) GPU-Accelerated Adjoint Algorithmic Differentiation. in Computer physics communications

publication icon
Merkel U.H. (2016) River flow analysis with adjoints - An efficient, universal methodology to quantify spatial interactions and sensitivities in Sustainable Hydraulics in the Era of Global Change - Proceedings of the 4th European Congress of the International Association of Hydroenvironment engineering and Research, IAHR 2016

Description This Follow-On project extracted the runtime library from the CompAD research compiler and integrated it with the NAG Fortran Library to provide a documented prototype Automatic Differentiation tool suitable for immediate uptake by naive users.
Exploitation Route 1. directly, by current users of the NAG Fortran Library
2. by the implementation of corresponding functionality in other languages, as has already happened with dco/c++
3. by applying the techniques developed in this project to other tools and environments, as with OpenFOAM and TAF
4. by enabling the extension of adjoint methods further into computational science, engineering and finance (see Dagstuhl Seminar 14371).
Sectors Aerospace, Defence and Marine,Agriculture, Food and Drink,Energy,Financial Services, and Management Consultancy,Manufacturing, including Industrial Biotechology,Transport

Description Following successful completion of the second phase of the CompAD project in September 2008, the research compiler was re-purposed for various end-users, as consultancy work carried out by members of the CompAD research project team (including then-UH research fellow Jan Riehme, currently senior research scientist at Software Tools for Computational Engineering, RWTH-Aachen). Meanwhile NAG Ltd has made the CompAD compiler available on a commercial basis to its customers, including engineering and financial sector organisations. In response to customer requests, NAG took the commercial decision to develop AD versions of a number of mathematical library routines, using an enhanced version of the CompAD compiler. In particular, NAG is already providing AD software and consultancy services based on the CompAD research to two major Tier 1 banks, who are using this software in real time to control risks on the futures market (option-swap decisions for high value amounts). The C++ tools involved were developed in partnership with RWTH-Aachen, building on the research pioneered by the CompAD project. They provide sensitivities ("Greeks") more accurately and orders of magnitude faster than previously available tools. In addition, the CompAD compiler is the basis for the tools currently used in TelemacAD (Telemac consortium), and for ICON-O (MPI-M). CompAD also inspired the significance analysis approach of EU project SCoRPiO.
First Year Of Impact 2013
Sector Aerospace, Defence and Marine,Construction,Environment,Financial Services, and Management Consultancy,Healthcare,Manufacturing, including Industrial Biotechology,Transport
Impact Types Economic

Description Numerical Algorithms Group Ltd 
Organisation Numerical Algorithms Group Ltd
Country United Kingdom 
Sector Private 
Start Year 2006
Description RWTH Aachen University 
Organisation RWTH Aachen University
Country Germany 
Sector Academic/University 
Start Year 2006