Development & Clinical Translation of Scalable HPC Ultrasound Models

Lead Research Organisation: University College London
Department Name: Medical Physics and Biomedical Eng

Abstract

The use of ultrasound as a diagnostic imaging tool is well known, particularly during pregnancy where ultrasound is used to create images of the developing foetus. In recent years, a growing number of therapeutic applications of ultrasound have also been demonstrated. The goal of therapeutic ultrasound is to modify the function or structure of the tissue, rather than produce an anatomical image. This is possible because the mechanical vibrations caused by the ultrasound waves can affect tissue in different ways, for example, by causing the tissue to heat up, or by generating internal radiation forces that can agitate the cells or tissue scaffolding. These ultrasound bioeffects offer a huge potential to develop new ways to treat major diseases such as cancer, to improve the delivery of drugs while minimising side-effects, and to treat a wide spectrum of neurological and psychiatric conditions.

The fundamental challenge shared by all applications of therapeutic ultrasound is that the ultrasound energy must be delivered accurately, safely, and non-invasively to the target region within the body. This is difficult because bones and other tissue interfaces can severely distort the shape of the ultrasound beam. This has a significant impact on the safety and effectiveness of therapeutic ultrasound, and presents a major hurdle for the wider clinical acceptance of these exciting technologies. In principle, any distortions to the ultrasound beam could be accounted for using advanced computer models. However, the underlying physics is complex, and the scale of the modelling problem requires extremely large amounts of computer memory. Using existing software, a single simulation running on a supercomputer can take many days to complete, which is too long to be clinically useful.

The aim of this proposal is to develop more efficient computer models to accurately predict how ultrasound waves travel through the human body. This will involve implementing new approaches that efficiently divide the computational problem across large numbers of interconnected computer cores on a supercomputer. New approaches to reduce the huge quantity of output data will also be implemented, including calculating clinically important parameters while the simulation runs, and optimising how the data is stored to disk. We will also develop a professional user interface and package the code within the regulatory framework required for medical software. This will allow end-users, such as doctors, to easily use the code for applications in therapeutic ultrasound without needing to be an expert in computer science. In collaboration with our clinical partners, the computer models will then be applied to different applications of therapeutic ultrasound to allow the precise delivery of ultrasound energy to be predicted for the individual patient.

Planned Impact

The novel software developed as part of this proposal will generate impact in two different ways. First, there is considerable potential for the computational models to impact clinical practice by extending and expediting the use of ultrasound as a tool to treat cancer and other diseases. Second, the computational performance of the models is likely to have significant impact in the many allied areas of research and industry in which waves play a role.

Clinical Impact: Over the last decade, novel applications of therapeutic ultrasound have excited much interest within the research community. However, wider clinical acceptance has been modest due to challenges in controlling the ultrasound fields with the required accuracy. The scalable k-space models and clinical user interface proposed in this application could overcome this barrier and act as a catalyst for the widespread clinical application of these technologies. Impact will arise from (i) the significantly enhanced model accuracy, (ii) the unprecedented levels of computational performance which will allow large-scale ultrasound simulations in previously unrealisable compute times, (iii) the low barrier to entry for end-users that are not experts in HPC, and (iv) adherence to the regulatory framework required for the clinical application of scientific software.

Academic and Industrial Impact: Acoustic models are used for many applications outside biomedical ultrasound, including non-destructive evaluation, seismology, oil and gas exploration, architectural acoustics, ultrasonic welding, and ocean acoustics. Our efficient large-scale models will be of direct relevance to these fields, offering improved modelling accuracy and unprecedented computational performance. The techniques for domain decomposition and the optimisation of the code for emerging many-core HPC architectures will also be of interest to those modelling waves in other contexts where spectral methods are commonly used, for example in computational electromagnetics and computational quantum chemistry.

Economic Impact: The enhanced computational performance, clinician-led user interface design, and regulatory compliance of the software developed during this project will provide a significant competitive edge over the simulation packages currently used in academia and industry. These advances will make the software commercially attractive to medical device companies. It is expected the generated IP will lead to licensing agreements or the development of new spinout companies, with the UK becoming a base for future international investment into computational acoustics and clinical applications of therapeutic ultrasound.

Publications

10 25 50

publication icon
Brown M (2016) Control of broadband optically generated ultrasound pulses using binary amplitude holograms in The Journal of the Acoustical Society of America

publication icon
Martin E (2016) Simulating Focused Ultrasound Transducers Using Discrete Sources on Regular Cartesian Grids. in IEEE transactions on ultrasonics, ferroelectrics, and frequency control

publication icon
Suomi V (2016) Nonlinear 3-D simulation of high-intensity focused ultrasound therapy in the Kidney. in Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference

publication icon
Jaros J (2015) Large-scale Ultrasound Simulations Using the Hybrid OpenMP/MPI Decomposition in Proceedings of the 3rd International Conference on Exascale Applications and Software

publication icon
Jaros J (2015) Large-Scale Ultrasound Simulations with Local Fourier Basis Decomposition in SC15: International Conference for High Performance Computing, Networking, Storage and Analysis

publication icon
Jaros J (2015) Full-wave nonlinear ultrasound simulation on distributed clusters with applications in high-intensity focused ultrasound in The International Journal of High Performance Computing Applications

 
Description The aim of this project was to develop next generation modelling tools for therapeutic ultrasound that scale efficiently across large computer clusters, and can be easily accessed by end users such as clinicians. The project has now been completed, and these objectives have been achieved. First, new domain decomposition techniques have been developed that allow the computational problem to be more efficiently mapped onto supercomputing resources. These include pencil and local domain decomposition techniques for global domain decomposition, and highly efficient local domain decomposition based on local Fourier basis for running simulations on a cluster of graphical processing units (GPUs). Second, the backend of a clinical interface has been established, including pipelines for secure data transfer between local and remote compute resources, new techniques for on-the-fly visualisation and compression, and the development of a transducer library for several single element therapy transducers. Third, a regulatory QMS framework has been put in place for the clinical user interface. This allows the use of the software within a medical context, and for the software to be later CE-marked as a medical device. Finally, the models have been validated using clinical data from several partner sites. These results have been disseminated through a large number of scientific publications in high-quality journals and conferences, and through open-source software releases as part of the k-Wave Toolbox. This toolbox now has more than 11,000 registered users world-wide, making it one of the most widely used tools in acoustics.
Exploitation Route The developed software tools are likely to generate impact both on the targeted delivery of ultrasound therapy, as well as academics working to understand wave physics and ultrasound metrology. This impact is likely to come about through our open-source software releases, as well as the development of new modelling and domain decomposition approaches that can be applied in many other fields.
Sectors Digital/Communication/Information Technologies (including Software),Healthcare

 
Description The impact arising from this grant arises in two key areas: Model development: We developed new models to address the challenges of physical complexity and computational scale. In particular, we: (1) developed new governing equations that describe ultrasound absorption in bones, (2) developed a new open-source elastic wave model based on Fourier spectral methods, (3) developed new domain decomposition techniques that allow ultrasound simulations to efficiently scale over thousands of computer cores or hundreds of GPUs, (4) developed adaptive moving-mesh schemes based on local bandwidth that allow simulations to be performed with significantly less computer memory, (5) generalised the idea of k-space methods to establish non-standard PSTD schemes that allow the exact solution of time-dependent partial differential equations, (6) applied the non-standard PSTD framework to develop a highly-efficient solver for Pennes' bioheat equation that allows tissue-realistic coupled acoustic and thermal simulations, and (7) developed fast one-step methods for rapidly calculating the acoustic field from continuous wave sources. These developments represent a significant contribution to the state-of-the-art. The acoustic models are being used by ultrasound and acoustic companies in the private sector, including for transducer design, image reconstruction, and ultrasound dose calculations. Translation and planning interface: We translated the treatment planning tools into a software suite that can be used by clinicians and researchers. The developed software package, called k-Plan, is an advanced modelling tool for precision planning of transcranial ultrasound procedures. It uses a streamlined and intuitive workflow that allows users to select an ultrasound device, position the device using a template or medical image, and specify the sonication parameters. High-resolution calculations of the ultrasound field and temperature inside the skull and brain are then automatically calculated in the cloud with a single click. No knowledge of numerical modelling or high-performance computing is required.
First Year Of Impact 2016
Sector Digital/Communication/Information Technologies (including Software),Education,Healthcare,Manufacturing, including Industrial Biotechology
Impact Types Economic

 
Description Capital Award for Core Equipment at UCL
Amount £650,000 (GBP)
Funding ID EP/T023651/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 11/2019 
End 05/2021
 
Description Spectral element methods for fractional differential equations, with applications in applied analysis and medical imaging
Amount £103,887 (GBP)
Funding ID EP/T022280/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 07/2021 
End 06/2024
 
Description ThUNDDAR Network Pilot Funding
Amount £49,298 (GBP)
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 09/2018 
End 02/2019
 
Description UCL EPSRC IAA 2022-25 FUNDING
Amount £87,417 (GBP)
Funding ID EPSRC IAA 2022-25 KEI2022-02-03 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 02/2023 
End 01/2024
 
Description UCL Knowledge Exchange and Innovation Fund
Amount £86,087 (GBP)
Funding ID EPSRC IAA 2017-20 Discovery-To-Use 
Organisation University College London 
Sector Academic/University
Country United Kingdom
Start 01/2021 
End 12/2021
 
Description UCL Knowledge Exchange and Innovation Fund
Amount £28,163 (GBP)
Organisation UCL Business 
Sector Private
Country United Kingdom
Start 03/2018 
End 09/2018
 
Title Repeatability and reproducibility of hydrophone measurements of medical ultrasound fields 
Description This data was collected in order to study the repeatability and reproducibility of hydrophone measurements of ultrasound fields. Sets of independent measurements were made with two probe (0.2 mm, 40 µm) and two membrane hydrophones (0.4 mm, 0.2 mm differential) (all from Precision Acoustics) to examine the repeatability of measurements. The pressures measured by these hydrophones in three different ultrasound fields, with both linear and nonlinear, pulsed and steady state driving conditions, were acquired to assess the reproducibility of measurements between hydrophones. Repeatability measurements: Sets of five independent measurements were made with each hydrophone of the field generated by a single element focusing bowl transducer (Sonic Concepts H151) driven at a frequency of 1.1 MHz, with both a 4 cycle burst and under quasi steady state conditions. Axial and lateral line scans passing through the focus were acquired at a drive level which generated a weakly nonlinear field. Reproducibility measurements: Two single element focusing bowl transducers (H151 at 1.1 MHz, and H101 at 3.3 MHz, Sonic Concepts) and one diagnostic linear array (L14-5 at 5 MHz, Ultrasonix) sources were used. For the single element transducers, axial and lateral line scans passing through the focus were acquired with each hydrophone at two drive levels to generate both a linear and a weakly nonlinear field, with both a 4 cycle burst and under quasi steady state conditions. For the diagnostic linear array, lateral line scans were acquired passing through the beam axis at an axial distance of 40 mm. The transducer was driven with a 4 cycle burst at a power level that generated harmonics up to 30 MHz.All measurements were acquired using an automated scanning tank filled with degassed, deionised water. The transducers mounted in a fixed xyz position with automated tilt, rotate adjustment. Hydrophones were mounted on an automated xyz stage, with manual tilt, rotate adjustment. In total this study contains 12 datasets, the corresponding figure or table in the paper is given in brackets: 1-4: Repeatability and reproducibility - H151 x 4 hydrophones (Figs 1-4, Table 3) Each dataset contains axial and lateral line scans at 2 drive levels, with a 4 cycle and a 40 cycle burst, with 5 sets of scans at the high drive level and one set of scans at the low drive level 5-8: Reproducibility - H101 x 4 hydrophones (Figs 4-5, Table 3) Each dataset contains a single set of axial and lateral line scan at each of 2 drive levels, with a 4 cycle and a 120 cycle burst. 9-12: Reproducibility - L14-5 x 4 hydrophones (Fig 6, Table 3) Each dataset contains lateral scans at 1 power level. 
Type Of Material Database/Collection of data 
Year Produced 2021 
Provided To Others? Yes  
Impact Experimental measurements are critical for the development of medical ultrasound software and devices, including for validation of modelling tools and for comparison of measurement equipment and protocols. Data sharing encourages reproducibility and consistency across labs, and provides access to other researchers who may not have the equipment or expertise to conduct their own measurements. 
URL https://rdr.ucl.ac.uk/articles/dataset/Repeatability_and_reproducibility_of_hydrophone_measurements_...
 
Title Simulating Focused Ultrasound Transducers using Discrete Sources on Regular Cartesian Grids 
Description This data was collected in order to validate models of curved sources on cartesian grids. A single element spherically focusing ultrasound transducer (H101 at 1.1 MHz, Sonic Concepts) was used to generate an acoustic field. Measurements were performed with a 0.2 mm PVDF needle hydrophone (Precision Acoustics) to characterise the source under quasi continuous wave and short burst conditions. These measurements include planar scans in the prefocal region for the two driving regimes, and axial scans at the same drive level for both drive regimes. There are additional axial scans at one further higher drive level (very weakly nonlinear) for each of the driving regimes which were acquired for comparison with the model with scaled input source amplitude. All measurements were acquired using an automated scanning tank filled with degassed, deionised water. The transducers mounted in a fixed xyz position with automated tilt, rotate adjustment. Hydrophones were mounted on an automated xyz stage, with manual tilt, rotate adjustment. In total this study contains 6 datasets contained in one file, the corresponding figure or table in the paper is given in brackets: 1: Planar scan with 45 cycle burst (qCW) at z = 42.5 mm, linear field 2: Axial scan 45 cycle burst (qCW), linear field (conditions as in 1), Fig 8, 9. 3: Axial scan 45 cycle burst (qCW), weakly nonlinear field, Fig 8, 9. 4: Planar scan with 4 cycle burst at z = 42.5 mm, linear field 5: Axial scan 4 cycle burst, linear field (conditions as in 4), Fig 10, 11. 6. Axial scan 4 cycle burst, weakly nonlinear field, Fig 10, 11. 
Type Of Material Database/Collection of data 
Year Produced 2021 
Provided To Others? Yes  
Impact Experimental measurements are critical for the development of medical ultrasound software and devices, including for validation of modelling tools and for comparison of measurement equipment and protocols. Data sharing encourages reproducibility and consistency across labs, and provides access to other researchers who may not have the equipment or expertise to conduct their own measurements. 
URL https://rdr.ucl.ac.uk/articles/dataset/Simulating_Focused_Ultrasound_Transducers_using_Discrete_Sour...
 
Title k-Plan: Ultrasound Therapy Planning 
Description k-Plan is an advanced modelling tool for precision planning of transcranial ultrasound procedures. It uses a streamlined and intuitive workflow that allows users to select an ultrasound device, position the device using a template or medical image, and specify the sonication parameters. High-resolution calculations of the ultrasound field and temperature inside the skull and brain are then automatically calculated in the cloud with a single click. 
Type Of Technology Software 
Year Produced 2022 
Impact k-Plan is developed by researchers at University College London and the Brno University of Technology based on more than a decade of cutting-edge research into ultrasound modelling and planning for transcranial ultrasound therapy. It is the first software tool for model-based treatment planning for ultrasound therapy, and is being brought to market in collaboration with Brainbox, Ltd. 
URL https://k-plan.io/
 
Title k-Wave Acoustics Toolbox 
Description k-Wave is an open source MATLAB toolbox designed for the time-domain simulation of propagating acoustic waves in 1D, 2D, or 3D. The toolbox has a wide range of functionality, but at its heart is an advanced numerical model that can account for both linear and nonlinear wave propagation, an arbitrary distribution of heterogeneous material parameters, and power law acoustic absorption. The numerical model is based on the solution of three coupled first-order partial differential equations which are equivalent to a generalised form of the Westervelt equation. The equations are solved using a k-space pseudospectral method, where spatial gradients are calculated using a Fourier collocation scheme, and temporal gradients are calculated using a k-space corrected finite-difference scheme. The temporal scheme is exact in the limit of linear wave propagation in a homogeneous and lossless medium, and significantly reduces numerical dispersion in the more general case. Power law acoustic absorption is accounted for using a linear integro-differential operator based on the fractional Laplacian. A split-field perfectly matched layer (PML) is used to absorb the waves at the edges of the computational domain. The main advantage of the numerical model used in k-Wave compared to models based on finite-difference time domain (FDTD) schemes is that fewer spatial and temporal grid points are needed for accurate simulations. This means the models run faster and use less memory. Major k-Wave versions were released in 2014, 2017, and 2020. 
Type Of Technology Software 
Year Produced 2020 
Open Source License? Yes  
Impact The toolbox is widely used in academia and industry, and has been used for research into transcranial ultrasound, ultrasound therapy, the development of novel ultrasound sources, photoacoustic imaging, and many other applications. There have been ten releases of the toolbox. It currently has more than 16,000 registered users in 70 countries. A 2010 paper describing the first release of the toolbox has >1400 citations, and the active online user forum has >4500 posts.