Development & Clinical Translation of Scalable HPC Ultrasound Models
Lead Research Organisation:
University College London
Department Name: Medical Physics and Biomedical Eng
Abstract
The use of ultrasound as a diagnostic imaging tool is well known, particularly during pregnancy where ultrasound is used to create images of the developing foetus. In recent years, a growing number of therapeutic applications of ultrasound have also been demonstrated. The goal of therapeutic ultrasound is to modify the function or structure of the tissue, rather than produce an anatomical image. This is possible because the mechanical vibrations caused by the ultrasound waves can affect tissue in different ways, for example, by causing the tissue to heat up, or by generating internal radiation forces that can agitate the cells or tissue scaffolding. These ultrasound bioeffects offer a huge potential to develop new ways to treat major diseases such as cancer, to improve the delivery of drugs while minimising side-effects, and to treat a wide spectrum of neurological and psychiatric conditions.
The fundamental challenge shared by all applications of therapeutic ultrasound is that the ultrasound energy must be delivered accurately, safely, and non-invasively to the target region within the body. This is difficult because bones and other tissue interfaces can severely distort the shape of the ultrasound beam. This has a significant impact on the safety and effectiveness of therapeutic ultrasound, and presents a major hurdle for the wider clinical acceptance of these exciting technologies. In principle, any distortions to the ultrasound beam could be accounted for using advanced computer models. However, the underlying physics is complex, and the scale of the modelling problem requires extremely large amounts of computer memory. Using existing software, a single simulation running on a supercomputer can take many days to complete, which is too long to be clinically useful.
The aim of this proposal is to develop more efficient computer models to accurately predict how ultrasound waves travel through the human body. This will involve implementing new approaches that efficiently divide the computational problem across large numbers of interconnected computer cores on a supercomputer. New approaches to reduce the huge quantity of output data will also be implemented, including calculating clinically important parameters while the simulation runs, and optimising how the data is stored to disk. We will also develop a professional user interface and package the code within the regulatory framework required for medical software. This will allow end-users, such as doctors, to easily use the code for applications in therapeutic ultrasound without needing to be an expert in computer science. In collaboration with our clinical partners, the computer models will then be applied to different applications of therapeutic ultrasound to allow the precise delivery of ultrasound energy to be predicted for the individual patient.
The fundamental challenge shared by all applications of therapeutic ultrasound is that the ultrasound energy must be delivered accurately, safely, and non-invasively to the target region within the body. This is difficult because bones and other tissue interfaces can severely distort the shape of the ultrasound beam. This has a significant impact on the safety and effectiveness of therapeutic ultrasound, and presents a major hurdle for the wider clinical acceptance of these exciting technologies. In principle, any distortions to the ultrasound beam could be accounted for using advanced computer models. However, the underlying physics is complex, and the scale of the modelling problem requires extremely large amounts of computer memory. Using existing software, a single simulation running on a supercomputer can take many days to complete, which is too long to be clinically useful.
The aim of this proposal is to develop more efficient computer models to accurately predict how ultrasound waves travel through the human body. This will involve implementing new approaches that efficiently divide the computational problem across large numbers of interconnected computer cores on a supercomputer. New approaches to reduce the huge quantity of output data will also be implemented, including calculating clinically important parameters while the simulation runs, and optimising how the data is stored to disk. We will also develop a professional user interface and package the code within the regulatory framework required for medical software. This will allow end-users, such as doctors, to easily use the code for applications in therapeutic ultrasound without needing to be an expert in computer science. In collaboration with our clinical partners, the computer models will then be applied to different applications of therapeutic ultrasound to allow the precise delivery of ultrasound energy to be predicted for the individual patient.
Planned Impact
The novel software developed as part of this proposal will generate impact in two different ways. First, there is considerable potential for the computational models to impact clinical practice by extending and expediting the use of ultrasound as a tool to treat cancer and other diseases. Second, the computational performance of the models is likely to have significant impact in the many allied areas of research and industry in which waves play a role.
Clinical Impact: Over the last decade, novel applications of therapeutic ultrasound have excited much interest within the research community. However, wider clinical acceptance has been modest due to challenges in controlling the ultrasound fields with the required accuracy. The scalable k-space models and clinical user interface proposed in this application could overcome this barrier and act as a catalyst for the widespread clinical application of these technologies. Impact will arise from (i) the significantly enhanced model accuracy, (ii) the unprecedented levels of computational performance which will allow large-scale ultrasound simulations in previously unrealisable compute times, (iii) the low barrier to entry for end-users that are not experts in HPC, and (iv) adherence to the regulatory framework required for the clinical application of scientific software.
Academic and Industrial Impact: Acoustic models are used for many applications outside biomedical ultrasound, including non-destructive evaluation, seismology, oil and gas exploration, architectural acoustics, ultrasonic welding, and ocean acoustics. Our efficient large-scale models will be of direct relevance to these fields, offering improved modelling accuracy and unprecedented computational performance. The techniques for domain decomposition and the optimisation of the code for emerging many-core HPC architectures will also be of interest to those modelling waves in other contexts where spectral methods are commonly used, for example in computational electromagnetics and computational quantum chemistry.
Economic Impact: The enhanced computational performance, clinician-led user interface design, and regulatory compliance of the software developed during this project will provide a significant competitive edge over the simulation packages currently used in academia and industry. These advances will make the software commercially attractive to medical device companies. It is expected the generated IP will lead to licensing agreements or the development of new spinout companies, with the UK becoming a base for future international investment into computational acoustics and clinical applications of therapeutic ultrasound.
Clinical Impact: Over the last decade, novel applications of therapeutic ultrasound have excited much interest within the research community. However, wider clinical acceptance has been modest due to challenges in controlling the ultrasound fields with the required accuracy. The scalable k-space models and clinical user interface proposed in this application could overcome this barrier and act as a catalyst for the widespread clinical application of these technologies. Impact will arise from (i) the significantly enhanced model accuracy, (ii) the unprecedented levels of computational performance which will allow large-scale ultrasound simulations in previously unrealisable compute times, (iii) the low barrier to entry for end-users that are not experts in HPC, and (iv) adherence to the regulatory framework required for the clinical application of scientific software.
Academic and Industrial Impact: Acoustic models are used for many applications outside biomedical ultrasound, including non-destructive evaluation, seismology, oil and gas exploration, architectural acoustics, ultrasonic welding, and ocean acoustics. Our efficient large-scale models will be of direct relevance to these fields, offering improved modelling accuracy and unprecedented computational performance. The techniques for domain decomposition and the optimisation of the code for emerging many-core HPC architectures will also be of interest to those modelling waves in other contexts where spectral methods are commonly used, for example in computational electromagnetics and computational quantum chemistry.
Economic Impact: The enhanced computational performance, clinician-led user interface design, and regulatory compliance of the software developed during this project will provide a significant competitive edge over the simulation packages currently used in academia and industry. These advances will make the software commercially attractive to medical device companies. It is expected the generated IP will lead to licensing agreements or the development of new spinout companies, with the UK becoming a base for future international investment into computational acoustics and clinical applications of therapeutic ultrasound.
Publications
Suomi V
(2016)
Nonlinear 3-D simulation of high-intensity focused ultrasound therapy in the Kidney.
in Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference
Nabergoj Makovec U
(2023)
Pharmacist-led clinical medication review service in primary care: the perspective of general practitioners.
in BMC primary care
Wise E
(2017)
Mesh Density Functions Based on Local Bandwidth Applied to Moving Mesh Methods
in Communications in Computational Physics
Suomi V
(2018)
Full Modeling of High-Intensity Focused Ultrasound and Thermal Heating in the Kidney Using Realistic Patient Models.
in IEEE transactions on bio-medical engineering
Suomi V
(2018)
Full Modeling of High-Intensity Focused Ultrasound and Thermal Heating in the Kidney Using Realistic Patient Models.
in IEEE transactions on bio-medical engineering
Martin E
(2017)
Rapid Spatial Mapping of Focused Ultrasound Fields Using a Planar Fabry-Pérot Sensor.
in IEEE transactions on ultrasonics, ferroelectrics, and frequency control
Treeby B
(2018)
Equivalent-Source Acoustic Holography for Projecting Measured Ultrasound Fields Through Complex Media.
in IEEE transactions on ultrasonics, ferroelectrics, and frequency control
Martin E
(2016)
Simulating Focused Ultrasound Transducers Using Discrete Sources on Regular Cartesian Grids.
in IEEE transactions on ultrasonics, ferroelectrics, and frequency control
Arridge S
(2016)
On the adjoint operator in photoacoustic tomography
in Inverse Problems
Wise E
(2018)
Bandwidth-based mesh adaptation in multiple dimensions
in Journal of Computational Physics
Description | The aim of this project was to develop next generation modelling tools for therapeutic ultrasound that scale efficiently across large computer clusters, and can be easily accessed by end users such as clinicians. The project has now been completed, and these objectives have been achieved. First, new domain decomposition techniques have been developed that allow the computational problem to be more efficiently mapped onto supercomputing resources. These include pencil and local domain decomposition techniques for global domain decomposition, and highly efficient local domain decomposition based on local Fourier basis for running simulations on a cluster of graphical processing units (GPUs). Second, the backend of a clinical interface has been established, including pipelines for secure data transfer between local and remote compute resources, new techniques for on-the-fly visualisation and compression, and the development of a transducer library for several single element therapy transducers. Third, a regulatory QMS framework has been put in place for the clinical user interface. This allows the use of the software within a medical context, and for the software to be later CE-marked as a medical device. Finally, the models have been validated using clinical data from several partner sites. These results have been disseminated through a large number of scientific publications in high-quality journals and conferences, and through open-source software releases as part of the k-Wave Toolbox. This toolbox now has more than 11,000 registered users world-wide, making it one of the most widely used tools in acoustics. |
Exploitation Route | The developed software tools are likely to generate impact both on the targeted delivery of ultrasound therapy, as well as academics working to understand wave physics and ultrasound metrology. This impact is likely to come about through our open-source software releases, as well as the development of new modelling and domain decomposition approaches that can be applied in many other fields. |
Sectors | Digital/Communication/Information Technologies (including Software) Healthcare |
Description | The impact arising from this grant arises in two key areas: Model development: We developed new models to address the challenges of physical complexity and computational scale. In particular, we: (1) developed new governing equations that describe ultrasound absorption in bones, (2) developed a new open-source elastic wave model based on Fourier spectral methods, (3) developed new domain decomposition techniques that allow ultrasound simulations to efficiently scale over thousands of computer cores or hundreds of GPUs, (4) developed adaptive moving-mesh schemes based on local bandwidth that allow simulations to be performed with significantly less computer memory, (5) generalised the idea of k-space methods to establish non-standard PSTD schemes that allow the exact solution of time-dependent partial differential equations, (6) applied the non-standard PSTD framework to develop a highly-efficient solver for Pennes' bioheat equation that allows tissue-realistic coupled acoustic and thermal simulations, and (7) developed fast one-step methods for rapidly calculating the acoustic field from continuous wave sources. These developments represent a significant contribution to the state-of-the-art. The acoustic models are being used by ultrasound and acoustic companies in the private sector, including for transducer design, image reconstruction, and ultrasound dose calculations. Translation and planning interface: We translated the treatment planning tools into a software suite that can be used by clinicians and researchers. The developed software package, called k-Plan, is an advanced modelling tool for precision planning of transcranial ultrasound procedures. It uses a streamlined and intuitive workflow that allows users to select an ultrasound device, position the device using a template or medical image, and specify the sonication parameters. High-resolution calculations of the ultrasound field and temperature inside the skull and brain are then automatically calculated in the cloud with a single click. No knowledge of numerical modelling or high-performance computing is required. |
First Year Of Impact | 2016 |
Sector | Digital/Communication/Information Technologies (including Software),Education,Healthcare,Manufacturing, including Industrial Biotechology |
Impact Types | Economic |
Description | Capital Award for Core Equipment at UCL |
Amount | £650,000 (GBP) |
Funding ID | EP/T023651/1 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 11/2019 |
End | 05/2021 |
Description | Spectral element methods for fractional differential equations, with applications in applied analysis and medical imaging |
Amount | £103,887 (GBP) |
Funding ID | EP/T022280/1 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 06/2021 |
End | 06/2024 |
Description | ThUNDDAR Network Pilot Funding |
Amount | £49,298 (GBP) |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 08/2018 |
End | 02/2019 |
Description | UCL EPSRC IAA 2022-25 FUNDING |
Amount | £87,417 (GBP) |
Funding ID | EPSRC IAA 2022-25 KEI2022-02-03 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 02/2023 |
End | 01/2024 |
Description | UCL Knowledge Exchange and Innovation Fund |
Amount | £86,087 (GBP) |
Funding ID | EPSRC IAA 2017-20 Discovery-To-Use |
Organisation | University College London |
Sector | Academic/University |
Country | United Kingdom |
Start | 01/2021 |
End | 12/2021 |
Description | UCL Knowledge Exchange and Innovation Fund |
Amount | £28,163 (GBP) |
Organisation | UCL Business |
Sector | Private |
Country | United Kingdom |
Start | 03/2018 |
End | 09/2018 |
Title | Repeatability and reproducibility of hydrophone measurements of medical ultrasound fields |
Description | This data was collected in order to study the repeatability and reproducibility of hydrophone measurements of ultrasound fields. Sets of independent measurements were made with two probe (0.2 mm, 40 µm) and two membrane hydrophones (0.4 mm, 0.2 mm differential) (all from Precision Acoustics) to examine the repeatability of measurements. The pressures measured by these hydrophones in three different ultrasound fields, with both linear and nonlinear, pulsed and steady state driving conditions, were acquired to assess the reproducibility of measurements between hydrophones. Repeatability measurements: Sets of five independent measurements were made with each hydrophone of the field generated by a single element focusing bowl transducer (Sonic Concepts H151) driven at a frequency of 1.1 MHz, with both a 4 cycle burst and under quasi steady state conditions. Axial and lateral line scans passing through the focus were acquired at a drive level which generated a weakly nonlinear field. Reproducibility measurements: Two single element focusing bowl transducers (H151 at 1.1 MHz, and H101 at 3.3 MHz, Sonic Concepts) and one diagnostic linear array (L14-5 at 5 MHz, Ultrasonix) sources were used. For the single element transducers, axial and lateral line scans passing through the focus were acquired with each hydrophone at two drive levels to generate both a linear and a weakly nonlinear field, with both a 4 cycle burst and under quasi steady state conditions. For the diagnostic linear array, lateral line scans were acquired passing through the beam axis at an axial distance of 40 mm. The transducer was driven with a 4 cycle burst at a power level that generated harmonics up to 30 MHz.All measurements were acquired using an automated scanning tank filled with degassed, deionised water. The transducers mounted in a fixed xyz position with automated tilt, rotate adjustment. Hydrophones were mounted on an automated xyz stage, with manual tilt, rotate adjustment. In total this study contains 12 datasets, the corresponding figure or table in the paper is given in brackets: 1-4: Repeatability and reproducibility - H151 x 4 hydrophones (Figs 1-4, Table 3) Each dataset contains axial and lateral line scans at 2 drive levels, with a 4 cycle and a 40 cycle burst, with 5 sets of scans at the high drive level and one set of scans at the low drive level 5-8: Reproducibility - H101 x 4 hydrophones (Figs 4-5, Table 3) Each dataset contains a single set of axial and lateral line scan at each of 2 drive levels, with a 4 cycle and a 120 cycle burst. 9-12: Reproducibility - L14-5 x 4 hydrophones (Fig 6, Table 3) Each dataset contains lateral scans at 1 power level. |
Type Of Material | Database/Collection of data |
Year Produced | 2021 |
Provided To Others? | Yes |
Impact | Experimental measurements are critical for the development of medical ultrasound software and devices, including for validation of modelling tools and for comparison of measurement equipment and protocols. Data sharing encourages reproducibility and consistency across labs, and provides access to other researchers who may not have the equipment or expertise to conduct their own measurements. |
URL | https://rdr.ucl.ac.uk/articles/dataset/Repeatability_and_reproducibility_of_hydrophone_measurements_... |
Title | Sensitivity of simulated transcranial ultrasound fields to acoustic medium property maps |
Description | This data was collected in order to validate models of ultrasound propagation through skull bone phantoms. A single element spherically focusing ultrasound transducer (PA332 at 1 MHz, Precision Acoustics) was used to generate an acoustic field. Measurements were performed with a 0.2 mm PVDF needle hydrophone (Precision Acoustics) to characterise the source under short burst conditions (3 cycles). These measurements include planar scans in the prefocal region in free field for characterisation of the source, and planar scans further from the source after propagation through 3 different bone phantoms: a parametric araldite resin phantom, a mesh based skull bone phantom obtained from a T1 weighted MRI scan of the head, cast in araldite and printed in VeroBlack. Medium maps used in simulations, which match the experimental set up are included as a supplementary file, these include speed of sound, attenuation coefficient and density. The .stl file containing the mesh is also included as a supplementary file. All measurements were acquired using an automated scanning tank filled with degassed, deionised water. The transducer was mounted in a fixed xyz position. Hydrophones were mounted on an automated xyz stage, with manual tilt, rotate adjustment. In total this study contains 5 datasets contained 5 files, the corresponding figure or table in the paper is given in brackets: 1: Free field XY planar transverse scan at z = 45 mm, corresponds to dataset 2, which was performed in the same measurement session. 2. XY planar transverse scan at z = 58 mm after propagation through a parametric araldite 1302 resin phantom. (Fig 13) 3. Free field XY planar transverse scan at 45 mm, corresponds to datasets 4 and 5, which were performed in the same measurement session. 4. Planar transverse scan at z = 85 mm after propagation through a mesh based skull bone phantom cast in araldite 1302. (Fig 14) 5. Planar transverse scan at z = 85 mm after propagation though a mesh based skull bone phantom printed in VeroBlack. (Fig 14) Supplementary files: 6. *_supplementary_01.h5, h5 file containing fields medium1, medium2 and medium3, which contain grid based medium maps (sound speed, attenuation coefficient at 1 MHz, and density), with coordinates and description, for medium1: parametric resin phantom, medium2: mesh based resin phantom, medium3: mesh based VeroBlack phantom. 7. *_supplementary-02.stl, .stl file containing mesh used to construct the anatomical bone phantom |
Type Of Material | Database/Collection of data |
Year Produced | 2022 |
Provided To Others? | Yes |
URL | https://rdr.ucl.ac.uk/articles/dataset/Sensitivity_of_simulated_transcranial_ultrasound_fields_to_ac... |
Title | Simulating Focused Ultrasound Transducers using Discrete Sources on Regular Cartesian Grids |
Description | This data was collected in order to validate models of curved sources on cartesian grids. A single element spherically focusing ultrasound transducer (H101 at 1.1 MHz, Sonic Concepts) was used to generate an acoustic field. Measurements were performed with a 0.2 mm PVDF needle hydrophone (Precision Acoustics) to characterise the source under quasi continuous wave and short burst conditions. These measurements include planar scans in the prefocal region for the two driving regimes, and axial scans at the same drive level for both drive regimes. There are additional axial scans at one further higher drive level (very weakly nonlinear) for each of the driving regimes which were acquired for comparison with the model with scaled input source amplitude. All measurements were acquired using an automated scanning tank filled with degassed, deionised water. The transducers mounted in a fixed xyz position with automated tilt, rotate adjustment. Hydrophones were mounted on an automated xyz stage, with manual tilt, rotate adjustment. In total this study contains 6 datasets contained in one file, the corresponding figure or table in the paper is given in brackets: 1: Planar scan with 45 cycle burst (qCW) at z = 42.5 mm, linear field 2: Axial scan 45 cycle burst (qCW), linear field (conditions as in 1), Fig 8, 9. 3: Axial scan 45 cycle burst (qCW), weakly nonlinear field, Fig 8, 9. 4: Planar scan with 4 cycle burst at z = 42.5 mm, linear field 5: Axial scan 4 cycle burst, linear field (conditions as in 4), Fig 10, 11. 6. Axial scan 4 cycle burst, weakly nonlinear field, Fig 10, 11. |
Type Of Material | Database/Collection of data |
Year Produced | 2021 |
Provided To Others? | Yes |
Impact | Experimental measurements are critical for the development of medical ultrasound software and devices, including for validation of modelling tools and for comparison of measurement equipment and protocols. Data sharing encourages reproducibility and consistency across labs, and provides access to other researchers who may not have the equipment or expertise to conduct their own measurements. |
URL | https://rdr.ucl.ac.uk/articles/dataset/Simulating_Focused_Ultrasound_Transducers_using_Discrete_Sour... |
Title | k-Plan: Ultrasound Therapy Planning |
Description | k-Plan is an advanced modelling tool for precision planning of transcranial ultrasound procedures. It uses a streamlined and intuitive workflow that allows users to select an ultrasound device, position the device using a template or medical image, and specify the sonication parameters. High-resolution calculations of the ultrasound field and temperature inside the skull and brain are then automatically calculated in the cloud with a single click. |
Type Of Technology | Software |
Year Produced | 2022 |
Impact | k-Plan is developed by researchers at University College London and the Brno University of Technology based on more than a decade of cutting-edge research into ultrasound modelling and planning for transcranial ultrasound therapy. It is the first software tool for model-based treatment planning for ultrasound therapy, and is being brought to market in collaboration with Brainbox, Ltd. |
URL | https://k-plan.io/ |
Title | k-Wave Acoustics Toolbox |
Description | k-Wave is an open source MATLAB toolbox designed for the time-domain simulation of propagating acoustic waves in 1D, 2D, or 3D. The toolbox has a wide range of functionality, but at its heart is an advanced numerical model that can account for both linear and nonlinear wave propagation, an arbitrary distribution of heterogeneous material parameters, and power law acoustic absorption. The numerical model is based on the solution of three coupled first-order partial differential equations which are equivalent to a generalised form of the Westervelt equation. The equations are solved using a k-space pseudospectral method, where spatial gradients are calculated using a Fourier collocation scheme, and temporal gradients are calculated using a k-space corrected finite-difference scheme. The temporal scheme is exact in the limit of linear wave propagation in a homogeneous and lossless medium, and significantly reduces numerical dispersion in the more general case. Power law acoustic absorption is accounted for using a linear integro-differential operator based on the fractional Laplacian. A split-field perfectly matched layer (PML) is used to absorb the waves at the edges of the computational domain. The main advantage of the numerical model used in k-Wave compared to models based on finite-difference time domain (FDTD) schemes is that fewer spatial and temporal grid points are needed for accurate simulations. This means the models run faster and use less memory. Major k-Wave versions were released in 2014, 2017, and 2020. |
Type Of Technology | Software |
Year Produced | 2020 |
Open Source License? | Yes |
Impact | The toolbox is widely used in academia and industry, and has been used for research into transcranial ultrasound, ultrasound therapy, the development of novel ultrasound sources, photoacoustic imaging, and many other applications. There have been ten releases of the toolbox. It currently has more than 16,000 registered users in 70 countries. A 2010 paper describing the first release of the toolbox has >1400 citations, and the active online user forum has >4500 posts. |