Extreme volumetric imaging using single-shot optical tomography with compressive sensing

Lead Research Organisation: Imperial College London
Department Name: Physics

Abstract

Standard approaches to 3D imaging typically rely on taking a series of 2D images one after another and reconstructing the 3D volume. This puts an upper limit on the volumetric frame rate achievable, since it requires a certain amount of time to record the 10s-1000s of individual 2D images separately. This project aims to develop an imaging platform capable of full 3D volumetric imaging at acquisition rates exceeding 1000 volumes per second by acquiring multiple images in parallel using cost-effective high-speed cameras.

The approach is based on the use of 4 high-speed cameras capable of frame rates in excess of 1000 frames per second. Previously, cameras that could operate at such high frame rates were prohibitively expensive, but recent advances in the technology has seen their price reduce significantly and brought them in line with the cost of more standard scientific cameras. Each camera will be combined with a novel optical system that produces two side-by-side images on the camera sensor, each viewing the sample at a slightly different angle. The four cameras will be placed around the sample and acquire eight images from different directions simultaneously, running at 1000s of frames a second. Each set of 8 images can them be combined to reconstruct the 3D scene, producing a 3D data set running at 1000s of volumes per second.

The particular arrangement of the cameras and the approach to reconstruction will depend on the sample being imaged. In this project we will apply this extreme volumetric imaging system to two experimental investigations. To image semi-transparent biological samples a technique analogous to X-ray computed tomography, called optical projection tomography (OPT), will be applied. While typical OPT data sets require 100s of angularly resolved images, by applying advances computational analysis techniques we will be able to reconstruct the volume from the eight images acquired. This will permit OPT imaging of samples that have not been immobilized (e.g. no need for anesthesia) and can measure very fast biological processes in 3D, such as transient cell signaling events.

While the use of high-speed cameras has been limited in biological imaging, they have been routinely used in fluid dynamics experiments to capture very fast events (e.g. droplet-surface interactions). We will extend this to full 3D imaging of such interactions by imaging them from eight different directions simultaneously and performing 3D reconstructions for every set of images (i.e. running at 1000s of volumes per second). For single droplet interactions we will employ surface measurement and reconstruction techniques, while for sprays (i.e. large number of droplets) we will employ particle tracking and optical scattering to quantify the trajectories and size of all the droplets. These experiments will be used to confirm and develop computational fluid dynamic simulations.

Publications

10 25 50
 
Description During this project we have developed two single-shot optical tomography systems for rapid volumetric imaging for applications in both in vivo biological studies and imaging of fluid dynamics/flows. These instruments are based on the projection approach to 3D imaging, as used in X ray computed tomography, where a series of wide-field images of the sample from different perspectives are acquired and then used to reconstruct the 3D structure. In this project we realised single-shot tomography by acquiring 8 images at different angles simultaneously on 4 cameras, producing a series of 3D reconstruction with a time-lapse resolution defined by the frame rate of the cameras. This compares to the standard approach to projection tomography, where the images are acquired sequentially and therefore a can only produce a reconstruction once all the images have been acquired, which not only limits the volumetric frame-rate, but can also lead to significant artefacts if the sample moves during the acquisition time.

The first of these instruments uses 4 cost-effective CMOS cameras that are significantly cheaper than the standard scientific sCMOS cameras typically used for biological microscopy/imaging. Operating at a magnification of 2.8x, it reconstructs a volume of ~1mm^3 and is therefore applicable to transparent model organisms, such as zebrafish embryos and nematode worms, or potentially larger cell culture assays such as spheroids. With a component cost less than £5000, we demonstrated synchronised operation at 62.5 frames per second, therefore producing independent volumetric reconstructions at a volumetric time-lapse resolution of 16 ms. Using this we demonstrated the first, to our knowledge, full volumetric imaging of free-swimming non anaesthetised zebrafish embryos, for both transmitted light imaging and fluorescence imaging. In addition to free-swimming model organisms, we are exploring its application to rapid 3D imaging/characterisation in a high-throughput flow implementation, where numerous samples/organisms could potentially be imaged in 3D per minute as they flow through the volume of interest.

The second of these instruments uses 4 high-speed cameras to record the 8 projection images, which can operate full-frame at 2000 frame per second. With a magnification of 0.5x, it reconstructs a volume of ~1cm^3 and is applicable to higher-speed phenomena such as fluid dynamics interactions, with a volumetric time-lapse resolution of 0.5 ms.

Both systems employ a novel telecentric imaging system design that acquires 2 images side by-side on a single camera at a relative viewing angle up to a few degrees. This design uses commercially available stock optics, consisting of a single objective lens and a pair of laterally shifted secondary lenses and associated apertures, producing a composite tube lens. We have identified companies that can cut the lenses as required and use 3D printed mounts to hold the composite lenses, making this approach accessible to other research institutions and groups that do not have access to in-house opto-mechanical facilities.
Exploitation Route We have demonstrated that, through a combination of multiplexed imaging (i.e. 2 images acquired per camera) in combination with compressed sensing and advanced reconstruction techniques, it is possible to acquire tomographic data at the frame-rate of the imaging cameras, including high-speed cameras operating at 1000's of frames per second. This could provide vital experimental data for challenging 3D dynamic samples, such as fluid dynamic, combustion and mixing systems for comparison with and the development of sophisticated physical models and computational simulations (e.g. in chemical engineering). The significant reduction in acquisition time could also improve the through-put for assay-type applications, where multiple samples, time-points and/or conditions need to be imaged. This technology could be incorporated into a flow cytometer or robotic handling protocol for imaging model organisms and/or plants in pharmaceutical or agricultural research. Initially we aim to transfer this technology and know-how through open-source routes in academia, with potentially longer-term commercialisation of the approaches with SME partners for commercial applications.
Sectors Agriculture

Food and Drink

Chemicals

Pharmaceuticals and Medical Biotechnology

 
Description We are collaborating with an SME to make our open-hardware designs available as commercial modular systems for research institutions/groups.
First Year Of Impact 2021
Sector Pharmaceuticals and Medical Biotechnology
 
Title Multiplexed telecentric imaging system 
Description A combination of a compound tube lens, consisting of two laterally shifted lenses, with a matching laterally shifted double aperture mask to produce an object-side telecentric dual channel imaging system. This provides two side-by-side projection images on a single camera sensor acquired at a few degrees relative angle of view. 
Type Of Technology New/Improved Technique/Technology 
Year Produced 2021 
Impact Optical projection tomography (OPT) requires a reduced imaging numerical aperture to achieve and increased depth of field. Therefore most of the numerical aperture of typical imaging lenses is not used. Our multiplexed imaging system uses twice as much of the available numerical aperture by using two laterally shifted apertures (each with its own laterally shifted tube lens) rather than one on-axis aperture. This doubles the light-collection efficiency and the multiplexed angular acquisition reduces the overall acquisition time for standard OPT by half. As demonstrated in this project, single-shot acquisition can be realised when multiple imaging systems are triggered in parallel. 
 
Description Biophotonics Congress 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Single-Shot Volumetric Imaging Using Optical Projection Tomography
Connor Darling, Samuel P. X. Davis, Sunil Kumar, Paul M. W. French, and James McGinty
We present a single-shot volumetric imaging method, utilising optical projection tomography. We record up to 70 1x1x1.9mm full-field volumes/second by recording projections simultaneously and implementing compressive sensing and machine learning.
Year(s) Of Engagement Activity 2021
URL https://opg.optica.org/abstract.cfm?URI=NTM-2021-NF2C.2
 
Description European Congress on Biomedical Optics (ECBO) conference presentation 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Single-shot volumetric imaging using optical projection tomography
Connor Darling, Samuel P. X. Davis, Sunil Kumar, Paul M. W. French, James McGinty
We present a single-shot volumetric imaging method, utilizing optical projection tomography. We record projections simultaneously, implementing compressive sensing and machine learning to record up to 70 (camera limited) 1x1x1.9mm volumes/second.
Year(s) Of Engagement Activity 2021
URL https://www.spiedigitallibrary.org/conference-proceedings-of-spie/11922/2615706/Single-shot-volumetr...
 
Description Focus on Microscopy conference presentation 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Single-shot full-field volumetric imaging using optical projection tomography
Connor Darling, Samuel P. X. Davis, Sunil Kumar, Paul M. W. French, James McGinty
We present a single-shot volumetric imaging method, capable of imaging mm-sized samples at up to 70 volumes/second utilising optical projection tomography (OPT) implemented with compressive sensing and machine learning. The technique can utilise absorption and/or fluorescence contrast in weakly scattering samples.
Year(s) Of Engagement Activity 2021
URL https://www.focusonmicroscopy.org/past/2021/PDF/1151_Darling.pdf