Application of novel computing and data analysis methods in electron microscopy

Lead Research Organisation: University of Warwick
Department Name: Physics

Abstract

The volume of data generated by electron microscopes is increasing rapidly as detector technology evolves. Aquisition of a data set over time, to increase signal-to-noise ratio, to study dynamic processes, or while some imaging parameter is varied, is now becoming common and can generate MB or TB of data. This project will work on methods to extract meaningful information from large datasets and use computer control to acquire new types of data. The techniques will be applied to materials produced by the ADEPT programme grant, based in Southampton, including phase change memory materials and thermoelectric nanowire materials.

Publications

10 25 50
publication icon
Ede Jeffrey M. (2019) Partial Scan Electron Microscopy with Deep Learning in arXiv e-prints

publication icon
Ede Jeffrey M. (2018) Autoencoders, Kernels, and Multilayer Perceptrons for Electron Micrograph Restoration and Compression in arXiv e-prints

publication icon
Ede Jeffrey M. (2019) Adaptive Learning Rate Clipping Stabilizes Learning in arXiv e-prints

publication icon
Ede Jeffrey M. (2020) Exit Wavefunction Reconstruction from Single Transmission Electron Micrographs with Deep Learning in arXiv e-prints

publication icon
Ede Jeffrey M. (2019) Deep Learning Supersampled Scanning Transmission Electron Microscopy in arXiv e-prints

publication icon
Ede JM (2019) Improving electron micrograph signal-to-noise with an atrous convolutional encoder-decoder. in Ultramicroscopy

Studentship Projects

Project Reference Relationship Related To Start End Student Name
EP/N509796/1 01/10/2016 30/09/2021
1917382 Studentship EP/N509796/1 02/10/2017 01/02/2021 Jeffrey Ede
 
Description A lot. I'll list a key point for each of my first author papers/preprints:

1. Signal-to-noise ratios of images taken by electron microscopes can be improved by deep learning, achieving higher performance than traditional methods.
2. Adapting neural network learning rates to reduce trainable parameter perturbations by high loss spikes. This prevents high loss spikes from destabilizing learning.
3. Half of wavefunction information is undetected by conventional transmission electron microscopy. Neural networks can recover lost phase information.
4. Neural networks can completed from partial scans, reducing electron dose and scan time by over an order of magnitude.
5. Neural networks can supersample electron micrographs, reducing electron dose and scan time by over an order of magnitude.
6. Half of wavefunction information is undetected by measurement. Neural networks can recover lost phase information in transmission electron microscopy.
Exploitation Route URL field is only accepting one url... Other key URLs: https://warwick.ac.uk/fac/sci/physics/research/condensedmatt/microscopy/research/machinelearning/ and https://arxiv.org/search/?query=jeffrey+ede&searchtype=all&source=header

1. To improve signal-to-noise of electron micrographs.
2. To stabilize neural network training, especially for small batch sizes.
3. To recover exit wavefunctions. These contain structure information, can be used for aberration correction, information storage (as holograms), thickness measurement, and imaging electronic and magnetic structures. In our paper, we propose a new approach to holography that builds on our research.
4. and 5. To reduce electron dose and scan time.
6. Recovering lost wavefunction information.
Sectors Digital/Communication/Information Technologies (including Software),Other
URL https://github.com/Jeffrey-Ede?tab=repositories
 
Title Adaptive Learning Rate Clipping 
Description Artificial neural network training with small batch sizes can be destabilized by high loss spikes. I developed a method to stabilize learning that complements existing methods. 
Type Of Material Technology assay or reagent 
Year Produced 2019 
Provided To Others? Yes  
Impact Submitted for publication. Used extensively in my following research. 
URL https://github.com/Jeffrey-Ede/ALRC
 
Title Beanland Atlas 
Description C++ code for fast automation of large angle convergent beam electron diffraction patterns. 
Type Of Material Technology assay or reagent 
Year Produced 2018 
Provided To Others? Yes  
Impact Can be applied/built upon to automate data collection. 
URL https://github.com/Jeffrey-Ede/Beanland-Atlas
 
Title DLSS STEM 
Description Deep learning supersampling of scanning transmission electron micrographs can be used to decrease electron dose and scan time, and increase resolution. 
Type Of Material Technology assay or reagent 
Year Produced 2019 
Provided To Others? Yes  
Impact Pretrained models are available so neural networks can be directly applied to images. Source code is available for further training or development. 
URL https://github.com/Jeffrey-Ede/DLSS-STEM
 
Title Electron Micrograph Denoiser 
Description Demonstrated that neural networks can improve electron micrograph signal to noise better than traditional methods. 
Type Of Material Technology assay or reagent 
Year Produced 2019 
Provided To Others? Yes  
Impact Published in Ultramicroscopy. Starting point for future research, and establishes neural network capability. Pretrained neural networks have been made publicly available that can be directly applied to images. 
URL https://github.com/Jeffrey-Ede/Electron-Micrograph-Denoiser
 
Title Electron Microscopy Datasets 
Description New publicly accessible dataserver for electron microscopy machine learning datasets and supporting materials. New datasets include 17267 TEM images and 16227 STEM images collected from University of Warwick dataservers. "Mini" datasets where images are downsampled to 96x96 have been made available to encourage rapid develop. New datasets containing 98340 exit wavefunction simulated with clTEM are also available. 
Type Of Material Improvements to research infrastructure 
Year Produced 2019 
Provided To Others? Yes  
Impact Two requests for the TEM dataset. The datasets are also used extensively in my research. At the moment, impact is limited as our first papers since setting up our publicly accessible dataserver are under review, or still in preparation. 
URL https://warwick.ac.uk/fac/sci/physics/research/condensedmatt/microscopy/research/machinelearning/
 
Title GitHub Repositories 
Description Source code. Some of which has been published, will be published (currently preprints), or is for miscellaneous projects. Providing source code can massively reduce the time needed to reproduce work. Even if work is reproduced or extended on in another language, the original source code can be a valuable reference. 
Type Of Material Technology assay or reagent 
Year Produced 2018 
Provided To Others? Yes  
Impact Research reproducible, easier to build upon. 
URL https://github.com/Jeffrey-Ede?tab=repositories
 
Title Partial STEM 
Description Neural networks to complete electron micrographs from partial scans, reducing electron dose and scan time. 
Type Of Material Technology assay or reagent 
Year Produced 2019 
Provided To Others? Yes  
Impact Pretrained models are available that can be directly applied to partial scans. Source code is available for further training or development. 
URL https://github.com/Jeffrey-Ede/partial-STEM
 
Title Electron Microscopy Datasets 
Description New publicly accessible dataserver for electron microscopy machine learning datasets and supporting materials. New datasets include 17267 TEM images and 16227 STEM images collected from University of Warwick dataservers. "Mini" datasets where images are downsampled to 96x96 have been made available to encourage rapid development. New datasets containing 98340 exit wavefunction simulated with clTEM are also available, alongside 1000 experimental focal series. 
Type Of Material Database/Collection of data 
Year Produced 2019 
Provided To Others? Yes  
Impact Two requests for the TEM dataset. The datasets are also used extensively in my research. At the moment, impact is limited as our first papers since setting up our publicly accessible dataserver are under review, or still in preparation. 
URL https://warwick.ac.uk/fac/sci/physics/research/condensedmatt/microscopy/research/machinelearning/
 
Title GitHub Repositories 
Description Source code. Some of which has been published, will be published (currently preprints), or is for miscellaneous projects. Providing source code can massively reduce the time needed to reproduce work. Even if work is reproduced or extended on in another language, the original source code can be a valuable reference. Source code also improves clararity. For example. deep learning repositories contain source code for neural networks, and therefore exact graphs. 
Type Of Material Computer model/algorithm 
Year Produced 2018 
Provided To Others? Yes  
Impact Research is reproducible, clearer, and easier to build upon. 
URL https://github.com/Jeffrey-Ede?tab=repositories
 
Description Neutrino Vertex Classification 
Organisation University of Warwick
Department Department of Physics
Country United Kingdom 
Sector Academic/University 
PI Contribution Collaboration with a particle physicist to a develop neural networks to classify neural vertex positions in the new DUNE detector. I proposed the project, taught my collaborator how to train neural networks, directed him to resources, and proposed machine learning models.
Collaborator Contribution Jhanzeb Ahmed aquired training data, and trained neural networks. He was also investigating support vector machines.
Impact The main outcome is the machine learning skills transferred to my collaborator. He's now transferred to another project. He's hoping to finish experiments afterwards.
Start Year 2019
 
Description One-Shot Exit Wavefunction Reconstruction 
Organisation University of Warwick
Department Department of Physics
Country United Kingdom 
Sector Academic/University 
PI Contribution Half of wavefunction information is lost at measurement. I proposed that deep learning can be used to recover lost phase information, and formed a collaboration to investigate. As lead investigator, I'm responsible for the idea, organization and delegation of roles, neural network training and validation, multislice wavefunction simulations, repositing code, repositing new datasets, lead authorship, manuscript submission, and being corresponding author.
Collaborator Contribution Jonathan P. P. Peters maintains clTEM multislice simulation code. He helped me get clTEM set up, and jointly wrote a script to automate exit wavefunction simulations. Jeremy Sloan internally reviewed an early draft of our paper. He then used my data to create images of crystal structures to add a figure and improve a figure. Richard Beanland provided quality control for experimental focal series that wavefunctions can be reconstructed from. Jonathan P. P. Peters, Jeremy Sloan and Richard Beanland provided feedback to improve our manuscript.
Impact GitHub repository with source code, links to pre-trained models, scripts to process data, perform analysis and create figures for paper, and compiled clTEM code for multislice simualtions used in our research: https://github.com/Jeffrey-Ede/One-Shot New wavefunction datasets: https://warwick.ac.uk/fac/sci/physics/research/condensedmatt/microscopy/research/machinelearning/ Manuscript submitted to Ultramicroscopy. This collaboration is multidisciplinary. Disciplines involved include electron microscopy, machine learning, and high-performance computing. High-performance computing is listed separately to machine learning as calculations involved OpenCL/C++ for GPU-accelerated multislice simulations used to generate training data.
Start Year 2019
 
Title GitHub Repositories 
Description GitHub repositories for new or improved techniques in electron microscopy/machine learning. Noteably, ALRC: Algorithm for adaptive learning rate clipping to stabilize deep learning. One-Shot: Pretrained models and source code for exit wavefunction reconstruction with deep learning. Electron-Micrograph-Denoiser: Pretrained models and source code for improving electron micrograph signal-to-noise with deep learning. partial-STEM: Pretrained models and source code for completing spirals and other partial scanning transmission electron micrographs with deep learning. DLSS-STEM: Pretrained models and source code for supersampling electron micrographs deep learning. 
Type Of Technology New/Improved Technique/Technology 
Year Produced 2019 
Impact Tools are available for use by others, and to be built upon. Source code also enhances reproducibility, and makes research clearer. 
URL https://github.com/Jeffrey-Ede?tab=repositories
 
Description arXiv Preprints 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Making preprints for research papers publicly available is an important step to lower barriers to science. It makes science directly available to the general public, academics, and industrial professional without potentially expensive subscriptions to journals. Importantly, it reduces dependence on secondary sources that summarize published work.

The main outputs are two requests for new 100+ GB training datasets introduced in papers, which I transferred via a Google datacenter. We've now set up our own publicly accessible dataserver to handle requests.

Another benefit is that preprint servers reduce delay before science is available. After the partial STEM preprint was published, I was contacted by a group doing the same research in America and we discussed methodology.
Year(s) Of Engagement Activity 2018,2019
URL https://arxiv.org/search/?query=jeffrey+ede&searchtype=all&source=header