Application of novel computing and data analysis methods in electron microscopy
Lead Research Organisation:
University of Warwick
Department Name: Physics
Abstract
The volume of data generated by electron microscopes is increasing rapidly as detector technology evolves. Aquisition of a data set over time, to increase signal-to-noise ratio, to study dynamic processes, or while some imaging parameter is varied, is now becoming common and can generate MB or TB of data. This project will work on methods to extract meaningful information from large datasets and use computer control to acquire new types of data. The techniques will be applied to materials produced by the ADEPT programme grant, based in Southampton, including phase change memory materials and thermoelectric nanowire materials.
People |
ORCID iD |
Richard Beanland (Primary Supervisor) | |
Jeffrey Ede (Student) |
Publications
Ede J
(2021)
Deep learning in electron microscopy
in Machine Learning: Science and Technology
Ede J
(2021)
Adaptive partial scanning transmission electron microscopy with reinforcement learning
in Machine Learning: Science and Technology
Ede Jeffrey M.
(2019)
Adaptive Learning Rate Clipping Stabilizes Learning
in arXiv e-prints
Ede Jeffrey M.
(2019)
Deep Learning Supersampled Scanning Transmission Electron Microscopy
in arXiv e-prints
Ede Jeffrey M.
(2019)
Partial Scan Electron Microscopy with Deep Learning
in arXiv e-prints
Ede Jeffrey M.
(2018)
Autoencoders, Kernels, and Multilayer Perceptrons for Electron Micrograph Restoration and Compression
in arXiv e-prints
Ede Jeffrey M.
(2020)
Exit Wavefunction Reconstruction from Single Transmission Electron Micrographs with Deep Learning
in arXiv e-prints
Ede JM
(2020)
Partial Scanning Transmission Electron Microscopy with Deep Learning.
in Scientific reports
Ede JM
(2019)
Improving electron micrograph signal-to-noise with an atrous convolutional encoder-decoder.
in Ultramicroscopy
Studentship Projects
Project Reference | Relationship | Related To | Start | End | Student Name |
---|---|---|---|---|---|
EP/N509796/1 | 30/09/2016 | 29/09/2021 | |||
1917382 | Studentship | EP/N509796/1 | 01/10/2017 | 01/02/2021 | Jeffrey Ede |
Description | A lot. I'll list a key point for each of my first author papers/preprints: 1. Signal-to-noise ratios of images taken by electron microscopes can be improved by deep learning, achieving higher performance than traditional methods. 2. Adapting neural network learning rates to reduce trainable parameter perturbations by high loss spikes. This prevents high loss spikes from destabilizing learning. 3. Half of wavefunction information is undetected by conventional transmission electron microscopy. Neural networks can recover lost phase information. 4. Neural networks can completed from partial scans, reducing electron dose and scan time by over an order of magnitude. 5. Neural networks can supersample electron micrographs, reducing electron dose and scan time by over an order of magnitude. 6. Half of wavefunction information is undetected by measurement. Neural networks can recover lost phase information in transmission electron microscopy. |
Exploitation Route | URL field is only accepting one url... Other key URLs: https://warwick.ac.uk/fac/sci/physics/research/condensedmatt/microscopy/research/machinelearning/ and https://arxiv.org/search/?query=jeffrey+ede&searchtype=all&source=header 1. To improve signal-to-noise of electron micrographs. 2. To stabilize neural network training, especially for small batch sizes. 3. To recover exit wavefunctions. These contain structure information, can be used for aberration correction, information storage (as holograms), thickness measurement, and imaging electronic and magnetic structures. In our paper, we propose a new approach to holography that builds on our research. 4. and 5. To reduce electron dose and scan time. 6. Recovering lost wavefunction information. |
Sectors | Digital/Communication/Information Technologies (including Software) Other |
URL | https://github.com/Jeffrey-Ede?tab=repositories |
Title | Adaptive Learning Rate Clipping |
Description | Artificial neural network training with small batch sizes can be destabilized by high loss spikes. I developed a method to stabilize learning that complements existing methods. |
Type Of Material | Technology assay or reagent |
Year Produced | 2019 |
Provided To Others? | Yes |
Impact | Submitted for publication. Used extensively in my following research. |
URL | https://github.com/Jeffrey-Ede/ALRC |
Title | Beanland Atlas |
Description | C++ code for fast automation of large angle convergent beam electron diffraction patterns. |
Type Of Material | Technology assay or reagent |
Year Produced | 2018 |
Provided To Others? | Yes |
Impact | Can be applied/built upon to automate data collection. |
URL | https://github.com/Jeffrey-Ede/Beanland-Atlas |
Title | DLSS STEM |
Description | Deep learning supersampling of scanning transmission electron micrographs can be used to decrease electron dose and scan time, and increase resolution. |
Type Of Material | Technology assay or reagent |
Year Produced | 2019 |
Provided To Others? | Yes |
Impact | Pretrained models are available so neural networks can be directly applied to images. Source code is available for further training or development. |
URL | https://github.com/Jeffrey-Ede/DLSS-STEM |
Title | Electron Micrograph Denoiser |
Description | Demonstrated that neural networks can improve electron micrograph signal to noise better than traditional methods. |
Type Of Material | Technology assay or reagent |
Year Produced | 2019 |
Provided To Others? | Yes |
Impact | Published in Ultramicroscopy. Starting point for future research, and establishes neural network capability. Pretrained neural networks have been made publicly available that can be directly applied to images. |
URL | https://github.com/Jeffrey-Ede/Electron-Micrograph-Denoiser |
Title | Electron Microscopy Datasets |
Description | New publicly accessible dataserver for electron microscopy machine learning datasets and supporting materials. New datasets include 17267 TEM images and 16227 STEM images collected from University of Warwick dataservers. "Mini" datasets where images are downsampled to 96x96 have been made available to encourage rapid develop. New datasets containing 98340 exit wavefunction simulated with clTEM are also available. |
Type Of Material | Improvements to research infrastructure |
Year Produced | 2019 |
Provided To Others? | Yes |
Impact | Two requests for the TEM dataset. The datasets are also used extensively in my research. At the moment, impact is limited as our first papers since setting up our publicly accessible dataserver are under review, or still in preparation. |
URL | https://warwick.ac.uk/fac/sci/physics/research/condensedmatt/microscopy/research/machinelearning/ |
Title | GitHub Repositories |
Description | Source code. Some of which has been published, will be published (currently preprints), or is for miscellaneous projects. Providing source code can massively reduce the time needed to reproduce work. Even if work is reproduced or extended on in another language, the original source code can be a valuable reference. |
Type Of Material | Technology assay or reagent |
Year Produced | 2018 |
Provided To Others? | Yes |
Impact | Research reproducible, easier to build upon. |
URL | https://github.com/Jeffrey-Ede?tab=repositories |
Title | Partial STEM |
Description | Neural networks to complete electron micrographs from partial scans, reducing electron dose and scan time. |
Type Of Material | Technology assay or reagent |
Year Produced | 2019 |
Provided To Others? | Yes |
Impact | Pretrained models are available that can be directly applied to partial scans. Source code is available for further training or development. |
URL | https://github.com/Jeffrey-Ede/partial-STEM |
Title | Electron Microscopy Datasets |
Description | New publicly accessible dataserver for electron microscopy machine learning datasets and supporting materials. New datasets include 17267 TEM images and 16227 STEM images collected from University of Warwick dataservers. "Mini" datasets where images are downsampled to 96x96 have been made available to encourage rapid development. New datasets containing 98340 exit wavefunction simulated with clTEM are also available, alongside 1000 experimental focal series. |
Type Of Material | Database/Collection of data |
Year Produced | 2019 |
Provided To Others? | Yes |
Impact | Two requests for the TEM dataset. The datasets are also used extensively in my research. At the moment, impact is limited as our first papers since setting up our publicly accessible dataserver are under review, or still in preparation. |
URL | https://warwick.ac.uk/fac/sci/physics/research/condensedmatt/microscopy/research/machinelearning/ |
Title | GitHub Repositories |
Description | Source code. Some of which has been published, will be published (currently preprints), or is for miscellaneous projects. Providing source code can massively reduce the time needed to reproduce work. Even if work is reproduced or extended on in another language, the original source code can be a valuable reference. Source code also improves clararity. For example. deep learning repositories contain source code for neural networks, and therefore exact graphs. |
Type Of Material | Computer model/algorithm |
Year Produced | 2018 |
Provided To Others? | Yes |
Impact | Research is reproducible, clearer, and easier to build upon. |
URL | https://github.com/Jeffrey-Ede?tab=repositories |
Description | Neutrino Vertex Classification |
Organisation | University of Warwick |
Department | Department of Physics |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | Collaboration with a particle physicist to a develop neural networks to classify neural vertex positions in the new DUNE detector. I proposed the project, taught my collaborator how to train neural networks, directed him to resources, and proposed machine learning models. |
Collaborator Contribution | Jhanzeb Ahmed aquired training data, and trained neural networks. He was also investigating support vector machines. |
Impact | The main outcome is the machine learning skills transferred to my collaborator. He's now transferred to another project. He's hoping to finish experiments afterwards. |
Start Year | 2019 |
Description | One-Shot Exit Wavefunction Reconstruction |
Organisation | University of Warwick |
Department | Department of Physics |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | Half of wavefunction information is lost at measurement. I proposed that deep learning can be used to recover lost phase information, and formed a collaboration to investigate. As lead investigator, I'm responsible for the idea, organization and delegation of roles, neural network training and validation, multislice wavefunction simulations, repositing code, repositing new datasets, lead authorship, manuscript submission, and being corresponding author. |
Collaborator Contribution | Jonathan P. P. Peters maintains clTEM multislice simulation code. He helped me get clTEM set up, and jointly wrote a script to automate exit wavefunction simulations. Jeremy Sloan internally reviewed an early draft of our paper. He then used my data to create images of crystal structures to add a figure and improve a figure. Richard Beanland provided quality control for experimental focal series that wavefunctions can be reconstructed from. Jonathan P. P. Peters, Jeremy Sloan and Richard Beanland provided feedback to improve our manuscript. |
Impact | GitHub repository with source code, links to pre-trained models, scripts to process data, perform analysis and create figures for paper, and compiled clTEM code for multislice simualtions used in our research: https://github.com/Jeffrey-Ede/One-Shot New wavefunction datasets: https://warwick.ac.uk/fac/sci/physics/research/condensedmatt/microscopy/research/machinelearning/ Manuscript submitted to Ultramicroscopy. This collaboration is multidisciplinary. Disciplines involved include electron microscopy, machine learning, and high-performance computing. High-performance computing is listed separately to machine learning as calculations involved OpenCL/C++ for GPU-accelerated multislice simulations used to generate training data. |
Start Year | 2019 |
Title | GitHub Repositories |
Description | GitHub repositories for new or improved techniques in electron microscopy/machine learning. Noteably, ALRC: Algorithm for adaptive learning rate clipping to stabilize deep learning. One-Shot: Pretrained models and source code for exit wavefunction reconstruction with deep learning. Electron-Micrograph-Denoiser: Pretrained models and source code for improving electron micrograph signal-to-noise with deep learning. partial-STEM: Pretrained models and source code for completing spirals and other partial scanning transmission electron micrographs with deep learning. DLSS-STEM: Pretrained models and source code for supersampling electron micrographs deep learning. |
Type Of Technology | New/Improved Technique/Technology |
Year Produced | 2019 |
Impact | Tools are available for use by others, and to be built upon. Source code also enhances reproducibility, and makes research clearer. |
URL | https://github.com/Jeffrey-Ede?tab=repositories |
Description | arXiv Preprints |
Form Of Engagement Activity | A magazine, newsletter or online publication |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Making preprints for research papers publicly available is an important step to lower barriers to science. It makes science directly available to the general public, academics, and industrial professional without potentially expensive subscriptions to journals. Importantly, it reduces dependence on secondary sources that summarize published work. The main outputs are two requests for new 100+ GB training datasets introduced in papers, which I transferred via a Google datacenter. We've now set up our own publicly accessible dataserver to handle requests. Another benefit is that preprint servers reduce delay before science is available. After the partial STEM preprint was published, I was contacted by a group doing the same research in America and we discussed methodology. |
Year(s) Of Engagement Activity | 2018,2019 |
URL | https://arxiv.org/search/?query=jeffrey+ede&searchtype=all&source=header |