Quantitative and Real-Time Image Analysis for Advanced Light Microscopy.
Lead Research Organisation:
University of Oxford
Department Name: Weatherall Inst of Molecular Medicine
Abstract
In microscopy based research it is common to acquire and analyze microscopy images of cells sampled from populations which have undergone differential treatments. A non-exhaustive number of samples (i.e. images of cells) are often acquired due to experimental limitations that include: restricted acquisition periods, provision of expensive equipment or pressures to move onto the next experiment. Random sampling is a methodology that is used to sample representatively. There are some good reasons however for not sampling randomly. For example, cells maybe expressing a reporter protein and the copy number (quantity) of this molecule maybe too high or too low for subsequent analysis and so preference is shown toward those examples where the expression levels are within a certain range. Or, alternatively, the cells in an area maybe compromised by an artifact, be unhealthy, or too dense or sparse for a particular analysis. There are many factors that will influence the choices of an imaging researcher that are fully justifiable, but it is a problem as to how this information is documented and shared with other scientists. Removing the scientist from the pipeline of acquisition and analysis is counter-productive, a better solution is to provide tools that allow them to document and describe any experimental subjectivity as well as boosting reproducibility through automation. The objectives of this first part of the project (1-4) revolve around solving these issues of documenting, describing and automating image acquisition so that experimental design can be better communicated between users and laboratories.
Furthermore, we would like to develop tools and approaches, which allow better visualization and real-time feedback for advanced approaches so as to better inform imaging scientists as they perform their research and make experimental decisions. The second part of the project relates to this goal, objectives 5-6.
Furthermore, we would like to develop tools and approaches, which allow better visualization and real-time feedback for advanced approaches so as to better inform imaging scientists as they perform their research and make experimental decisions. The second part of the project relates to this goal, objectives 5-6.
Technical Summary
In terms of microscopy analysis, great inroads have been made in terms of impartial and systematic analysis but little has been done to ensure that cells under the microscope are selected impartially. For this project DW will develop algorithms that can statistically quantify and describe cellular appearances utilizing the latest machine learning, computer vision (CV) and signal processing techniques and technologies.
DW has in ongoing work investigated the use of CV algorithms in microscopy and has shown very promising results can be achieved by utilizing object detection convolutional neural networks for cellular detection. DW will extend on his use of neural networks for localizing cells (objectives 1-2) and will also develop methods to statistically describe cellular appearance using networks derived from auto-encoders (objective 3), a type of compression network. Neural networks are implemented in several ways, a popular method is to use Tensorflow and DW is an expert in this language. To effectively train neural networks, powerful GPUs are required. Fortunately the WIMM has various computational facilities and DW has access to two powerful GPU equipped servers.
For developing CV and real-time analysis approaches to work with camera and detector hardware, DW will work toward developing algorithms that can be embedded in miniaturized electronics (objective 4). The Nvidia Jetson TX2 Developer kit is a resource which allows you to create and develop neural networks and distribute them onto small hardware boards. DW will develop and test this hardware technique with microscopy based hardware and algorithms.
For objectives 5 and 6 GPU code will be systematically produced in CUDA and will be made as compatible and distributable as possible. Oxford University has several microscopy software development projects (e.g. in Micron) and it is intended that the software and libraries produced by this project will be made compatible with these other projects.
DW has in ongoing work investigated the use of CV algorithms in microscopy and has shown very promising results can be achieved by utilizing object detection convolutional neural networks for cellular detection. DW will extend on his use of neural networks for localizing cells (objectives 1-2) and will also develop methods to statistically describe cellular appearance using networks derived from auto-encoders (objective 3), a type of compression network. Neural networks are implemented in several ways, a popular method is to use Tensorflow and DW is an expert in this language. To effectively train neural networks, powerful GPUs are required. Fortunately the WIMM has various computational facilities and DW has access to two powerful GPU equipped servers.
For developing CV and real-time analysis approaches to work with camera and detector hardware, DW will work toward developing algorithms that can be embedded in miniaturized electronics (objective 4). The Nvidia Jetson TX2 Developer kit is a resource which allows you to create and develop neural networks and distribute them onto small hardware boards. DW will develop and test this hardware technique with microscopy based hardware and algorithms.
For objectives 5 and 6 GPU code will be systematically produced in CUDA and will be made as compatible and distributable as possible. Oxford University has several microscopy software development projects (e.g. in Micron) and it is intended that the software and libraries produced by this project will be made compatible with these other projects.
People |
ORCID iD |
| Dominic Waithe (Principal Investigator / Fellow) |
Publications
Brown JM
(2018)
A tissue-specific self-interacting chromatin domain forms independently of enhancer-promoter interactions.
in Nature communications
Büttner M
(2021)
Challenges of Using Expansion Microscopy for Super-resolved Imaging of Cellular Organelles.
in Chembiochem : a European journal of chemical biology
Carravilla P
(2019)
Molecular recognition of the native HIV-1 MPER revealed by STED microscopy of single virions.
in Nature communications
Chagraoui H
(2018)
SCL/TAL1 cooperates with Polycomb RYBP-PRC1 to suppress alternative lineages in blood-fated cells.
in Nature communications
Chen YL
(2019)
Proof-of-concept clinical trial of etokimab shows a key role for IL-33 in atopic dermatitis pathogenesis.
in Science translational medicine
Hailstone M
(2020)
CytoCensus, mapping cell identity and division in tissues and organs using machine learning
in eLife
Hockman D
(2018)
Striking parallels between carotid body glomus cell and adrenal chromaffin cell development.
in Developmental biology
Juban G
(2021)
Oncogenic Gata1 causes stage-specific megakaryocyte differentiation delay.
in Haematologica
Rodríguez Cruz PM
(2020)
Congenital myasthenic syndrome due to mutations in MUSK suggests that the level of MuSK phosphorylation is crucial for governing synaptic structure.
in Human mutation
Salio M
(2020)
Ligand-dependent downregulation of MR1 cell surface expression.
in Proceedings of the National Academy of Sciences of the United States of America
Schneider F
(2018)
Statistical Analysis of Scanning Fluorescence Correlation Spectroscopy Data Differentiates Free from Hindered Diffusion.
in ACS nano
Schneider F
(2018)
Nanoscale Spatiotemporal Diffusion Modes Measured by Simultaneous Confocal and Stimulated Emission Depletion Nanoscopy Imaging.
in Nano letters
Sezgin E
(2019)
Measuring nanoscale diffusion dynamics in cellular membranes with super-resolution STED-FCS.
in Nature protocols
Spiess M
(2018)
Active and inactive ß1 integrins segregate into distinct nanoclusters in focal adhesions.
in The Journal of cell biology
Waithe D
(2018)
Optimized processing and analysis of conventional confocal microscopy generated scanning FCS data.
in Methods (San Diego, Calif.)
Waithe D
(2021)
Open-source browser-based software simplifies fluorescence correlation spectroscopy data analysis
in Nature Photonics
Waithe D
(2020)
Object detection networks and augmented reality for cellular detection in fluorescence microscopy.
in The Journal of cell biology
Waithe D
(2021)
Summary of two questionnaires designed to understand the research climate for Bioimage Analysts in the UK between 2016-2019.
in F1000Research
| Description | Chair of the Image Analysis Focused Interest Group of the Royal Microscopy Society |
| Geographic Reach | National |
| Policy Influence Type | Membership of a guideline committee |
| Impact | The bioimage analysis community is not well represented within the research community in the UK and beyond. The goal of this focused interest group, which I was asked to chair by the Royal Microscopy society, is designed to highlight and draw together the community of bioimage analysts in the UK. So far we have developed a number of questionnaires, staged events and contributed talks at a number of scientific conferences to raise awareness for this group. |
| URL | http://iafig-rms.org/ |
| Description | Organised and taught on a Python Bioimage Analysis course |
| Geographic Reach | Europe |
| Policy Influence Type | Influenced training of practitioners or researchers |
| Impact | We trained 46 bioimage scientists to become fully trained Bioimage Analysts. They developed Python image analysis skills and will apply these research skills across the UK and continental Europe. |
| URL | https://github.com/IAFIG-RMS/Python-for-Bioimage-Analysis |
| Description | Training of Bioimage Analysis Instructors. |
| Geographic Reach | National |
| Policy Influence Type | Influenced training of practitioners or researchers |
| Impact | We trained a number of bioimage analyst (16) to be trainers. This has lead to subsequent enhanced and additional training. |
| URL | https://analyticalscience.wiley.com/do/10.1002/micro.2914/full/ |
| Description | UK management committee member for NEUBIAS |
| Geographic Reach | Europe |
| Policy Influence Type | Membership of a guideline committee |
| Impact | Neubias is an COST funded initiative designed to raise awareness for and to support image analysis on the EU and worldwide level. I along with Graeme Ball represent the UK in this scheme. I attend meetings representing the UK and contribute to workgroups which influence how this group contributes to things like ontologies, training and databases for software, as well as best practise for the community. e.g. https://twitter.com/matuskalas/status/1093479463155912711 |
| URL | http://eubias.org/NEUBIAS/venue/who-are-we/uk-members/ |
| Title | Automated Microscope Control Algorithm software repository. |
| Description | This is a software framework which allows a microscope to be autonomously controlled using the feedback from a machine learning/computer vision algorithm. https://github.com/dwaithe/amca It relates to the paper published as a pre-print here: https://www.biorxiv.org/content/10.1101/544833v1 |
| Type Of Material | Technology assay or reagent |
| Year Produced | 2019 |
| Provided To Others? | Yes |
| Impact | There is a lot of interest from the research community. The paper pre-print has already an altmetric of 42 and has been viewed around the world: https://biorxiv.altmetric.com/details/55202241 |
| URL | https://github.com/dwaithe/amca |
| Title | Open-source browser-based software simplifies fluorescence correlation spectroscopy data analysis |
| Description | FoCuS-fit-JS is a JavaScript browser based software for fitting correlated FCS (Fluorescence Correlation Spectroscopy) curves. |
| Type Of Material | Improvements to research infrastructure |
| Year Produced | 2021 |
| Provided To Others? | Yes |
| Impact | A nature photonics paper: https://rdcu.be/cIK5a I have also created an instantly accessible resource for anyone in the world to use. The site is also hosted by Github, so it is free to access for anyone. |
| URL | https://github.com/dwaithe/FCSfitJS |
| Title | Fluorescence Microscopy Data for Cellular Detection using Object Detection Networks. |
| Description | As a by-product of my research we have developed a public collection of bioimage datasets with annotations. This can be freely downloaded and used by the community as a whole. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2019 |
| Provided To Others? | No |
| Impact | No notable impact yet, but publication citations should follow. |
| URL | https://zenodo.org/record/2548493#.XIZMUxOwnJw |
| Title | Fluorescence Microscopy Data for Cellular Detection using Object Detection Networks. |
| Description | This data accompanies work from the paper entitled:
Object Detection Networks and Augmented Reality for Cellular Detection in Fluorescence Microscopy Acquisition and Analysis. Waithe D1*,2,, Brown JM3, Reglinski K4,6,7, Diez-Sevilla I
5, Roberts D
5, Christian Eggeling1,4,6,8 1 Wolfson Imaging Centre Oxford and 2 MRC WIMM Centre for Computational Biology and 3 MRC Molecular Haematology Unit and 4 MRC Human Immunology Unit, Weatherall Institute of Molecular Medicine, University of Oxford, OX3 9DS, Oxford, United Kingdom. 5 Nuffield Division of Clinical Laboratory Sciences, Radcliffe Department of Medicine, John Radcliffe Hospital, University of Oxford, Headley Way, Oxford, OX3 9DU.
6 Institute of Applied Optics and Biophysics, Friedrich-Schiller-University Jena, Max-Wien Platz 4, 07743 Jena, Germany. 7 University Hospital Jena (UKJ), Bachstraße 18, 07743 Jena, Germany. 8 Leibniz Institute of Photonic Technology e.V., Albert-Einstein-Straße 9, 07745 Jena, Germany. Further details of these datasets can be found in the methods section of the above paper. Erythroblast DAPI (+glycophorin A): erythroblast cells were stained with DAPI and for glycophorin A protein (CD235a antibody, JC159 clone, Dako) and with Alexa Fluor 488 secondary antibody (Invitrogen). DAPI staining was performed through using VectaShield Hard Set mounting solution with DAPI (Vector Lab). Num. of images used for training: 80 and testing: 80. Average number of cells per image: 4.5. Neuroblastoma phalloidin (+DAPI): images of neuroblastoma cells (N1E115) stained with phalloidin and DAPI were acquired from the Cell Image Library [26]. Cell images in the original dataset were acquired with a larger field of view than our system and so we divided each image into four sub-images and also created ROI bounding boxes for each of the cells in the image. The images were stained for FITC-phalloidin and DAPI. Num. of images used for training: 180, testing: 180. Average number of cells per image: 11.7. Fibroblast nucleopore: fibroblast (GM5756T) cells were stained for a nucleopore protein (anti-Nup153 mouse antibody, Abcam) and detected with anti-mouse Alexa Fluor 488. Num. of images for training: 26 and testing: 20. Average number of cells per image: 4.8. Eukaryote DAPI: eukaryote cells were stained with DAPI and fixed and mounted in Vectashield (Vector Lab). Num. of images for training: 40 and testing: 40. Average number of cells per image: 8.9. C127 DAPI: C127 cells were initially treated with a technique called RASER-FISH[27], stained with DAPI and fixed and mounted in Vectashield (Vector Lab). Num. of images for training: 30 and testing: 30. Average number of cells per image: 7.1. HEK peroxisome All: HEK-293 cells expressing peroxisome-localized GFP-SCP2 protein. Cells were transfected with GFP-SCP2 protein, which contains the PTS-1 localization signal, which redirects the fluorescently tagged protein to the actively importing peroxisomes[28]. Cells were fixed and mounted. Num. of images for training: 55 and testing: 55. Additionally we sub-categorised the cells as 'punctuate' and 'non-punctuate', where 'punctuate' would represent cells that have staining where the peroxisomes are discretely visible and 'non-punctuate' would be diffuse staining within the cell. The 'HEK peroxisome All' dataset contains ROI for all the cells: average number of cells per image: 7.9. The 'HEK peroxisome' dataset contains only those cells with punctuate fluorescence: average number of punctuate cells per image: 3.9. Erythroid DAPI All: Murine embryoid body-derived erythroid cells, differentiated from mES cells. Stained with DAPI and fixed and mounted in Vectashield (Vector Lab). Num. of images for training: 51 and testing: 50. Multinucleate cells are seen with this differentiation procedure. There is a variation in size of the nuclei (nuclei become smaller as differentiation proceeds). The smaller, 'late erythroid' nuclei contain heavily condensed DNA and often have heavy 'blobs' of heterochromatin visible. Apoptopic cells are also present, with apoptotic bodies clearly present. The 'Erythroid DAPI All' dataset contains ROI for all the cells in the image. Average number of cells per image: 21.5. The subset 'Erythroid DAPI' contains non-apoptotic cells only: average number of cells per image: 11.9 COS-7 nucleopore. Slides were acquired from GATTAquant. GATTA-Cells 1C are single color COS-7 cells stained for Nuclear pore complexes (Anti-Nup) and with Alexa Fluor 555 Fab(ab')2 secondary stain. GATTA-Cells are embedded in ProLong Diamond. Num. of images for training: 50 and testing: 50. Average number of cells per image: 13.2 COS-7 nucleopore 40x. Same GATTA-Cells 1C slides (GATTAquant) as above but imaged on Nikon microscope, with 40x NA 0.6 objective. Num. of images for testing: 11. Average number of cells per image: 31.6. COS-7 nucleopore 10x. Same GATTA-Cells 1C slides (GATTAquant) as above but imaged on Nikon microscope, with 10x NA 0.25 objective. Num. of images for testing: 20. Average number of cells per image: 24.6 Dataset Annotation Datasets were annotated by a skilled user. These annotations represent the ground-truth of each image with bounding boxes (regions) drawn around each cell present within the staining. Annotations were produced using Fiji/ImageJ [29] ROI Manager and also through using the OMERO [30] ROI drawing interface (https://www.openmicroscopy.org/omero/). The dataset labels were then converted into a format compatible with Faster-RCNN (Pascal), YOLOv2, YOLOv3 and also RetinaNet. The scripts used to perform this conversion are documented in the repository (https://github.com/dwaithe/amca/scripts/). |
| Type Of Material | Database/Collection of data |
| Year Produced | 2020 |
| Provided To Others? | Yes |
| URL | https://zenodo.org/record/2548492 |
| Title | Fluorescence Microscopy Data for Cellular Detection using Object Detection Networks. |
| Description | This data accompanies work from the paper entitled:
Object Detection Networks and Augmented Reality for Cellular Detection in Fluorescence Microscopy Acquisition and Analysis. Waithe D1*,2,, Brown JM3, Reglinski K4,6,7, Diez-Sevilla I
5, Roberts D
5, Christian Eggeling1,4,6,8 1 Wolfson Imaging Centre Oxford and 2 MRC WIMM Centre for Computational Biology and 3 MRC Molecular Haematology Unit and 4 MRC Human Immunology Unit, Weatherall Institute of Molecular Medicine, University of Oxford, OX3 9DS, Oxford, United Kingdom. 5 Nuffield Division of Clinical Laboratory Sciences, Radcliffe Department of Medicine, John Radcliffe Hospital, University of Oxford, Headley Way, Oxford, OX3 9DU.
6 Institute of Applied Optics and Biophysics, Friedrich-Schiller-University Jena, Max-Wien Platz 4, 07743 Jena, Germany. 7 University Hospital Jena (UKJ), Bachstraße 18, 07743 Jena, Germany. 8 Leibniz Institute of Photonic Technology e.V., Albert-Einstein-Straße 9, 07745 Jena, Germany. Further details of these datasets can be found in the methods section of the above paper. Erythroblast DAPI (+glycophorin A): erythroblast cells were stained with DAPI and for glycophorin A protein (CD235a antibody, JC159 clone, Dako) and with Alexa Fluor 488 secondary antibody (Invitrogen). DAPI staining was performed through using VectaShield Hard Set mounting solution with DAPI (Vector Lab). Num. of images used for training: 80 and testing: 80. Average number of cells per image: 4.5. Neuroblastoma phalloidin (+DAPI): images of neuroblastoma cells (N1E115) stained with phalloidin and DAPI were acquired from the Cell Image Library [26]. Cell images in the original dataset were acquired with a larger field of view than our system and so we divided each image into four sub-images and also created ROI bounding boxes for each of the cells in the image. The images were stained for FITC-phalloidin and DAPI. Num. of images used for training: 180, testing: 180. Average number of cells per image: 11.7. Fibroblast nucleopore: fibroblast (GM5756T) cells were stained for a nucleopore protein (anti-Nup153 mouse antibody, Abcam) and detected with anti-mouse Alexa Fluor 488. Num. of images for training: 26 and testing: 20. Average number of cells per image: 4.8. Eukaryote DAPI: eukaryote cells were stained with DAPI and fixed and mounted in Vectashield (Vector Lab). Num. of images for training: 40 and testing: 40. Average number of cells per image: 8.9. C127 DAPI: C127 cells were initially treated with a technique called RASER-FISH[27], stained with DAPI and fixed and mounted in Vectashield (Vector Lab). Num. of images for training: 30 and testing: 30. Average number of cells per image: 7.1. HEK peroxisome All: HEK-293 cells expressing peroxisome-localized GFP-SCP2 protein. Cells were transfected with GFP-SCP2 protein, which contains the PTS-1 localization signal, which redirects the fluorescently tagged protein to the actively importing peroxisomes[28]. Cells were fixed and mounted. Num. of images for training: 55 and testing: 55. Additionally we sub-categorised the cells as 'punctuate' and 'non-punctuate', where 'punctuate' would represent cells that have staining where the peroxisomes are discretely visible and 'non-punctuate' would be diffuse staining within the cell. The 'HEK peroxisome All' dataset contains ROI for all the cells: average number of cells per image: 7.9. The 'HEK peroxisome' dataset contains only those cells with punctuate fluorescence: average number of punctuate cells per image: 3.9. Erythroid DAPI All: Murine embryoid body-derived erythroid cells, differentiated from mES cells. Stained with DAPI and fixed and mounted in Vectashield (Vector Lab). Num. of images for training: 51 and testing: 50. Multinucleate cells are seen with this differentiation procedure. There is a variation in size of the nuclei (nuclei become smaller as differentiation proceeds). The smaller, 'late erythroid' nuclei contain heavily condensed DNA and often have heavy 'blobs' of heterochromatin visible. Apoptopic cells are also present, with apoptotic bodies clearly present. The 'Erythroid DAPI All' dataset contains ROI for all the cells in the image. Average number of cells per image: 21.5. The subset 'Erythroid DAPI' contains non-apoptotic cells only: average number of cells per image: 11.9 COS-7 nucleopore. Slides were acquired from GATTAquant. GATTA-Cells 1C are single color COS-7 cells stained for Nuclear pore complexes (Anti-Nup) and with Alexa Fluor 555 Fab(ab')2 secondary stain. GATTA-Cells are embedded in ProLong Diamond. Num. of images for training: 50 and testing: 50. Average number of cells per image: 13.2 COS-7 nucleopore 40x. Same GATTA-Cells 1C slides (GATTAquant) as above but imaged on Nikon microscope, with 40x NA 0.6 objective. Num. of images for testing: 11. Average number of cells per image: 31.6. COS-7 nucleopore 10x. Same GATTA-Cells 1C slides (GATTAquant) as above but imaged on Nikon microscope, with 10x NA 0.25 objective. Num. of images for testing: 20. Average number of cells per image: 24.6 Dataset Annotation Datasets were annotated by a skilled user. These annotations represent the ground-truth of each image with bounding boxes (regions) drawn around each cell present within the staining. Annotations were produced using Fiji/ImageJ [29] ROI Manager and also through using the OMERO [30] ROI drawing interface (https://www.openmicroscopy.org/omero/). The dataset labels were then converted into a format compatible with Faster-RCNN (Pascal), YOLOv2, YOLOv3 and also RetinaNet. The scripts used to perform this conversion are documented in the repository (https://github.com/dwaithe/amca/scripts/). |
| Type Of Material | Database/Collection of data |
| Year Produced | 2020 |
| Provided To Others? | Yes |
| URL | https://zenodo.org/record/3894389 |
| Description | Application of image analysis algorithms for Molecular recognition of the native HIV-1 MPER. |
| Organisation | University of the Basque Country |
| Country | Spain |
| Sector | Academic/University |
| PI Contribution | Algorithms developed by myself for the analysis of STED images were used in their publication. |
| Collaborator Contribution | They used the algorithms to great affect and we developed together based on their feedback a more optimised and expansive solution. |
| Impact | Carravilla P, Chojnacki J, Rujas E, Insausti S, Largo E, Waithe D..... Nieva JL, (2019). Molecular recognition of the native HIV-1 MPER revealed by STED microscopy of single virions.. Nature communications, 10 (1), pp. 78 |
| Start Year | 2018 |
| Description | Twitter posts relating to research |
| Form Of Engagement Activity | Engagement focused website, blog or social media channel |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | Tweeting links to papers and providing interactive resources: https://twitter.com/dwaithe/status/1094195706905194500 |
| Year(s) Of Engagement Activity | 2018,2019 |
| URL | https://twitter.com/dwaithe/status/1094195706905194500 |