Developing diagnostic methods for clinical genetics - phenotyping from faces in photos.

Lead Research Organisation: University of Oxford
Department Name: Physiology Anatomy and Genetics

Abstract

I am leading research that will help clinicians to diagnose rare diseases using automated computer analysis of photos.

Rare diseases are numerous - so much so that as a group they are very common. One person in 17 has a rare genetic disorder, but most fail to receive a genetic diagnosis. Diagnosing the genetic cause of a disorder, even when there are only a handful of cases in the world, represents the first of many steps in finding effective treatments. Even though new genetic tests promise to assist in diagnosing some of these patients, the tests are expensive and currently only available to a few people in the wealthiest countries.

For the past 65 years expert clinical doctors have been matching a diagnosis to patients based on facial features and follow up clinical tests. We are developing algorithms through which a computer will learn and apply these skills objectively.

Identifying patients with the same genetic disorders allows comparisons to be made between them. In turn, this can improve estimates of how the disease might progress and allow direct therapeutic benefits, for instance by showing which symptoms are caused by the genetic disorder and which symptoms might be caused by other clinical issues that can be treated.

Using the latest research in computer vision and machine learning the algorithm automatically analyses patient photographs and finds their place in "Clinical Face Phenotype Space" (CFPS). Patients that share a specific dysmorphic disease or syndrome, will cluster together in CFPS. The CFPS model is created and shaped using ordinary, family album photos and accounts for variations between images that are not disease relevant (such as lighting, image quality, background, pose, age, gender, ethnicity, and facial expression).

In the present application I seek funding to develop methods for clinical geneticists to query CFPS for clinically relevant information. The work will develop means by which a patient's similarity to other patient groups can be visualised, explored and tested through robust statistical modelling.

Furthermore it will make it possible to overlay a patient's DNA with CFPS to identify disease causing mutations. This will improve our understanding of how rare diseases disrupt the normal functioning of the body, and in turn influence decisions in treatment strategies.

A clinician should, in future, be able to take a smartphone picture of a patient and query CFPS to quickly find out which genetic disease the person might have. For diseases unknown to medical science, CFPS will find if there are any other patients around the world that might have the same disease.

CFPS will learn from our faces to help diagnose rare diseases.

Technical Summary

Aim 1. Enable complete phenotype descriptions from photos for clinicians. I will adapt existing computer vision and machine learning methods for the specific purpose of obtaining phenotype descriptions. CFPS already performs exceptionally well given the limited information that it exploits thus far. Developing the accuracy of craniofacial point annotations and feature descriptors for capturing phenotype characteristics, inferring 3D orientation from 2D images, and applying machine learning approaches will ensure that CFPS reaches its full potential.

Aim 2. Robustly predict causative mutations from DNA variants and CFPS. I will lead research that will aid genetic diagnoses with CFPS. The collective body of work to bring CFPS and DNA variant data together is essential for CFPS to become ready for clinical genetic data. To do this I will create family representations to differentiate between de novo and inherited phenotype. With a vector representation of a face during development CFPS will be able to capture and predict disease progression. I will also put in place a robust statistical framework to infer causative DNA variants in rare de novo mutation diseases to empower the search for ultra-rare syndromes.

Aim 3. Develop the big data framework for implementing CFPS in global healthcare. Current ethical, law and data security concerns are a major hurdle in the path of bringing CFPS to patients. Within the scope of this Fellowship I will address each of these challenges. Patient images through collaborations and patient contributions will involve applying for REC approval to ask for consent and for people to contribute photos from their family albums. This will be both an exercise in public engagement but also en masse data collection. I will be working with ethics, law and eHealth data management questions to enable CFPS to develop into a form which will be not only powerful, but also usable in clinical settings.

Planned Impact

This project is highly collaborative both within the UK and internationally. This is also translational and interdisciplinary science, criss-crossing the fields of computer vision, computational biology, and clinical genetics. Consequently it provides several direct pathways to impact. The most important of these is that my work will aid in the clinical diagnosis worldwide of the genetics of rare disorders. I have established and will be expanding an international network of collaborations to apply and develop our approach using patient data. I expect this project in 4 years' time to have progressed to being in the early stages of implementation in clinical genetics counselling, initially in the UK and Europe and then elsewhere.

In the space of three years the following directions will lead to measurable impacts:

The methods to interrogate CFPS together with genotype will add statistical power to the inference of genetic diagnoses. In collaboration with clinical sequencing efforts our approach will be applied to patient image data to add value to these existing datasets and on-going projects. This will result in research participants receiving diagnoses that otherwise they would not.

I will set up partnerships with patient support groups for syndromes with identified genetic origins. This will be a means to present the science and the project to directly engage the public, and offer the opportunity to contribute photos (under a REC approved informed consent model) for the further refinement of the CFPS. This is public engagement of the best and most interactive kind, which can be measured by the number of images of individuals (>10,000) and syndromes (>200) that I aim to collect over the coming 4 years.

Collaborations will be expanded to additional clinical geneticists across the world. For clinicians the practical utility would be to evaluate the approach as a means of narrowing down hypotheses for the clinical diagnosis of known syndromes. As the work to interrogate DNA variant data together with CFPS develops we will be able to narrow the search for causative variants in patient data even where no established diagnosis exists. Results will also directly feed back to the project, as they will expand the number of images of clinically defined syndromes that will iteratively improve the CFPS - the more people that use it the better it gets.

Extending contacts within the MRC Units in The Gambia and Uganda would be a beneficial means of geographically widening our research to efficiently test and apply CFPS in the future. At the end of the 4 year Fellowship CFPS will be ready for testing and accreditation for use in the NHS and be ready to begin testing in clinics in more countries around the world. With care during the methodological development of the CFPS this should be achievable on a large scale rapidly.

Publications

10 25 50

publication icon
Bengani H (2017) Clinical and molecular consequences of disease-associated de novo mutations in SATB2. in Genetics in medicine : official journal of the American College of Medical Genetics

publication icon
Dawson Mitchell (2018) From Same Photo: Cheating on Visual Kinship Challenges in arXiv e-prints

publication icon
Deciphering Developmental Disorders Study (2017) Prevalence and architecture of de novo mutations in developmental disorders. in Nature

publication icon
Ferlaino Michael (2018) Towards Deep Cellular Phenotyping in Placental Histology in arXiv e-prints

publication icon
Hallowell N (2019) Big data phenotyping in rare diseases: some ethical issues. in Genetics in medicine : official journal of the American College of Medical Genetics

publication icon
Van Der Donk R (2018) Next-generation phenotyping using computer vision algorithms in rare genomic neurodevelopmental disorders. in Genetics in medicine : official journal of the American College of Medical Genetics

 
Description Invited presented to APPG - 'How does data-sharing support global health and policy'?
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description MRC Methodology Research Grant
Amount £404,250 (GBP)
Funding ID MR/M01326X/1 
Organisation Medical Research Council (MRC) 
Sector Academic/University
Country United Kingdom
Start 04/2016 
End 03/2019
 
Description Wellcome Trust Small Grant in Humanities and Social Science
Amount £27,408 (GBP)
Funding ID 208818/Z/17/Z 
Organisation Wellcome Trust 
Sector Charity/Non Profit
Country United Kingdom
Start 09/2017 
End 02/2020
 
Title LAOFITW 
Description Online public data set of faces from different ancestral origins for the purpose of training deep neural networks to be unbiased and fair. 
Type Of Material Database/Collection of data 
Year Produced 2018 
Provided To Others? Yes  
Impact Publication of a method to create deep neural network embeddings that are fair and able to disentangle desired phenotypic traits from spurious biases inherent in datasets. 
URL http://www.robots.ox.ac.uk/~vgg/data/laofiw/
 
Title Minerva&Me public participation in research website 
Description Minerva and Me is a secure public participation in research website through which people can choose to share some of their family photographs to help drive research forward. This is currently restricted within the domain of computational phenotyping from facial images to help the diagnosis of rare diseases but has been built to be modular and expandable. The study is fully research ethics approved and relies fundamentally on a dynamic consent model for participant information and continuing data control. Minerva&Me is intended to allow controlled access for other researchers, but this will always be under the dynamic and partcipant centric data control model (so to a degree the data sharing is not ours to determine). Any such additional projects or efforts will need to add extra data control options to the website and will be overseen by the Minerva&Me Advisory Board (which has legal, data security, clinical and patient representatives). 
Type Of Material Database/Collection of data 
Year Produced 2017 
Provided To Others? No  
Impact This has been build with broad engagement of patient representative groups as a framework for public participation in research with personally identifiable data such as facial photographs. 
URL https://www.minervaandme.com
 
Title Toward GitHub 
Description Placental histology cellular phenoytyping model 
Type Of Material Computer model/algorithm 
Year Produced 2018 
Provided To Others? Yes  
Impact Was published in Medical Imaging Deep Learning conference Amsterdam. Models and project are leading to followup projects and funding applciations. Models being used by other groups and organisations around the world. 
URL https://github.com/Nellaker-group/TowardsDeepPhenotyping
 
Description Minerva Consortium 
Organisation Baylor College of Medicine
Country United States 
Sector Hospitals 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Minerva Consortium 
Organisation Charité - University of Medicine Berlin
Country Germany 
Sector Academic/University 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Minerva Consortium 
Organisation Charles University
Department School of Medicine Charles Prague
Country Czech Republic 
Sector Academic/University 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Minerva Consortium 
Organisation Freeman Hospital
Country United Kingdom 
Sector Hospitals 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Minerva Consortium 
Organisation Great Ormond Street Hospital (GOSH)
Country United Kingdom 
Sector Hospitals 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Minerva Consortium 
Organisation Hospital for Sick Children, Toronto
Country Canada 
Sector Hospitals 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Minerva Consortium 
Organisation Imagine Institute
Country France 
Sector Hospitals 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Minerva Consortium 
Organisation King Edward Memorial Hospital
Country Australia 
Sector Hospitals 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Minerva Consortium 
Organisation Manchester University NHS Foundation Trust
Country United Kingdom 
Sector Public 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Minerva Consortium 
Organisation Oregon Health and Science University
Country United States 
Sector Academic/University 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Minerva Consortium 
Organisation Radboud University Nijmegen Medical Center
Country Netherlands 
Sector Academic/University 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Minerva Consortium 
Organisation The Garvan Institute for Medical Research
Country Australia 
Sector Charity/Non Profit 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Minerva Consortium 
Organisation The Jackson Laboratory
Country United States 
Sector Charity/Non Profit 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Minerva Consortium 
Organisation The Wellcome Trust Sanger Institute
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Minerva Consortium 
Organisation University of Cambridge
Department Cambridge Institute for Medical Research (CIMR)
Country United Kingdom 
Sector Academic/University 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Minerva Consortium 
Organisation University of Cape Town
Country South Africa 
Sector Academic/University 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Minerva Consortium 
Organisation University of Edinburgh
Country United Kingdom 
Sector Academic/University 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Minerva Consortium 
Organisation University of Melbourne
Department Murdoch Children's Research Centre
Country Australia 
Sector Academic/University 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Minerva Consortium 
Organisation University of Oxford
Department Weatherall Institute of Molecular Medicine (WIMM)
Country United Kingdom 
Sector Public 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Minerva Consortium 
Organisation University of Washington
Country United States 
Sector Academic/University 
PI Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Collaborator Contribution We launched the Minerva Consortium at the start of 2017 to enable the development of computational phenotyping from clinical imaging, specifically to aid the diagnosis of rare and developmental diseases. The MC structure is intended to coordinate research efforts and patient data sets for projects in an open academic collaboration.
Impact Newly launched, no outputs yet.
Start Year 2017
 
Description Comment on FDNA paper 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Interviewed by Nature News and New Scientist with regards to a publication by one of the major groups in the same field as my research.
Year(s) Of Engagement Activity 2019
URL https://www.nature.com/articles/d41586-019-00027-x
 
Description Media engagement around eLIFE publication 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact More than 5 new potential collaborations around the world were initiated as a direct result of media covering the publication of our article in eLIFE.
A number of parents of patients contacted us directly for more information.


Three keynote invitations to present at scientific conferences were offered after the media attention.
New collaborations with clinicians and scientists around the world have been started.
Year(s) Of Engagement Activity 2014,2015,2016
URL https://www.obs-gyn.ox.ac.uk/team/christoffer-nellaker
 
Description Pint of Science 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Public/other audiences
Results and Impact This was a talk for the Pint of Science initiative, presenting the current research to a lay audience. Plenty of questions asked and a lovely evening of meeting interested members of the public
Year(s) Of Engagement Activity 2015,2017
URL https://pintofscience.co.uk/