Real-Time 4D Facial Sensing and Modelling
Lead Research Organisation:
University of Portsmouth
Department Name: Faculty of Creative and Cultural Ind
Abstract
Chronic Facial Palsy is caused by conditions including Bell's palsy and Stroke, afflicting, at a conservative estimate, 51,640 new people per year in the UK. Estimates based on UK incidence data for Bell's palsy patients and those left with residual weakness requiring specialist management (approximately 30%) suggests that to treat all new patients could cost between £2,543,000 (therapy alone) to £7,948,000 (therapy plus botulinum toxin injections) per annum. Approximately 30% of patients affected by facial palsy suffer ongoing chronic disfigurement, anxiety and/or depression. It has been shown that service provision for these patients is limited. Facial palsy management is expensive. With the NHS requiring to save billions annually (NHS, 2011), it is imperative to rethink conventional treatment pathways.
The NHS National Clinical Guidelines for Stroke physiotherapy recommends 45 minutes daily facial therapy for patients with facial paralysis. To meet the guidelines, each patient would need daily face-to-face therapy for a period of at least 12 weeks (broadly defined as the acute phase), costing the NHS £2,400 per patient. With 26,000 new cases annually this would represent a prohibitive cost of £62,400,000 for these new patients per annum on top of the cost of treating existing patients. However, costs could be reduced by developing a home-based rehabilitative technology, allowing greater numbers of patients to receive gold-standard treatment. The proposed technologies will provide patients with real-time feedback when undergoing therapy at home and thus can significantly reduce the time of visiting therapists for face-to-face feedback.
This project will investigate 4D (dynamic three-dimension) techniques and a computational model integrating Mirror Visual Feedback MVF theory for home-based therapy using a depth sensor. This framework envisions developing easy to use facial palsy therapy technologies, which can provide real-time feedback assessing treatment responses of patients integrating MVF theory. Research studies using MVF therapy to treat phantom limb pain and complex regional pain syndrome have produced promising results.
Facial palsy patients can recover more quickly if they exercise their facial muscles. However, it can be painful for patients to face mirrors due to the anxiety of looking at their asymmetric and deformed facial features. Therefore, we will apply MVF to adaptively mirror the healthy side of the face and facial movement over the unhealthy side and thus allow patients to observe healthy whole faces when exercising their facial muscles. It has been reported that biofeedback therapy based on MVF has been effective for Bell's facial palsy and hemi-facial pain of trigeminal neuralgia. However, these studies either use a physical mirror box or simple image-based mapping, which provide little feedback or inaccurate facial movement information. Patients have limited awareness of the abnormal movements their faces display so without feedback about these movements their facial functions may worsen, developing permanently abnormal movements.
Therefore, there is a strong need for novel computational 4D sensing and modelling methods to develop therapy which can capture and map accurate facial muscle movements according to MVF. The ultimate goal is to programme this method into a software package for home-based therapy. Currently, there are no 3D or 4D products or technologies based on MVF available for therapy. Thus, the proposed methods for building computational 4D sensing and modelling models integrating MVF will be hugely beneficial to patients and the NHS by providing appropriate feedback in assessing treatment responses of patients and ultimately improve the ability to scale home-based therapy. It can also be adapted to tele-rehabilitation for practical applications so clinical consultants and therapists can remotely observe patients' home-based therapy.
The NHS National Clinical Guidelines for Stroke physiotherapy recommends 45 minutes daily facial therapy for patients with facial paralysis. To meet the guidelines, each patient would need daily face-to-face therapy for a period of at least 12 weeks (broadly defined as the acute phase), costing the NHS £2,400 per patient. With 26,000 new cases annually this would represent a prohibitive cost of £62,400,000 for these new patients per annum on top of the cost of treating existing patients. However, costs could be reduced by developing a home-based rehabilitative technology, allowing greater numbers of patients to receive gold-standard treatment. The proposed technologies will provide patients with real-time feedback when undergoing therapy at home and thus can significantly reduce the time of visiting therapists for face-to-face feedback.
This project will investigate 4D (dynamic three-dimension) techniques and a computational model integrating Mirror Visual Feedback MVF theory for home-based therapy using a depth sensor. This framework envisions developing easy to use facial palsy therapy technologies, which can provide real-time feedback assessing treatment responses of patients integrating MVF theory. Research studies using MVF therapy to treat phantom limb pain and complex regional pain syndrome have produced promising results.
Facial palsy patients can recover more quickly if they exercise their facial muscles. However, it can be painful for patients to face mirrors due to the anxiety of looking at their asymmetric and deformed facial features. Therefore, we will apply MVF to adaptively mirror the healthy side of the face and facial movement over the unhealthy side and thus allow patients to observe healthy whole faces when exercising their facial muscles. It has been reported that biofeedback therapy based on MVF has been effective for Bell's facial palsy and hemi-facial pain of trigeminal neuralgia. However, these studies either use a physical mirror box or simple image-based mapping, which provide little feedback or inaccurate facial movement information. Patients have limited awareness of the abnormal movements their faces display so without feedback about these movements their facial functions may worsen, developing permanently abnormal movements.
Therefore, there is a strong need for novel computational 4D sensing and modelling methods to develop therapy which can capture and map accurate facial muscle movements according to MVF. The ultimate goal is to programme this method into a software package for home-based therapy. Currently, there are no 3D or 4D products or technologies based on MVF available for therapy. Thus, the proposed methods for building computational 4D sensing and modelling models integrating MVF will be hugely beneficial to patients and the NHS by providing appropriate feedback in assessing treatment responses of patients and ultimately improve the ability to scale home-based therapy. It can also be adapted to tele-rehabilitation for practical applications so clinical consultants and therapists can remotely observe patients' home-based therapy.
Planned Impact
1) Impact on clinical applications
The project can be considerably beneficial to clinical and healthcare professionals either for clinical or statistical analysis. The direct impact of this work lies in rehabilitation and clinical applications especially regarding facial palsy as it will have practical clinical applications. It provides unique advantages compared with existing conventional methods:
(a) effective therapy through implementation of Mirror Visual Feedback (MVF)
(b) home-based therapy with real-time 4D-based feedback
(c) adaptable for other tele-rehabilitation therapies.
The results of the project aim to change the current practice of home-based therapy and rehabilitation relating to face-related therapy and rehabilitation. This will be particular helpful for many patients who have difficulties in visiting therapists or hospital.
The techniques to be developed in this project are of key interest to facial palsy specialists in the NHS, healthcare industries and patients. The project aims to translate research into clinical practice assisting facial palsy rehabilitation through more effective and convenient technologies. It will accordingly improve the quality of life allowing people to remain active in both working and social contexts.
Management and treatment of facial palsy will be one of the future challenges for the National Health Service (NHS) in the UK. A compact and home-based therapy technology will provide a new opportunity for patients to obtain efficient therapy to meet the NHS National Clinical Guidelines for Stroke physiotherapy. Moreover, the proposed work opens up exciting possibilities in home-based 4D sensing and imaging for personal usage especially for healthcare, rehabilitation and therapy. The 4D methods to be developed in this research will be an essential part of tele-rehabilitation. Therefore the research will be of benefit to a large number of clinicians involved in tele-rehabilitation therapies and patients.
2) Impact on imaging areas
The lightweight 4D sensing and modelling technologies and their application still face challenges. For a sustainable growth of the technologies, it is crucial to enhance cross-disciplinary collaborations and involve more informed researchers.
This work will provide significant contributions to a variety of imaging industries. As advanced 4D technologies will play increasingly important roles in healthcare and other science and engineering sections, there is a strong need for developing methods for lightweight and easy-to-use 4D imaging and modelling systems. The 4D imaging and modelling technologies provide a new dimension to compensate patient motion, which is a vital consideration in many applications mentioned above. The proposed technologies will provide flexibility for facial animation generation and significantly relieve the burden of quality animation creation through real-time 4D sensing and modelling. The approaches could be further extended and applied to more general applications. Thus, this research will be targeted to improve the state-of-the-art 4D technologies and take these technologies to new applications.
Apart from the high research value offered by the proposal, the range of the use of 4D facial sensing and modelling technologies in different fields and businesses is considerably broad, which suggests a wide potential commercial application such as gaming, virtual reality, facial therapy and security and so on.
The project can be considerably beneficial to clinical and healthcare professionals either for clinical or statistical analysis. The direct impact of this work lies in rehabilitation and clinical applications especially regarding facial palsy as it will have practical clinical applications. It provides unique advantages compared with existing conventional methods:
(a) effective therapy through implementation of Mirror Visual Feedback (MVF)
(b) home-based therapy with real-time 4D-based feedback
(c) adaptable for other tele-rehabilitation therapies.
The results of the project aim to change the current practice of home-based therapy and rehabilitation relating to face-related therapy and rehabilitation. This will be particular helpful for many patients who have difficulties in visiting therapists or hospital.
The techniques to be developed in this project are of key interest to facial palsy specialists in the NHS, healthcare industries and patients. The project aims to translate research into clinical practice assisting facial palsy rehabilitation through more effective and convenient technologies. It will accordingly improve the quality of life allowing people to remain active in both working and social contexts.
Management and treatment of facial palsy will be one of the future challenges for the National Health Service (NHS) in the UK. A compact and home-based therapy technology will provide a new opportunity for patients to obtain efficient therapy to meet the NHS National Clinical Guidelines for Stroke physiotherapy. Moreover, the proposed work opens up exciting possibilities in home-based 4D sensing and imaging for personal usage especially for healthcare, rehabilitation and therapy. The 4D methods to be developed in this research will be an essential part of tele-rehabilitation. Therefore the research will be of benefit to a large number of clinicians involved in tele-rehabilitation therapies and patients.
2) Impact on imaging areas
The lightweight 4D sensing and modelling technologies and their application still face challenges. For a sustainable growth of the technologies, it is crucial to enhance cross-disciplinary collaborations and involve more informed researchers.
This work will provide significant contributions to a variety of imaging industries. As advanced 4D technologies will play increasingly important roles in healthcare and other science and engineering sections, there is a strong need for developing methods for lightweight and easy-to-use 4D imaging and modelling systems. The 4D imaging and modelling technologies provide a new dimension to compensate patient motion, which is a vital consideration in many applications mentioned above. The proposed technologies will provide flexibility for facial animation generation and significantly relieve the burden of quality animation creation through real-time 4D sensing and modelling. The approaches could be further extended and applied to more general applications. Thus, this research will be targeted to improve the state-of-the-art 4D technologies and take these technologies to new applications.
Apart from the high research value offered by the proposal, the range of the use of 4D facial sensing and modelling technologies in different fields and businesses is considerably broad, which suggests a wide potential commercial application such as gaming, virtual reality, facial therapy and security and so on.
People |
ORCID iD |
| Hui Yu (Principal Investigator) |
Publications
Guo Y
(2020)
Real-time facial affective computing on mobile devices
in Sensors
Liu X
(2020)
Region Based Parallel Hierarchy Convolutional Neural Network for Automatic Facial Nerve Paralysis Evaluation.
in IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society
Lou J
(2020)
A Review on Automated Facial Nerve Function Assessment From Visual Face Capture.
in IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society
Lou J
(2021)
Real-Time 3D Facial Tracking via Cascaded Compositional Learning.
in IEEE transactions on image processing : a publication of the IEEE Signal Processing Society
Lou J
(2020)
Realistic Facial Expression Reconstruction for VR HMD Users
in IEEE Transactions on Multimedia
Lou J
(2019)
Multi-subspace supervised descent method for robust face alignment
in Multimedia Tools and Applications
Sun J
(2020)
Two-stage deep regression enhanced depth estimation from a single RGB image
in IEEE Transactions on Emerging Topics in Computing
| Description | A new method that has been developed that can robustly recover a realistic and personalized 3D facial model using only a consumer-grade camera. The proposed method is fully automatic. The reconstruction process needs no manual intervention. A user only needs to show his/her face in front of an off-the-shelf camera for 3D face modelling. The proposed method consists of three main stages: (a) initial target face acquisition; (b) augmented adaptive facial template preparation; (c) robust personalized deformation for 3D facial modelling. The whole process is achieved in a linear way and thus can reconstruct 4D face in real time. The computational cost and the system cost are very low. The proposed method can run fast even on a low end laptop based on CPU computation. Experiments are conducted using both the live data from the camera and the offline data from a public database. We propose a parallel hierarchy convolutional neural network (PHCNN) to quantitatively assess the grading of facial nerve paralysis (FNP) by considering the region-based asymmetric facial features and temporal variation of the image sequences. FNP, such as Bell's palsy, is the most common facial symptom of neuromotor dysfunctions. (a) the proposed deep learning method can have effective FNP grading assement and extract paralysis detail (b)The proposed method can also distinguish the difference between normal faces and faces carrying FNP, showing superior consistency and robustness in FNP classification. |
| Exploitation Route | We will conduct clinical trial before it can be taken forward. |
| Sectors | Creative Economy,Digital/Communication/Information Technologies (including Software),Healthcare |
| Description | Part of the findings is methods and software packages for automatic facial palsy grading. At the core of the system is a deep learning architecture combining a convolutional neural network that can capture both asymmetric facial features and temporal information of facial movement for accurate facial palsy grading. It has been integrated into the product of the industrial partner of the project. |
| First Year Of Impact | 2022 |
| Impact Types | Societal,Economic |
| Description | Dynamic Facial Expression Reconstruction from Upper Half-face Data |
| Amount | £20,148 (GBP) |
| Funding ID | IFS1819\9 |
| Organisation | Royal Academy of Engineering |
| Sector | Charity/Non Profit |
| Country | United Kingdom |
| Start | 09/2018 |
| End | 12/2018 |
| Description | SUV: Summarizing Unconstrained Videos Via Saliency Detection |
| Amount | £60,725 (GBP) |
| Funding ID | NIF\R1\180909 |
| Organisation | The Royal Society |
| Sector | Charity/Non Profit |
| Country | United Kingdom |
| Start | 02/2019 |
| End | 01/2021 |
| Description | Sensor-Enabled Emotion Monitoring Eyewear for ambulatory monitoring of physical activity and facial expressions |
| Amount | £35,450 (GBP) |
| Funding ID | 92465-562215 |
| Organisation | Innovate UK |
| Sector | Public |
| Country | United Kingdom |
| Start | 11/2017 |
| End | 04/2019 |