3D human tracking and scene reconstruction for audio-visual AR/VR

In this research, the student will investigate identification and tracking of people throughout complex multi-person scenes using audio-visual input from a camera for AR/VR applications. Real-time 3D scene rendering and spatial audio reproduction algorithms require listener location and head pose (sweet spot) related to the display/loudspeakers and room geometry. Real-time scene tracking and adaptive 3D scene rendering will be the goal of this project. Two types of display scenarios will be considered. First one is a personalised Head-Mounted Display (HMD) system such as a VR headset with headphones. The second type is to use a communal display (e.g., TV or car windscreen) with loudspeakers.