Multi-activity 3D pose estimation on real environments

Human-machine interfaces, video surveillance, sport performance enhancement, physical therapy, smart environments, to name a few, are important societal challenges that require better automatic behaviour analysis to be fully addressed. In order to move closer to the level of human proficiency, fully automatic understanding of a scene requires a whole range of capabilities: reliable extraction of each actor involved, its pose and their activities. This involves the combined application of pose estimation, multi target tracking and activity recognition. While impressive progress has been made in those fields in isolation, reliable methods, able to be applied to real world and unconstrained environments, are still a challenge. In this project we will focus on the intermediate components of behaviour analysis, by disregarding the traditional cascade pipeline, where pose estimation frequently plays a secondary role or it is completely obliterated due to its complexity, and proposing a novel architecture which has 3D pose estimation as the key central component with feedback between each of the other components.

In this project, we propose to investigate the automated 3D pose estimation and tracking of multiple people in realistic scenarios. This research is suggested on the basis that all current methods perform under strong limitations and assumptions that preclude their application to real-world situations. Thus, while some methods require multiple high-resolution sensors, thereby ruling out the use of current and near future sensor network infrastructures, others struggle with scenes containing multiple persons, or they succeed on the basis of the subjects not interacting and also knowing the activity performed beforehand. This last assumption reduces the practical application of the pose estimation and prevents it use for activity recognition and/or behavioural analysis.

To address this limitation, in this project we propose to extend the assumption from one of a single known activity as prior model, to one where a class of multiple activities is assumed, e.g., walking, running, fighting, shaking hands etc. This requires us to develop a novel multi-activity model that could be used as prior information to accurately and robustly estimate the 3D pose under complex and real world conditions. This multi activity model will avoid presuming the performed activity by each of the subject in the scene among the given set of activities. The development and use of such a model is the key novel contribution of this proposal, and is a first step towards a fully activity-agnostic 3D pose estimation for real environments.

Furthermore, we propose a paradigm change to the conventional behaviour analysis chain, where pose estimation becomes the cornerstone of the system, and the feedback loops with tracking, to address occlusions and interactions, and activity recognition, to switch between a set of plausible activities during the estimation, allows us to deal with the aforementioned issues. By modelling transitions between this set of activities, and observing how predicted poses propagate in time through the activity space, the current activity can be recognised and used as feedback for refining the pose estimation. This is the second novelty of this proposal. Lastly, inaccuracies in the pose estimation, caused by occlusion and multiple persons interacting, can be overcome by using information from the tracker to determine image regions that provide reliable pose estimation information. Similarly, by knowing the pose and activity of subjects in the scene, the tracking performance can be improved. This is the third novel aspect of the proposal.

This research is key to the development of next generation video analytics and surveillance systems. These systems should be able to mimic human performance, thereby ensuring enhanced situation awareness and leading to timely decision making. This proposal seeks to take advantage existing surveillance camera infrastructure, which translates into a reduced cost for companies and the tax payer. The societal impact is two-fold: detecting suspicious behaviour in real-time to prevent crime, and provide forensic evidence that can lead to increased convictions

Next generation video analytics will be key for the protection of Critical National Infrastructures, whose protection will become a major societal concern in the new smart society. Our proposal will enable automatic monitoring and analysis of the behaviour of staff and intruders, to reduce the potential damage and injuries to the staff and to the infrastructure. This is of particular interest for the national security of UK citizens, where the Centre for the Protection of National Infrastructure has identified terrorism as a severe threat to vital services, and has named physical security as 1 of the 3 main disciplines that need to be improved, with appropriate investment in CCTV and intruder alarms. The UK Government has also committed £860m to its national Cyber Security Programme. CSIT, with the Home office, PSNI and GCHQ as industrial advisory board members and collaborators in previous projects, is the perfect environment for this research to achieve this impact

Britain's aging population, the small average family size and a wide geographical dispersal, leave many elderly citizens isolated, with serious consequences for physical and mental health. UK Department of Health expects the number of elderly people over 65 to grow by 51% by 2030. This is currently a key issue in Parliament and has had an impact on the NHS, where a national target was set to increase the number of older people living at home. Assistive technology has shown its value in helping people live independently, improving the quality of life for users and carers, but it has not yet been used to its full potential. Our research will enable the detection of subtle differences between dangerous and normal behaviour while using affordable video sensors, improve the automatic and remote monitoring, increasing the coverage and fast response, while avoiding an increase in cost

Exploiting innovative technology such as this will increase the UK's competitive position both within Europe and on the global stage. The UK Security Market is expected to grow from £2.8bn in 2014 to £3.4bn by 2017 (Competitive Analysis of the UK Cyber Security Sector, PAC, 2013). As a main path, the outcomes of this research will be commercialised though a CSIT spin-off company, Cognition Video. Cognition business model is based on supplying advance analytics to Physical Security Information Management systems and provides. PSIMs provide a platform designed to integrate multiple unconnected security applications in order to identify security breach situations. The outcomes of this research will be added into the current system to provide cutting-edge capabilities that will allow the PSIM company to compete, to grow and to differentiate from the rest of the market, since current PSIM rely on basic analytics, creating new employments and economic gains.

Results will be also disseminated to the CSIT industrial advisory board, which includes major companies in the security sector, such as Thales, BAE and Roke Manor. CSIT also holds regular meetings, summits and white papers, integrating researchers with end users. While taking advantage of CSIT's position as the UK's Innovation and Knowledge Centre for cyber security for maximising impact, this research will also increase the reputation of CSIT as a global innovation hub, enhancing our ability to attract further investment from industry and public sector funds in Europe and beyond