Seeing the future

Newspapers do not report stories about the sun rising or setting. This because what is expected or unchanging is of little interest, what is newsworthy is the unexpected or changing. In 1961 Horace Barlow, a British vision scientist, hypothesised that the sensory systems of the brain work on the same principle, it devotes little investigation to the expected and reports the unusual. It does this by trying to predict what is going to happen and when it fails it funnels resources at the "prediction errors". This makes intuitive sense, the things in a scene that merit attention are typically those that unpredictable or changing. This idea, "predictive coding" has proven very fruitful and today it underpins much research on how the brain works.

The aim of this project is to translate the idea of predictive coding from the fields of psychology and biology to the design of a computer vision system. The system will try to predict the upcoming camera image. Prediction will be based on the previous images seen (over immediate, intermediate and longer time scales), information about movement of the system through space (from the inertial sensor), and knowledge of how objects and people move. Prediction errors will be flagged.

This is a feasibility study, to determine and demonstrate the benefits of such a system and form a basis for future work.

Our specific objectives are to:

i) Evaluate the technical feasibility of building a predictive vision system.
ii) Assess the projected efficiency and accuracy of a predictive vision system.
iii) Evaluate the effectiveness of a predictive vision system for detecting changes or incongruencies in the scene.
iv) Assess the computational completeness and sufficiency of the predictive coding approach (this result will be fed back to the biology/psychology community).
v) Identify potential partners and explore potential industrial, security or healthcare applications of the technology.
vi) Take appropriate steps to protect intellectual property and then release the data and code to the broad research community.
vii) On successful completion seek follow-on funding (research council and/or industrial) for further work.

The project brings together expertise in computer vision, human vision and computational biology across two partner (GW4; Cardiff and Bristol) universities to work on this innovative and cutting edge work. The system is to be built from a stereo camera, an inertial sensor, and a CPU/GPU. This technology is available in some smartphones today. If the efficiency and accuracy benefits are realized then the technology could be based on a smartphone which opens up a broad range of applications, including in healthcare, security and ubiquitous computing.

This 18 month feasibility study will build a demonstrator predictive vision system, evaluate its strengths and weaknesses, and explore potential application areas and broader use. During planned activities (e.g. workshop with industrial, security and healthcare delegates) we will explore the future applications of a predictive vision system. Below we outline potential beneficiaries of the work.

Economic Impact: Building Industrial Niches in the UK
The worldwide machine vision industry is expected to grow from 8 billion USD to 12 billion USD by 2020. Within this industrial niche, there are a large number of potential applications of our new technology. If deployed on a smart phone (see below) there are an abundance of applications in reach at very little cost. This opens up major industrial/business opportunities many of which can be envisioned (some examples below) but many unexpected applications will be envisioned by innovative smart phone developers.

Video Surveillance is a commonplace, wide ranging and massively growing application of video camera, entire "Smart Cities" are under increasing surveillance and many public and work places are observed for security and safety monitoring. Police forces worldwide are currently being deployed with body worn camera, as are military personnel. Many private vehicles and Police/Security forces have in car cameras (e.g. GoPro). An obvious application of our 'smart' video capture system would be to record and feed back anomalies to security agents, without the need for massive data storage and operator screening requirements.

Next-Generation Video Compression, Change Detection and Saliency mapping: A successful demonstrator system could provide a foundation for new approaches to video compression, event detection and salience mapping in a variety of video analysis scenarios (in particular monitoring and surveillance).

Greener smart sensor technologies: The efficiency advantages of a predictive system should allow the development of lower-energy smart sensors.

Ubiquitous computing, smartphone apps: A key feature of the predictive system is that it detects "interesting" events (changes or incongruities) in the scene in real time. All components of our system already exist on a smartphone. In a few years time, smartphone sensors will no doubt improve in accuracy and smartphone processing power will increase to allow our methods to be effectively deployed on such devices, opening potential for many far reaching applications and future technologies.

Societal Impact: Building a healthy, competitive and resilient society

Predictive sensing in care of elderly or infirm: Medicine has many ready-made challenges that our technology could assist. For example, a device that could be placed in a person's home, a care home or hospital to monitor for abnormal movement patterns, at one extreme that could be falls, at the other it could be early signs of Parkinson's gait or lameness etc., both would be flagged as prediction errors, deviations from the priors.

Predictive sensing as an aid for those with attentional or perceptual deficits: Visually impaired people, due low vision, or people with attentional problems, following brain injury, stroke, or just general ageing, could be equipped with an assistive sensor (incorporated in a handheld device, or a device worn on the body, on the chest or on the head in a google Glass type arrangement), to scan the environment and alert them to important events in the scene.

Developing World Leading Scientists: A key role will be played by this project's PDRA who work close collaboration with the investigator team. They will present at international conferences (see resources) where they can build their own reputation and establish, moving forward, their own scientific career (with multidisciplinary collaborators).