The Internet of Silicon Retinas (IOSIRE): Machine-to-machine communications for neuromorphic vision sensing data

Lead Research Organisation: Kingston University
Department Name: Fac of Science Engineering and Computing

Abstract

This proposal starts with the notion that, when considering future visual sensing technologies for next-generation Internet-of-Things surveillance, drone technology, and robotics, it is quickly becoming evident that sampling and processing raw pixels is going to be extremely inefficient in terms of energy consumption and reaction times. After all, the most efficient visual computing systems we know, i.e., biological vision and perception in mammals, do not use pixels and frame-based sampling. Therefore, IOSIRE argues that we need to explore the feasibility of advanced machine-to-machine (M2M) communications systems that directly capture, compress and transmit neuromorphically-sampled visual information to cloud computing services in order to produce content classification or retrieval results with extremely low power and low latency.

IOSIRE aims to build on recently-devised hardware for neuromorphic sensing, a.k.a. dynamic vision sensors (DVS) or silicon retinas. Unlike conventional global-shutter (frame) based sensors, DVS cameras capture the on/off triggering corresponding to changes of reflectance in the observed scene. Remarkably, DVS cameras achieve this with (i) 10-fold reduction in power consumption (10-20 mW of power consumption instead of hundreds of milliwatts) and (ii) 100-fold increase in speed (e.g., when the events are rendered as video frames, 700-2000 frames per second can be achieved).

In more detail, the IOSIRE project proposes a fundamentally new paradigm where the DVS sensing and processing produces a layered representation that can be used locally to derive actionable responses via edge processing, but select parts can also be transmitted to a server in the cloud in order to derive advanced analytics and services. The classes of services considered by IOSIRE require a scalable and hierarchical representation for multipurpose usage of DVS data, rather than a fixed representation suitable for an individual application (such as motion analysis or object detection). Indeed, this is the radical difference of IOSIRE from existing DVS approaches: instead of constraining applications to on-board processing, we propose layered data representations and adaptive M2M transmission frameworks for DVS data representations, which are mapped to each application's quality metrics, response times, and energy consumption limits, and will enable a wide range of services by selectively offloading the data to the cloud. The targeted breakthrough by IOSIRE is to provide a framework with extreme scalability: in comparison to conventional designs for visual data processing and transmission over M2M networks, and under comparable reconstruction, recognition or retrieval accuracy in applications, up to 100-fold decrease in energy consumption (and associated delay in transmission/reaction time) will be pursued. Such ground-breaking boosting of performance will be pursued via proof-of-concept designs and will influence the design of future commercial systems.
 
Description A model for the data rate output by neuromorphic vision sensors has been developed. Based on features of the captured scene and information on the motion of the camera/sensor, the output data-rate is estimated. This is particularly useful to design the methods and technologies to transmit such data.

A statistical analysis of the traffic output by neuromorphic vision sensors has also been performed. This information can also inform the transmission strategy for such data.

A comparison of the compression methodologies for neuromorphic vision sensor data has been performed, highlighting what is the level of compression that we can achieve without loss of information ("lossless compression").

A comparison of scheduling strategies for visual data has been performed, highlighting which ones are more suitable for scenarios with stringent delay requirements.
Exploitation Route The model and the statistical analysis of the output traffic will enable identifying appropriate transmission technologies for data acquired via neuromorphic sensors.
The compression technologies studied will enable lower data rate / energy saving in data transmission.
Sectors Aerospace, Defence and Marine,Agriculture, Food and Drink,Digital/Communication/Information Technologies (including Software),Environment,Healthcare,Transport

URL https://ieeexplore.ieee.org/abstract/document/8580801
 
Description Collaboration with industries involved in the project (e.g. Inilabs, Thales, Samsung) 
Organisation Samsung
Country Global 
Sector Private 
PI Contribution Presentation of the plan for the project and preliminary results
Collaborator Contribution Feedback on the preliminary results and discussion of further sources of data from the company. Presentation of previous work from the company.
Impact No specific joint output yet, although the model developed took into account some discussions with the company.
Start Year 2017
 
Description Collaboration with industries involved in the project (e.g. Inilabs, Thales, Samsung) 
Organisation Thales Group
Department Thales UK Limited
Country United Kingdom 
Sector Private 
PI Contribution Presentation of the plan for the project and preliminary results
Collaborator Contribution Feedback on the preliminary results and discussion of further sources of data from the company. Presentation of previous work from the company.
Impact No specific joint output yet, although the model developed took into account some discussions with the company.
Start Year 2017
 
Description Collaboration with industries involved in the project (e.g. Inilabs, Thales, Samsung) 
Organisation iniLabs Ltd
Country Switzerland 
Sector Private 
PI Contribution Presentation of the plan for the project and preliminary results
Collaborator Contribution Feedback on the preliminary results and discussion of further sources of data from the company. Presentation of previous work from the company.
Impact No specific joint output yet, although the model developed took into account some discussions with the company.
Start Year 2017
 
Description Iosire project description - press release 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Media (as a channel to the public)
Results and Impact Introduction to the project and interview with KU PI (M. Martini) was reported in Kingston University news: http://www.kingston.ac.uk/news/article/1825/27-apr-2017-kingston-university-to-play-leading-role-in-study-examining-how-stateoftheart-camera-that-mimics-human/ and also distributed via Kingston University social media (Twitter/Facebook/Linkedin). The press release was reported in a number of magazines targeting scientific and general public. Examples include The Engineer : https://www.theengineer.co.uk/uk-team-to-lead-research-into-artificial-eye-technology/, optics.org, imv.europe.org, Pioneer magazine, E & T magazine (IET), Next Nature.
Year(s) Of Engagement Activity 2017
URL https://www.theengineer.co.uk/uk-team-to-lead-research-into-artificial-eye-technology/
 
Description New Scientist Live - London Excel 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact "New Scientist Live" event at London Excel, 20-23 September 2018

Kingston University - with Dr Nabeel Khan and Prof Maria Martini - participated in the "New Scientist Live" event at London Excel in September 2018, presenting to the general public visiting the Kingston University stand the results and ongoing activity of the IoSiRe project. A demo was presented, showing how neuromorphic sensors capture the scene and how such information is stored ans processed.
The "New Scientist Live" is a large science festival with more than 120 speakers and 100 exhibitors contributing to thought-provoking talks and presenting ground-breaking discoveries, interactive experiences, workshops and performances. Thousands of attendees visited the stands (https://live.newscientist.com).

https://twitter.com/kingstonuni/status/1043111356193497088?lang=bg
Year(s) Of Engagement Activity 2018
URL https://live.newscientist.com