The Internet of Silicon Retinas (IoSiRe): Machine to machine communications for neuromorphic vision sensing data

Lead Research Organisation: King's College London
Department Name: Informatics

Abstract

This proposal starts with the notion that, when considering future visual sensing technologies for next-generation Internet-of-Things surveillance, drone technology, and robotics, it is quickly becoming evident that sampling and processing raw pixels is going to be extremely inefficient in terms of energy consumption and reaction times. After all, the most efficient visual computing systems we know, i.e., biological vision and perception in mammals, do not use pixels and frame-based sampling. Therefore, IOSIRE argues that we need to explore the feasibility of advanced machine-to-machine (M2M) communications systems that directly capture, compress and transmit neuromorphically-sampled visual information to cloud computing services in order to produce content classification or retrieval results with extremely low power and low latency.

IOSIRE aims to build on recently-devised hardware for neuromorphic sensing, a.k.a. dynamic vision sensors (DVS) or silicon retinas. Unlike conventional global-shutter (frame) based sensors, DVS cameras capture the on/off triggering corresponding to changes of reflectance in the observed scene. Remarkably, DVS cameras achieve this with (i) 10-fold reduction in power consumption (10-20 mW of power consumption instead of hundreds of milliwatts) and (ii) 100-fold increase in speed (e.g., when the events are rendered as video frames, 700-2000 frames per second can be achieved).

In more detail, the IOSIRE project proposes a fundamentally new paradigm where the DVS sensing and processing produces a layered representation that can be used locally to derive actionable responses via edge processing, but select parts can also be transmitted to a server in the cloud in order to derive advanced analytics and services. The classes of services considered by IOSIRE require a scalable and hierarchical representation for multipurpose usage of DVS data, rather than a fixed representation suitable for an individual application (such as motion analysis or object detection). Indeed, this is the radical difference of IOSIRE from existing DVS approaches: instead of constraining applications to on-board processing, we propose layered data representations and adaptive M2M transmission frameworks for DVS data representations, which are mapped to each application's quality metrics, response times, and energy consumption limits, and will enable a wide range of services by selectively offloading the data to the cloud. The targeted breakthrough by IOSIRE is to provide a framework with extreme scalability: in comparison to conventional designs for visual data processing and transmission over M2M networks, and under comparable reconstruction, recognition or retrieval accuracy in applications, up to 100-fold decrease in energy consumption (and associated delay in transmission/reaction time) will be pursued. Such ground-breaking boosting of performance will be pursued via proof-of-concept designs and will influence the design of future commercial systems.

Planned Impact

Given the significant savings in bandwidth, energy consumption and network latencies expected through the adoption of IOSIRE based technology in video over M2M networks, the general public will benefit from the results of this research via the development of advanced classification, retrieval and video frame rate upscaling services that are impossible to achieve with conventional frame-based video systems. The project outcomes will enable a wide range of IoT-related applications for machine-to-machine networks of the future, thus helping to meet public expectations for the future of the Internet-of-Things paradigm. The role of industry, and in particular our industrial partners, will be paramount here, especially in view of the significance of international standards in the widespread adoption of our new media processing and M2M communication techniques in vertical sectors. The dissemination of our research outputs to standardisation bodies (such as 3GPP, IEEE and the ITU) as well as industry fora (Cambridge Wireless, MWC, etc.) will facilitate this impact.

Our industrial partners are well positioned to exploit the research outcomes within their products and services and the planned interactions with them will substantially facilitate this. As detailed in the Impact document, this potentially encompasses sensor/transceiver chip set development, M2M network architecture and management, and consumer products with enhanced video-over-wireless connectivity.

Publications

10 25 50
 
Description We have gathered with our collaborators new data and model on dynamic visual sensing, a new visual sensing method that enables transmission with less bandwidth and at lower latency, which will be used in this project and by others (including industry) for further study. The first article on DVS traffic data is made ready for submission. Furthermore, we have a had numerous publications on various aspects of learning/processing, modelling and communication of visual data based on DVS technology.
Exploitation Route Through our industrial partners: they will be informed of our findings, e.g. through the workshops such the one that we organised for this project in June 2018. Through one of our industrial partners, we have explored new potential applications of this technology in healthcare. Also iniLabs, the producer of DVS cameras and project partner, is keenly following the project results for improving the future versions of their products.
Also at King's 5G lab we are exploring implementation of our new DVS communication and routing methods in collaboration with the project partner, Ericsson.
Sectors Digital/Communication/Information Technologies (including Software),Healthcare,Transport

URL http://www.iosire.com
 
Description The project lead, Prof M. Shikh-Bahaei, arranged a workshop at King's, dedicated to this project, inviting external experts and project's industrial partners. The three academic partners presented their early results, and industrial partners gave their views in their presentation and also during the panel discussions. Challenges of implementing the technologies of the project were discussed in detail. Also at UCL the UCL PI arranged a meeting with other investigators and with industrial partners to put the methods of the project in use by industry. Also new data has been generated for DVS traffic, which will be potentially used by the industry.
First Year Of Impact 2018
Sector Digital/Communication/Information Technologies (including Software)
Impact Types Policy & public services

 
Description Collaboration with industries involved in the project (e.g. Inilabs, Thales, Samsung) 
Organisation Samsung
Country Global 
Sector Private 
PI Contribution Presentation of the plan for the project and preliminary results
Collaborator Contribution Feedback on the preliminary results and discussion of further sources of data from the company. Presentation of previous work from the company.
Impact No specific joint output yet, although the model developed took into account some discussions with the company.
Start Year 2017
 
Description Collaboration with industries involved in the project (e.g. Inilabs, Thales, Samsung) 
Organisation Thales Group
Department Thales UK Limited
Country United Kingdom 
Sector Private 
PI Contribution Presentation of the plan for the project and preliminary results
Collaborator Contribution Feedback on the preliminary results and discussion of further sources of data from the company. Presentation of previous work from the company.
Impact No specific joint output yet, although the model developed took into account some discussions with the company.
Start Year 2017
 
Description Collaboration with industries involved in the project (e.g. Inilabs, Thales, Samsung) 
Organisation iniLabs Ltd
Country Switzerland 
Sector Private 
PI Contribution Presentation of the plan for the project and preliminary results
Collaborator Contribution Feedback on the preliminary results and discussion of further sources of data from the company. Presentation of previous work from the company.
Impact No specific joint output yet, although the model developed took into account some discussions with the company.
Start Year 2017
 
Description IOSIRE 
Organisation Ericsson
Country Sweden 
Sector Private 
PI Contribution We feedback the results of our research in the project to these collaborators.
Collaborator Contribution They attend at our technical meetings and give advise on the research methodologies and on the best way to produce impact
Impact Our published research articles are results of our work within King's, UCL and Kingston, and are also of collaboration with these partners.
Start Year 2017
 
Description IOSIRE 
Organisation Keysight Technologies
Country Unknown 
Sector Private 
PI Contribution We feedback the results of our research in the project to these collaborators.
Collaborator Contribution They attend at our technical meetings and give advise on the research methodologies and on the best way to produce impact
Impact Our published research articles are results of our work within King's, UCL and Kingston, and are also of collaboration with these partners.
Start Year 2017
 
Description IOSIRE 
Organisation MediaTek Inc.
PI Contribution We feedback the results of our research in the project to these collaborators.
Collaborator Contribution They attend at our technical meetings and give advise on the research methodologies and on the best way to produce impact
Impact Our published research articles are results of our work within King's, UCL and Kingston, and are also of collaboration with these partners.
Start Year 2017
 
Description IOSIRE 
Organisation Thales Group
Department Thales Research & Technology (Uk) Ltd
Country United Kingdom 
Sector Private 
PI Contribution We feedback the results of our research in the project to these collaborators.
Collaborator Contribution They attend at our technical meetings and give advise on the research methodologies and on the best way to produce impact
Impact Our published research articles are results of our work within King's, UCL and Kingston, and are also of collaboration with these partners.
Start Year 2017
 
Description IOSIRE 
Organisation iniLabs Ltd
Country Switzerland 
Sector Private 
PI Contribution We feedback the results of our research in the project to these collaborators.
Collaborator Contribution They attend at our technical meetings and give advise on the research methodologies and on the best way to produce impact
Impact Our published research articles are results of our work within King's, UCL and Kingston, and are also of collaboration with these partners.
Start Year 2017