On-Sensor Computer Vision

Bringing advanced computer vision to edge devices such as robots, consumer electronics, or sensor networks is challenging due to the constraints of power, size and communication bandwidth under which they often operate. We propose vertically integrated research into the paradigm of on-sensor computer vision, where sensing and processing are unified into single chip which produces abstract, information-rich output rather than images. We aim to demonstrate that on-sensor computer vision can be much more powerful and general than seen in previous research, and that the correct hardware design, software framework and algorithm choices permit switchable or even simultaneous computation of a broad set of vision competences (such as motion estimation, segmentation and scene classification) on a single device. We propose to work on the design of on-sensor computer vision systems through a programme of work from pixel-processing architecture design and microelectronic hardware implementation, through software platform development, to unified algorithm design and experimentation to determine how to use this hardware in a full application. This will enable camera devices that not only capture images, but have a powerful built-in vision capability to understand what they are looking at, and ultimately can go from light to decision on a single sensor/processor chip, with unprecedented speed, low power consumption and small footprint. We hope to open up a new class of edge applications where cameras can be much more efficient and independent, or for smart cameras to be used in ways never previously considered.