TASCC: Pervasive low-TeraHz and Video Sensing for Car Autonomy and Driver Assistance (PATH CAD)

This project combines novel low-THz (LTHz) sensor development with advanced video analysis, fusion and cross learning. Using the two streams integrated within the sensing, information and control systems of a modern automobile, we aim to map terrain and identify hazards such as potholes and surface texture changes in all weathers, and to detect and classify other road users (pedestrians, car, cyclists etc.).

The coming era of autonomous and assisted driving necessitates new all-weather technology. Advanced concepts of interaction between the sensed and processed data, the control systems and the driver can lead to autonomy in decision and control, securing all the needed information for the driver to intervene in critical situations. The aims are to improve road safety through increased situational awareness, and increase energy efficiency by reducing the emission of pollutants caused by poor control and resource use in both on and off-road vehicles.

Video cameras remain at the heart of our system: there are many reasons for this: low cost, availability, high resolution, a large legacy of processing algorithms to interpret the data and driver/passenger familiarity with the output. However it is widely recognized that video and/or other optical sensors such as LIDAR (c.f. Google car) are not sufficient. The same conditions that challenge human drivers such as heavy rain, fog, spray, snow and dust limit the capability of electro-optical sensors. We require a new approach.

The key second sensor modality is a low-THz radar system operating within the 0.3-1 THz frequency spectrum. By its very nature radar is robust to the conditions that limit video. However it is the relatively short wavelength and wide bandwidth of this LTHz radar with respect to existing automotive radar systems that can bring key additional capabilities. This radar has the potential to provide: (i) imagery that is closer to familiar video than those provided by a conventional radar, and hence can begin to exploit the vast legacy of image processing algorithms; (ii) significantly improved across-road image resolution leading to correspondingly significant improvements in vehicle, pedestrian and other 'actor' (cyclists, animals etc.) detection and classification; (iii) 3D images that can highlight objects and act as an input to the guidance and control system; (iv) analysis of the radar image features, such as shadows and image texture that will contribute to both classification and control.

The project is a collaboration between three academic institutions - the University of Birmingham with its long standing excellence in automotive radar research and radar technologies, the University of Edinburgh with world class expertise in signal processing and radar imaging and Heriot-Watt University with equivalent skill in video analytics, LiDAR and accelerated algorithms. The novel approach will be based on a fusion of video and radar images in a cross-learning cognitive process to improve the reliability and quality of information acquired by an external sensing system operating in all-weather, all-terrain road conditions without dependency on navigation assisting systems.

Within the European community, the total number of fatalities in road traffic accidents decreased by 61 % between 1990 and 2012 (source: Eurostat Yearbook 2015), a welcome statistic that has been driven by technology, including automatic braking and traction control systems, GPS, navigation and road hazard warning, and mechanical aids including the seat belt and air bag. However, 90% of accidents are still caused by human error, so the next step towards greater safety and autonomous capability is through dynamic extra-sensory perception. Of course, advanced prototype automobiles are equipped with radar, LiDAR and video sensors, each of which has complementary strengths, but their impact is always going to be limited because electro-optical sensors cannot function effectively in bad weather, and current radar systems operate only in 2 dimensions and do not give sufficient spatial resolution for situational awareness. The key impact of this proposal which marries novel THz sensor development to advanced video analysis, fusion and cross learning is to enable detection and classification of road users, detailed 3D structure and surface property resolution for dynamic situational awareness in all weather conditions, essential for future deployment because such systems must especially be deployed in unfavorable as well as fair conditions. The project has the potential to improve road safety in conventional automobiles, and also enable autonomous situational awareness for both on- and off-road vehicles. Thus there are many potential beneficiaries of this research.

A significant beneficiary of this research will, of course, be the industrial sponsor Jaguar Land Rover (JLR). The main benefit that will accrue inside the partnership of JLR and this consortium is to provide JLR with a new world-leading sensing system that will ultimately enhance safety, efficiency and road capacity. In order to enable the launch of a new generation of truly" autonomous" vehicles operating in all weathers and all terrains, the development of an innovative sensing system is required in a relatively compressed timeframe. The ready availability of equipment and expertise within JLR and this university consortium will enable rapid progress, within the timeframe and cost, and lay the foundation for a novel sensing system with extended capabilities that industry can exploit. Thus there is clear potential for exploitation by JLR itself. Additionally, there are opportunities for other companies such as SME's who are active in taking innovative sensor/signal processing concepts to market.

The other industrial beneficiaries of this research include the large UK defence sector with interests in radar and sensing. This includes companies such as BAE Systems, Selex ES, QinetiQ, Thales UK, Roke Manor Research as well as DSTL and the MoD itself. All weather and all terrain operation is particularly important in this sector. These companies actively exploit significant innovation in the civilian sector through the use of what they call commercial-off-the-shelf technology (COTS). This has the economy of scale that is hard to achieve in the defence sector. The sensing system invented in this project would provide a unique set of capabilities for both autonomous and non-autonomous military vehicles. Commercial exploitation in the civilian sector will lead to integrated circuits and LTHz components that can be exploited by the defence sector for sensing and vehicle control.