Oxide Recurrent Neural Networks

The human brain masters the tasks of associative memory, object detection, body movement, speech and text recognition and classification, which are still challenging for the standard computer software to tackle. Brain-like, that is, neuromorphic algorithms have been developed to deal with the aforementioned problems.

The main component is the neural network of artificial neurons that are linked together by connection weights which can be "trained" to deliver the desired set of outputs, for example, to assign a meaning to a spoken word. The software that does "learning by doing" is one of the major constituents of the recent rapid development in the field of artificial intelligence technology, which has considerably improved the cognitive functionality of modern electronics, for example, face and speech recognition in smartphones.

But still, even though an average computer can process by far much more mathematical operations per second than the human brain, when it comes to the energy efficiency the human brain is the clear winner, being by several orders of magnitude more energy efficient. In particular, the so-called recurrent neural networks that were developed for processing signals in form of temporal sequences and do include the prehistory of the signal into computations, represent a complicated case in terms of energy-efficiency, which is highly desirable in mobile or always-on applications. The sequential nature of temporal signals makes it difficult to minimise energy costs due to weights storage and repeated vector-matrix multiplication.

There is already some success in using thin non-stoichiometric oxide films that demonstrate variable electric resistance, the so-called memristors, for performing parallel computations of vector-matrix products at low energy costs. Arrays of memristors prepared between sets of metallic crossbar electrodes can be implemented for image recognition in neural networks, with electric conductance values of memristors representing the corresponding weights.

We propose (i) to prepare and characterise thin, defect-rich oxide films that will play the role of a physical substrate where the incoming signals will be transformed in accordance with their prehistory, re-routed, and re-mixed to the output signals. (ii) to incorporate those films into prototypes of new type hardware operating by the approach of recurrent neural networks and being able to process, i.e. generate, predict or classify time-dependant signals typical for different medical, engineering and environmental monitoring sensors, robotics control, video and audio recordings; (iii) to test and benchmark those device prototypes in terms of performance and energy efficiency with respect to a set of tasks, for example, spoken word recognition.

The following groups of stakeholders will benefit from the outcomes of this research: Academic and industrial researchers working either on the hardware for neural networks, analog and neuromorphic computing or developing technologies which require processing and classification of sequential signals by performing for example pattern classification, pattern generation and time series forecasting. Successful realisation of a new substrate for framework of the recurrent neural networks using the approach of reservoir computing is interesting for the industry since its major advantage, the low training costs (in terms of time) and the simplicity of the training algorithm makes it easy to implement it into practical applications which need to operate in real time and offline. Physical realisation on a substrate made from thin oxide film will also allow for relatively robust, energy-efficient devices tolerant to different physical environments. Oxides are also easier to incorporate into the existing CMOS devices. In the long term this development can lead to increased business revenues.

Success of industrial researchers will mean development of new technologies and products in the following industrial branches (from G. Tanaka et al., Neural Networks 115, 100 (2019)):
a) biomedical applications dealing with monitoring heart rates, eye movement, electrocardiogram, electroencephalogram and electromyogram signals. For example, sensors that could provide preliminary analysis of real-time signals from heart, brain or muscles, will increase significantly quality and effectiveness of medical support for general public (i.e. additional societal impact);

b) machinery applications such as monitoring and controlling operation of vehicles, robots, sensors, motors, compressors, controllers and actuators. Here, there is large demand for compact and energy-efficient sensors and processors with machine learning capabilities;

c)variety of energy engineering applications that deal with monitoring power plants, power lines and batteries, steam generators, gas flow in the pipelines. Employing the machine learning in this area will reduce the involvement of the human operators to the minimum thus also reducing the risk of human failure, and could benefit the society in the long-term perspective by reducing risks of industrial accidents;

d)communication industry: operating radiocommunications, internet traffic and phone calls;

e)environmental monitoring: ozone concentration, rainwater, seismic activity, air quality monitoring. Wide distribution of energy-efficient sensors with AI capabilities for the purpose of risk monitoring, impact reduction and environmental protection will also be beneficial for the society;

f)financial sector: monitoring stock indices, stock prices and exchange rates;

g)entertainment and security electronics sector: recognition of spoken words, natural sounds and images (the latter, if encoded as sequential signals).