Hardware-aware Network Architecture Search under ML Training workloads

Achieving the optimal delay/energy/accuracy requires the co-optimization of Neural Network (NN) workload together with the system hardware it is deployed in. This is particularly important on mobile systems running on batteries, where to deploy more complex workloads, retraining-adaptation might be limited due to energy or computation constraints.
Neural Architecture Search (NAS) solutions enable the exploration of the NN graph characteristics in an efficient way, adapting the design specifics of the neural architecture to the machine learning task at hand. The aim of this PhD tackles the design of an open-source, hardware aware NAS methodology to optimize not only inference but also on-device training. Furthermore, hardware considerations will be expanded to lower-level processor specifications (for instance, NVM parameters, DVFS design) alongside system-level characteristics.
Key Research Questions
The objective of the research is to investigate the open problems in the domain of hardware-aware network architecture search (NAS). In particular, it will focus on a proof of concept system model that will be aware of established opportunities for efficiency. Examples of these include different frequency/voltage operation points (DVFS) and hybrid memories (off-chip/on-chip, non-volatile/volatile) with multiple power operation modes. A key challenge will be integrating this hardware-defined search-space with that of search-space of the neural architecture to perform joint-optimization of both neural architecture and hardware design spaces in a tractable manner. The design of this NAS method will be a useful tool, but just as important will be the results this tool is able to produce - specifically novel neural architectures and novel hardware designs that especially when run jointly produce high efficiency systems. It is expected that by analysing the workload-aware optimized systems, design and micro-architectural innovations will result in the exceptional power, performance and area metrics for such heterogeneous hardware designs.
Work expected of the ICASE student
During the Ph.D, the student will research and address the computing and memory requirements of a system model that includes the NAS scheme and optimises an on-device training workload.
While addressing the key research questions, the student will work in a top-class environment, with colleagues from the CaMLSys group at the University of Cambridge. There will be a minimum of a 3 month placement with the industrial partner IMEC where the student will work in a diverse environment with scientists devoted to world leading R&D and innovation in nanoelectronics and digital technologies.