Centre for Spatial Computational Learning
Lead Research Organisation:
Imperial College London
Department Name: Electrical and Electronic Engineering
Abstract
The rise of the Deep Neural Network as an increasingly universal paradigm of computation has been the defining feature across much of computing in the last few years. At the same time, the traditional von Neumann processor paradigm is being challenged by the rise of hardware spatial computational accelerators, such as FPGAs, which face serious usability and programmability challenges. The deep learning application domain is narrow enough to allow us to reconsider the entire stack of spatial compute, from arithmetic circuits to cloud-based systems, in an integrated and domain-specific manner. We have strong expertise across the breadth of this space. If successful, our joint research centre has the potential to put the UK's expertise at the centre of the technology revolutionising the way high performance and low energy computation is specified and delivered, opening opportunities from ultra-low energy machine learning in internet of things devices through to powering scientific discovery in high-end server farms.
This centre would mark a break-through in international coordination of research in the field. We will: (a) work together to evolve a joint research strategy, (b) deliver elements of that joint research strategy through the staff employed on this grant as well as through academic staff and PhD students in all institutions, (c) facilitate a managed researcher exchange programme through funding exchange / secondment expenses for investigators and visiting researchers, as well as supporting secondments to/from industry (d) hold annual showcase events of our work, and (e) deliver high-quality R&D and STEM advocacy outreach.
By the end of the three-year period, we expect to have a self-sustaining momentum of internationally-coordinated research, incorporating the initial investigators, but extending beyond to new groups and the network of SMEs developing in this area.
This centre would mark a break-through in international coordination of research in the field. We will: (a) work together to evolve a joint research strategy, (b) deliver elements of that joint research strategy through the staff employed on this grant as well as through academic staff and PhD students in all institutions, (c) facilitate a managed researcher exchange programme through funding exchange / secondment expenses for investigators and visiting researchers, as well as supporting secondments to/from industry (d) hold annual showcase events of our work, and (e) deliver high-quality R&D and STEM advocacy outreach.
By the end of the three-year period, we expect to have a self-sustaining momentum of internationally-coordinated research, incorporating the initial investigators, but extending beyond to new groups and the network of SMEs developing in this area.
Planned Impact
1. Economic Impact
Our strategy here will be to release software and programmable hardware-design tool and artefacts under a permissive open-source license such as the MIT Licence, allowing for future commercialisation, at the same time as exploring avenues for that exploitation through our advisory board. As is standard practice, background IP ownership will not be affected by this project; the PI will actively manage the project to minimise use of any background IP that is only available under restrictive licensing, and preference will always be given to commercialisable open-source licensed background in the event of various options being available. We expect existing industrial partners to benefit directly from this work, and will use the contacts of the ElecTech council, whose CEO will chair our advisory group, to ensure impact on UK SMEs beyond the initial set of partners.
2. Societal Impact
The nature of this research project - both the technical nature and the international nature - lends itself to some exciting outreach and public engagement possibilities. Both Constantinides and Merrett have strong records in outreach; Merrett will lead the outreach and public engagement programme of our project.
We aim to work with our industrial partners to engage the public in the following ways: (i) through participation in the Science Museum Lates series, through which we have had significant success in the past, (ii) through internal and external festivals / science fairs, which last year included a contribution from Bouganis on machine learning for drones, and (iii) through the production of public-facing high quality YouTube videos and Twitter, an approach taken by Merrett in the PRiME programme grant.
3. People Impact
One major impact of the proposed centre will be on the research staff and PhD students who will work under the centre's auspices. They will end their study / research contract immeasurably stronger in skill set and research depth for their exposure to the best international research environments in the area. Our programme of internships and placements will leave them as some of the most employable individuals within the high-tech world, whether in academia or in industry. Through participation in our annual retreats, they will have gained exposure to both academic and industrial thinking from the top players in three countries; investigators will track and mentor individuals in their career paths. No matter where these individuals move on to at the end of their project, this will leave the UK's ICT community immeasurably strengthened by having individuals with a strong positive experience of mentoring under the auspices of an EPSRC project, leaving key advocates in place for the importance of STEM research funding in the UK.
Our strategy here will be to release software and programmable hardware-design tool and artefacts under a permissive open-source license such as the MIT Licence, allowing for future commercialisation, at the same time as exploring avenues for that exploitation through our advisory board. As is standard practice, background IP ownership will not be affected by this project; the PI will actively manage the project to minimise use of any background IP that is only available under restrictive licensing, and preference will always be given to commercialisable open-source licensed background in the event of various options being available. We expect existing industrial partners to benefit directly from this work, and will use the contacts of the ElecTech council, whose CEO will chair our advisory group, to ensure impact on UK SMEs beyond the initial set of partners.
2. Societal Impact
The nature of this research project - both the technical nature and the international nature - lends itself to some exciting outreach and public engagement possibilities. Both Constantinides and Merrett have strong records in outreach; Merrett will lead the outreach and public engagement programme of our project.
We aim to work with our industrial partners to engage the public in the following ways: (i) through participation in the Science Museum Lates series, through which we have had significant success in the past, (ii) through internal and external festivals / science fairs, which last year included a contribution from Bouganis on machine learning for drones, and (iii) through the production of public-facing high quality YouTube videos and Twitter, an approach taken by Merrett in the PRiME programme grant.
3. People Impact
One major impact of the proposed centre will be on the research staff and PhD students who will work under the centre's auspices. They will end their study / research contract immeasurably stronger in skill set and research depth for their exposure to the best international research environments in the area. Our programme of internships and placements will leave them as some of the most employable individuals within the high-tech world, whether in academia or in industry. Through participation in our annual retreats, they will have gained exposure to both academic and industrial thinking from the top players in three countries; investigators will track and mentor individuals in their career paths. No matter where these individuals move on to at the end of their project, this will leave the UK's ICT community immeasurably strengthened by having individuals with a strong positive experience of mentoring under the auspices of an EPSRC project, leaving key advocates in place for the importance of STEM research funding in the UK.
Organisations
- Imperial College London (Lead Research Organisation)
- Cornell University (Collaboration)
- University of Toronto (Collaboration, Project Partner)
- University of California, Los Angeles (UCLA) (Collaboration)
- UNIVERSITY OF SYDNEY (Collaboration)
- KING'S COLLEGE LONDON (Collaboration)
- University of California, Los Angeles (Project Partner)
- ARM (United Kingdom) (Project Partner)
- Corerain Technologies (Project Partner)
- Imagination Technologies (United Kingdom) (Project Partner)
- Xilinx (United States) (Project Partner)
- Maxeler Technologies (United Kingdom) (Project Partner)
Publications
Wang E
(2023)
Logic Shrinkage: Learned Connectivity Sparsification for LUT-Based Neural Networks
in ACM Transactions on Reconfigurable Technology and Systems
Wang E
(2022)
Logic Shrinkage
Wang E
(2020)
LUTNet: Learning FPGA Configurations for Highly Efficient Neural Network Inference
in IEEE Transactions on Computers
Wang E
(2023)
Enabling Binary Neural Network Training on the Edge
in ACM Transactions on Embedded Computing Systems
Voss N
(2020)
Towards Real Time Radiotherapy Simulation
in Journal of Signal Processing Systems
Voss N
(2021)
On Predictable Reconfigurable System Design
in ACM Transactions on Architecture and Code Optimization
Vandebon J
(2021)
Enhancing High-Level Synthesis Using a Meta-Programming Approach
in IEEE Transactions on Computers
Vandebon J
(2020)
SLATE: Managing Heterogeneous Cloud Functions
Vandebon J
(2021)
Scheduling Hardware-Accelerated Cloud Functions
in Journal of Signal Processing Systems
Todman T
(2022)
Custom Instructions for Networked Processor Templates
in IEEE Transactions on Circuits and Systems II: Express Briefs
Tarafdar N
(2021)
AIgean : An Open Framework for Deploying Machine Learning on Heterogeneous Clusters
in ACM Transactions on Reconfigurable Technology and Systems
Tarafdar N
(2020)
AIgean: An Open Framework for Machine Learning on Heterogeneous Clusters
Sohrabizadeh A
(2020)
End-to-End Optimization of Deep Learning Applications
Sahebi A
(2023)
Distributed large-scale graph processing on FPGAs.
in Journal of big data
Sadiq S
(2024)
Enabling ImageNet-Scale Deep Learning on MCUs for Accurate and Efficient Inference
in IEEE Internet of Things Journal
Sabet A
(2022)
Similarity-Aware CNN for Efficient Video Recognition at the Edge
in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
Rasoulinezhad S
(2021)
Rethinking Embedded Blocks for Machine Learning Applications
in ACM Transactions on Reconfigurable Technology and Systems
Ramhorst B
(2023)
FPGA Resource-aware Structured Pruning for Real-Time Neural Networks
Rajagopal A
(2021)
perf4sight: A toolflow to model CNN training performance on Edge GPUs
Que Z
(2022)
Recurrent Neural Networks With Column-Wise Matrix-Vector Multiplication on FPGAs
in IEEE Transactions on Very Large Scale Integration (VLSI) Systems
Que Z
(2020)
Optimizing Reconfigurable Recurrent Neural Networks
Que Z
(2021)
In-circuit tuning of deep learning designs
in Journal of Systems Architecture
Que Z
(2024)
LL-GNN: Low Latency Graph Neural Networks on FPGAs for High Energy Physics
in ACM Transactions on Embedded Computing Systems
Que Z
(2020)
Mapping Large LSTMs to FPGAs with Weight Reuse
in Journal of Signal Processing Systems
Que Z
(2022)
Remarn: A Reconfigurable Multi-threaded Multi-core Accelerator for Recurrent Neural Networks
in ACM Transactions on Reconfigurable Technology and Systems
Que Z
(2024)
Low Latency Variational Autoencoder on FPGAs
in IEEE Journal on Emerging and Selected Topics in Circuits and Systems
Nakahara H
(2020)
R2CNN: Recurrent Residual Convolutional Neural Network on FPGA
Description | Please see https://twitter.com/spatialmlnet for full details |
Exploitation Route | Being explored with industrial advisory board |
Sectors | Digital/Communication/Information Technologies (including Software) Electronics |
URL | https://twitter.com/spatialmlnet |
Description | Used inside consortium academically and also this patent with Intel https://patents.justia.com/patent/20220366112 |
First Year Of Impact | 2022 |
Sector | Digital/Communication/Information Technologies (including Software),Electronics |
Impact Types | Economic |
Description | Two research students -- bespoke |
Amount | £955,605 (GBP) |
Organisation | Advanced Micro Devices (AMD) |
Sector | Private |
Country | United States |
Start | 12/2022 |
End | 04/2028 |
Title | Data for Similarity-aware CNN for Efficient Video Recognition at the Edge |
Description | This data is associated with the the "Similarity-aware CNN for Efficient Video Recognition at the Edge" article published in IEEE transaction on Computer-Aided Design of Integrated Circuits and Systems. |
Type Of Material | Database/Collection of data |
Year Produced | 2021 |
Provided To Others? | Yes |
URL | https://eprints.soton.ac.uk/452703/ |
Title | Dataset for "Dynamic Transformer for Efficient Machine Translation on Embedded Devices" |
Description | This dataset supports the publication: 'Dynamic Transformer for Efficient Machine Translation on Embedded Devices' in '3rd ACM/IEEE Workshop on Machine Learning for CAD (MLCAD'21)'. |
Type Of Material | Database/Collection of data |
Year Produced | 2021 |
Provided To Others? | Yes |
URL | https://eprints.soton.ac.uk/450454/ |
Title | Dataset in support of the article 'Enabling Image Net-scale deep learning on MCUs for accurate and efficient inference' |
Description | Data obtained in Enabling Image Net-Scale Deep Learning on MCUs for Accurate and Efficient Inference research. To support an article published in IEEE Internet of Things Journal |
Type Of Material | Database/Collection of data |
Year Produced | 2023 |
Provided To Others? | Yes |
URL | https://eprints.soton.ac.uk/id/eprint/483991 |
Description | Cornell (& Cornell Tech) |
Organisation | Cornell University |
Country | United States |
Sector | Academic/University |
PI Contribution | Research ideas |
Collaborator Contribution | Studentships |
Impact | https://twitter.com/spatialmlnet |
Start Year | 2022 |
Description | Kings |
Organisation | King's College London |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | Working with Prof Bashir Al-Hashimi |
Collaborator Contribution | Working on project as described |
Impact | See publications |
Start Year | 2020 |
Description | Sydney |
Organisation | University of Sydney |
Country | Australia |
Sector | Academic/University |
PI Contribution | Research ideas |
Collaborator Contribution | Studentship |
Impact | https://twitter.com/spatialmlnet |
Start Year | 2022 |
Description | Toronto |
Organisation | University of Toronto |
Country | Canada |
Sector | Academic/University |
PI Contribution | Research ideas |
Collaborator Contribution | Studentships |
Impact | https://twitter.com/spatialmlnet |
Start Year | 2020 |
Description | UCLA |
Organisation | University of California, Los Angeles (UCLA) |
Country | United States |
Sector | Academic/University |
PI Contribution | Research ideas |
Collaborator Contribution | Studentships |
Impact | https://twitter.com/spatialmlnet |
Start Year | 2020 |
Title | Apparatus, Device, Method, and Computer Program for Generating a Register Transfer Level Representation of a Circuit |
Description | Work with industrial partner Intel and centre student Coward |
IP Reference | 17/649,937 |
Protection | Patent / Patent application |
Year Protection Granted | 2022 |
Licensed | Commercial In Confidence |
Impact | Improved Intel tools for circuit design |
Title | PolyLUT |
Description | Neural network training for FPGA implementation |
Type Of Technology | Software |
Year Produced | 2023 |
Open Source License? | Yes |
Impact | Becoming widely cited. 25 stars, 2 forks on GitHub. |
URL | https://github.com/MartaAndronic/PolyLUT |
Title | fpgaConvNet |
Description | Tool for automatically mapping neural networks to FPGA hardware |
Type Of Technology | Software |
Year Produced | 2024 |
Impact | Widely cited. Note this is constantly refined software, so date is latest refinement. |
URL | https://cas.ee.ic.ac.uk/people/sv1310/fpgaConvNet.html |
Description | Online public seminars (and YouTube channel) |
Form Of Engagement Activity | A broadcast e.g. TV/radio/film/podcast (other than news/press) |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | Raise awareness of work of the centre. Widely shared on professional networks thousands of times. Youtube figures can be viewed via link below. |
Year(s) Of Engagement Activity | 2023,2024 |
URL | https://www.youtube.com/@spatialmlseminar/videos |