Breaking the Copper Bottleneck: Computer Architecture and Power Implications of Photonic Interconnect

Lead Research Organisation: University College London
Department Name: Electronic and Electrical Engineering

Abstract

The provision of future services in the digital economy is reliant on achieving more power efficient computers. Recent bandwidth improvements in electronic interconnects have only been achieved at the expense of dramatic increases in latency and power consumption [19]. Photonic technologies appear essential to make chip-to-chip communication sustainable for ever-higher data rates due to inherently lower power operation. Recent advances in silicon photonics, photonic printed circuit boards (PCB) and 3D integration technologies indicate great promise for short distance photonics. However, given the radical changes in computer design brought about by chip multiprocessors (CMP) and the fundamental differences between electronic and photonic communications, the design implications for complete computer systems are not clear. The proposed research will study the implications of upcoming photonic technologies on the power consumption and architecture of large computer systems such as high performance computers and data centres. The uniqueness of the proposal is its method and the holistic results that it should produce. I will start with a firm scientific foundation based on characterisation of emerging photonic devices leading to models of existing and predicted future components. FPGA-based emulation will enable investigation of complete multi-chip, multi-core computer systems and interconnect running at around 1/100th of real time. The outcome will be the knowledge to build large computer systems optimised for minimum power consumption. This multidisciplinary research therefore underpins several EPSRC themes: digital economy, next-generation healthcare and energy efficiency as well as responding to the EPSRC signposted Moore-for-Less microelectronic grand challenge.

Planned Impact

The provision of future services in the digital economy is reliant on achieving more power efficient computers. The project will determine the performance and power consumption benefits that can be obtained in future multicore, multichip, high performance computers by the use of photonic chip-to-chip networks. The industrial beneficiaries of the research in the medium term will be manufacturers of computer equipment and designers/installers/users of high-end computer systems. The proposal addresses two broad areas of computer design: high performance computing (HPC) and data centre systems. In both areas there are applications which require higher performance computers with reduced or stabilised power consumption. For example: - Supercomputers used in a variety of UK sectors: meteorology, defence, academic scientific computing, financial services, food, and semiconductors; - Business intelligence and analysis - On-line transaction processing - Web search systems - Cloud computing (virtulisation) - Video distribution In the longer term, the results of this work could lead to new mass market products (i.e. photonic PCI) which will lead to new opportunites for computer equipment and photonic system manufacturers. We intend to leverage the commercial possibilities of the work through appointing advisors to the project from interested companies and consideration of consultancy and spin off company opportunities as described in the impact statement. In terms of the impact on wider society, the research will promote sustainable data centres which will allow continued growth of web-based products and services to be developed by government and industry. Ultimately all computer users will benefit from new mass-market products such as photonic PCI. The research will also enable reductions in carbon dioxide emmisions though more sustainable HPC and data centre systems. Improved sustainability in the provision of data centres enables the increased use of web-based services which should lead to increased efficiency and sustainability in other sectors (i.e. reduction in demand for transport). These impacts will be obtained as a by-product of the commercial activity described above with possible government intervention. Sustainability and power consumption reduction are major research and policy themes. The research will define the potential economic and environmental impact of future large computer systems and enable policy makers, research councils and other funding bodies make better informed funding decisions and research plans.

Publications

10 25 50

publication icon
Alistarh D (2015) A High-Radix, Low-Latency Optical Switch for Data Centers in ACM SIGCOMM Computer Communication Review

publication icon
Andreades P. (2015) Experimental demonstration of 75 ns end-to-end latency in an optical top-of-rack switch in Optical Fiber Communication Conference, OFC 2015

publication icon
Audzevich Y (2014) Power Optimized Transceivers for Future Switched Networks in IEEE Transactions on Very Large Scale Integration (VLSI) Systems

publication icon
Funnell A (2017) Hybrid Wavelength Switched-TDMA High Port Count All-Optical Data Centre Switch in Journal of Lightwave Technology

publication icon
Gerard T (2020) Relative impact of channel symbol rate on transmission capacity in Journal of Optical Communications and Networking

publication icon
Madarbux M (2014) Towards zero latency photonic switching in shared memory networks in Concurrency and Computation: Practice and Experience

publication icon
Watts P (2012) Energy Implications of Photonic Networks With Speculative Transmission in Journal of Optical Communications and Networking

publication icon
Zilberman N (2015) Reconfigurable Network Systems and Software-Defined Networking in Proceedings of the IEEE

 
Description The project is investigating the performance and power consumption implications of future computer systems with integrated photonic network elements. The main findings as of March 2016 are:

(1) The power consumption of optical networks interconnecting a rack of servers was characterised with various allocation schemes. It was shown that the allocation scheme is a significant contributor to overall power and can cause power dissipated on the processor chip to vary by an order of magnitude at low loads.

(2) Although it has been previously demonstrated that photonic network-on-chip can reduce power consumption, determining the effect on computer performance is more challenging. By adding optical network models to the open source gem5 computer architecture simulator, we have shown that an optical crossbar interconnect carrying wavelength striped data can increase performance of a 16-core processor by up to 18% depending on the application. In addition we have determined the optimum optical parameters, e.g. number of wavelengths and switching time, for peak performance.

(3) By means of a complete , characterisation of the power consumption of existing optical transceiver circuits, it has been shown that transceiver power consumption can be reduced by 29% by adopting a burst mode physical layer protocol

(4) The main source of latency in an optical switch used for communicating the short messages in a shared memory computer is in calculating the optimum configuration of the switch for multiple completing requests (known as allocation or scheduling). We have demonstrated algorithms which use information from the memory controllers to setup the switch in advance of communication starting. Using these algorithms in common multicore applications in the PARESC benchmark suite, over 70% of all messages require no allocation stage.


(5) We have proposed and experimentally demonstrated an optical top-of-rack switch for data centres with end-to-end latency of 75 ns for server-to-server traffic, a 66% reduction compared with high performance electronic switches. The switch maintains a simplified server-side interface which avoids energy dissipation on the server chip where thermal issues are increasingly critical.

(6) In collaboration with Microsoft Research we have shown that optical switches can be constructed with over 1000 ports (compared with typically 32 - 96 ports for electronic switches. This greatly simplfies the design of the large data centres driving cloud services. Proof of concept experiments have demonstrated the optical physical layer and scheduler. Furthermore, techniques for building control planes for such large switches have been propsed and demonstrated.
Exploitation Route This project defines how future computer systems and processor chips with integrated photonic interconnect can be used to reduce power and maintain or increase performance. Potentially, the results will be used in all future computer systems, including consumer devices, data centres and high performance computing. The transceiver power models generated in this project (CONTEST) have been made available on an open source basis to the research community, academic and commercial. We have also made our switch control plane models available on an open source basis (NetEmulation). This open source IP is expected to lead to consultancy opportunities. The work also has commercial potential for exploitation through project partners (Xilinx, Inphi and Microsoft Research) or start up companies.
Sectors Digital/Communication/Information Technologies (including Software),Electronics

URL http://www.ee.ucl.ac.uk/ong/group-research/photonicinter
 
Description Microsoft Research, in collaboration with my group at UCL, Oclaro and Inphi, are investigating the use of high port-count switches developed in this project for use in future Microsoft data centres. Microsoft is one of the world's largest data centre operators. UCL and Microsoft are jointly developing a server-to-server data centre network demonstrator based on the technology developed during this project. Close links with industry have been maintained during the project. Two PhD students directly funded by the project have spend 6 month internships, with Bell Labs and leading UK processor technology company ARM, working on projects closely related to the project. Three PhD students spent internships working with Microsoft Research.
First Year Of Impact 2014
Sector Digital/Communication/Information Technologies (including Software),Electronics
Impact Types Economic

 
Description UCL Impact Award with Inphi Corporation providing 60% of funding
Amount £75,116 (GBP)
Organisation University College London 
Sector Academic/University
Country United Kingdom
Start 09/2014 
End 09/2018
 
Title CONTEST 
Description CONfigurable Transceiver Energy uSage Toolkit (CONTEST) is an open source model which allows the characterisation of the energy consumption of the physical layer of optical transceivers including line coding, frame alignment, channel bonding, serialisation and deseralisation, clock/data recovery and clock generation. 
Type Of Material Improvements to research infrastructure 
Year Produced 2013 
Provided To Others? Yes  
Impact 10.1109/TVLSI.2013.2283300 Funding from Inphi Corporation for PhD studentship (UCL Impact Award scheme) 
URL http://www.ee.ucl.ac.uk/ong/contest
 
Title NetEmulation 
Description An open source library of optical switch control plane models in SystemVerilog which allow (1) simulation of optical switch performance (2) energy consumption measurement using ASIC synthesis (3) demonstration of optical switching using FPGA synthesis. 
Type Of Material Improvements to research infrastructure 
Year Produced 2014 
Provided To Others? Yes  
Impact 10.1364/JOCN.4.000503 Invited paper in IEEE/OSA Journal of Optical Communications and Networking in final stages of review 
URL https://github.com/pmwatts/NetEmulation
 
Description Cambridge CAPE 
Organisation University of Cambridge
Country United Kingdom 
Sector Academic/University 
PI Contribution Expertise in optical network/switch control planes
Collaborator Contribution Expertise in integrated optical switch devices
Impact 10.1364/OFC.2014.Th4J.3 Invited paper in IEEE/OSA Journal of Optical Communications and Networking in final stages of review
Start Year 2013
 
Description Cambridge Computer Laboratory 
Organisation University of Cambridge
Country United Kingdom 
Sector Academic/University 
PI Contribution Expertise in physical layer optical communications and switching, 2 PhD EPSRC funded students working on the collaboration. Access to optical communications laboratory facilities.
Collaborator Contribution Expertise in computer architecture, digital and FPGA design and computer networking. Experimental facilities in computer networking.
Impact doi:10.1002/cpe.3334 10.1109/ATC.2013.6698126 10.1109/TVLSI.2013.2283300. 10.1109/HOTI.2013.14 978-1-4799-0457-0 10.1364/JOCN.4.000503. Development of the CONTEST open source transceiver model: http://www.ee.ucl.ac.uk/ong/contest
Start Year 2008
 
Description Microsoft Research 
Organisation Microsoft Research
Department Microsoft Research Cambridge
Country United Kingdom 
Sector Private 
PI Contribution We developed the concept of optical switches for data centres based on star couplers and coherent receivers whch scale to over 1000 ports. We performed physical layer feasibility experiments and contributed to the design of the scheduler and demonstrator.
Collaborator Contribution MSR deveoped the concept for the use of high port count switches within data centres to reduce latency, cost and operational complexity. They performed simulations with workloads from Microsoft data centres to demonstrate the benefits at the system level.
Impact Hardware demonstration of the physical layer at SIGCOMM 2015 Paper on the full concept submitted to SIGCOMM 2016 Paper on the physical layer accepted for presentation at OFC 2016 with journal paper submitted UCL and Microsoft Research are collaborating to build a server-to-server data centre network demonstrator based on the technology developed during the collaboration. The research covers all aspects of data centre networking from the physical layer optical components, scheduling to network layer and application level performance assessment.
Start Year 2014
 
Description Xilinx 
Organisation Xilinx Research
Country United States 
Sector Private 
PI Contribution Expertise in optical switching control planes
Collaborator Contribution Donation of FPGA development boards for optical switch demonstration. Expertise in FPGA and communication protocol technology.
Impact 10.1109/OIC.2012.6224446 Invited paper in Journal of Optical Communications and Networking
Start Year 2010