NEAT: NEtwork dATa plane measurements as first class primitive

Lead Research Organisation: Queen Mary, University of London
Department Name: Sch of Electronic Eng & Computer Science

Abstract

Data centre networks are poorly equipped to rapidly spot and address failures, resulting in countless well-documented application performance degradation or outages. This is because the investigation process is performed in centralised commodity servers (collectors) that do not have per-packet visibility, but instead aggregated and sampled statistics from the data plane. The NEAT project will address this deficiency by moving traffic analysis directly into switches that have per-packet visibility. Exploiting advances in programmable hardware, e.g. P4, NEAT will rethink data plane operation and will transform switches from just packet forwarder with limited monitoring capabilities to more intelligent systems capable of analysing traffic and exporting only relevant results. This will enable the level of fine-grained data plane visibility required to allow operators to rapidly identify and adapt to changes in network conditions, which hurts applications.

Planned Impact

The NEAT research programme targets one of the most critical part of today's Internet: data centres. Data centres are a fundamental part of UK infrastructure, underpinning a wide range of activities across government, business, and society. Specifically, NEAT focuses on improving the current monitoring and traffic analysis practices by leveraging data plane programmability on top of an SDN control plane. In this context, my impact plan will reach and influence two main two main constituencies: (1) industry and organisations providing the data centre infrastructure, platforms and services and (2) standardisation bodies developing and promoting open standards in both the SDN and programmable data plane area.

Enabling network traffic analysis techniques in the context of data centres impact a number of players spanning from industry to organisations. The letters of support accompanying this proposal indicate the serious and deep interest in NEAT from a broad array of players: Cisco Systems is a leading networking company, which designs, manufactures, and sells network equipment; Barefoot Networks is another leading networking company, which designs and produces programmable network switch silicon, systems and software. VMware is a subsidiary of Dell Technologies, that provides cloud and vitalisation software and services; CERN is a European research organisation that operates the biggest particle physics
laboratory in the world, where a large computing facility is used to store and analyse data from experiments, as well as simulate events. Those players, given the mixture of expertise ranging from data centre operation to switch design, are key elements to foster innovation in data plane assisted network traffic analysis. They are eager to provide support and benefit from the output of my research by helping a smooth transition from the theory to the practice.

NEAT project will contribute to open standards for programmable networking and monitoring. Standardisation efforts in SDN and data plane programmability are currently thriving with a broad commercial and academic support. I will take part in activities and working groups such as the Open Networking Foundation (ONF), P4 and the Internet Research Task Force (IRTF) to emphasise the importance of data plane assisted traffic analysis in the next generation data centres. My former status of ONF Research Associate and the engagement with Barefoot Networks (see letter of support) opens the door for a fruitful collaboration towards new open standards.

The open source NetFPGA platform is also of particular relevance to NEAT. NetFPGA is an open source hardware/software system with high impact. The NetFPGA platform is a joint project between Stanford University and University of Cambridge, providing an ideal means to disseminate NEAT technology developments to a wide community of researchers, adopters, and investors. Leveraging my status of active member of the official NetFPGA development team and to stimulate a quick uptake of the ideas and prototypes developed under NEAT, I will make the code available under an open source licence. My goal is to build an active user group to eventually provide the research community and industry with mature instruments for exploring theoretical and practical aspects of the NEAT vision.

The novelty, timeliness, and relevance of the NEAT project will ensure a widespread interest in its results from the international community of practitioners, e.g. RIPE, NANOG, MENOG. There will also be abundant opportunities for the technology to be exploited by the industrial supporters. Through those supporters, NEAT will provide significant benefits to the UK industry, increasing its competitiveness in the growing data centre marketplace. The project will lead to high-profile publications and to exciting opportunities for future research. This will further increase the UK's aspirations for the development and use of SDN-enabled technologies.
 
Description Automatic Optimization of Software Data Planes 
Organisation Budapest University of Technology and Economics
Country Hungary 
Sector Academic/University 
PI Contribution I contributed with expertise, intellectual input and access to equipment.
Collaborator Contribution The partners contributed with expertise and intellectual input.
Impact Poster publication: https://dl.acm.org/doi/abs/10.1145/3405837.3411379 Poster publication: https://dl.acm.org/doi/10.1145/3485983.3493349 Paper publication: https://dl.acm.org/doi/10.1145/3503222.3507769
Start Year 2020
 
Description Automatic Optimization of Software Data Planes 
Organisation NEC Corporation
Department NEC Laboratories Europe GmbH
Country Germany 
Sector Private 
PI Contribution I contributed with expertise, intellectual input and access to equipment.
Collaborator Contribution The partners contributed with expertise and intellectual input.
Impact Poster publication: https://dl.acm.org/doi/abs/10.1145/3405837.3411379 Poster publication: https://dl.acm.org/doi/10.1145/3485983.3493349 Paper publication: https://dl.acm.org/doi/10.1145/3503222.3507769
Start Year 2020
 
Description Automatic Optimization of Software Data Planes 
Organisation Polytechnic University of Turin
Country Italy 
Sector Academic/University 
PI Contribution I contributed with expertise, intellectual input and access to equipment.
Collaborator Contribution The partners contributed with expertise and intellectual input.
Impact Poster publication: https://dl.acm.org/doi/abs/10.1145/3405837.3411379 Poster publication: https://dl.acm.org/doi/10.1145/3485983.3493349 Paper publication: https://dl.acm.org/doi/10.1145/3503222.3507769
Start Year 2020
 
Description Building better virtual switches 
Organisation VMware, Inc.
Country United States 
Sector Private 
PI Contribution I contributed with expertise and intellectual input.
Collaborator Contribution The partners contributed with expertise, intellectual input and access to equipment.
Impact paper: https://dl.acm.org/doi/10.1145/3452296.3472914
Start Year 2020
 
Description Detecting Routing Loops in the Data Plane 
Organisation Brno University of Technology
Country Czech Republic 
Sector Academic/University 
PI Contribution I contributed with expertise, intellectual input and access to equipment.
Collaborator Contribution The partners contributed with expertise, intellectual input and access to equipment.
Impact Paper publication: https://doi.org/10.1145/3386367.3431303
Start Year 2020
 
Description Detecting Routing Loops in the Data Plane 
Organisation Harvard University
Country United States 
Sector Academic/University 
PI Contribution I contributed with expertise, intellectual input and access to equipment.
Collaborator Contribution The partners contributed with expertise, intellectual input and access to equipment.
Impact Paper publication: https://doi.org/10.1145/3386367.3431303
Start Year 2020
 
Description Disaggregated Sketches for Traffic Analysis 
Organisation Boston University
Country United States 
Sector Academic/University 
PI Contribution I contributed with expertise and intellectual input.
Collaborator Contribution They contributed with expertise, intellectual input and equipment.
Impact Poster Publication: https://doi.org/10.1145/3386367.3431674
Start Year 2020
 
Description Disaggregated Sketches for Traffic Analysis 
Organisation Harvard University
Country United States 
Sector Academic/University 
PI Contribution I contributed with expertise and intellectual input.
Collaborator Contribution They contributed with expertise, intellectual input and equipment.
Impact Poster Publication: https://doi.org/10.1145/3386367.3431674
Start Year 2020
 
Description Disaggregated Sketches for Traffic Analysis 
Organisation University College London
Country United Kingdom 
Sector Academic/University 
PI Contribution I contributed with expertise and intellectual input.
Collaborator Contribution They contributed with expertise, intellectual input and equipment.
Impact Poster Publication: https://doi.org/10.1145/3386367.3431674
Start Year 2020
 
Description Disaggregated Sketches for Traffic Analysis 
Organisation University of Rome Tor Vergata
Country Italy 
Sector Academic/University 
PI Contribution I contributed with expertise and intellectual input.
Collaborator Contribution They contributed with expertise, intellectual input and equipment.
Impact Poster Publication: https://doi.org/10.1145/3386367.3431674
Start Year 2020
 
Description Scalable Network Collectors 
Organisation Facebook
Country United States 
Sector Private 
PI Contribution I contributed with expertise, intellectual input and access to equipment.
Collaborator Contribution The partners contributed with expertise and intellectual input.
Impact paper: https://dl.acm.org/doi/10.1145/3484266.3487366
Start Year 2021
 
Description Scalable Network Collectors 
Organisation Harvard University
Country United States 
Sector Academic/University 
PI Contribution I contributed with expertise, intellectual input and access to equipment.
Collaborator Contribution The partners contributed with expertise and intellectual input.
Impact paper: https://dl.acm.org/doi/10.1145/3484266.3487366
Start Year 2021
 
Description Scalable Network Collectors 
Organisation University College London
Country United Kingdom 
Sector Academic/University 
PI Contribution I contributed with expertise, intellectual input and access to equipment.
Collaborator Contribution The partners contributed with expertise and intellectual input.
Impact paper: https://dl.acm.org/doi/10.1145/3484266.3487366
Start Year 2021
 
Title Morpheus: Domain Specific Run Time Optimization for Software Data Planes - Artifact for ASPLOS'22 
Description This is the artifact for the "Morpheus: Domain Specific Run Time Optimization for Software Data Planes" paper published at ASPLOS'22. Morpheus is a system working alongside static compilers that continuously optimizes the targeted networking code. It introduces a number of new techniques, from static code analysis to adaptive code instrumentation, together with a toolbox of domain specific optimizations used to manipulate the code on-the-fly depending on runtime traffic patterns and control plane configurations. The Morpheus core exploits the LLVM compiler toolchain (v10.0.1) for code manipulation and run-time code generation. It works at the intermediate representation (IR) level as it allows to reason about the running code using a relatively high-level language framework without compromising on code generation time. This artifact contains the source code, the experimental workflow, and additional information to 1) compile and build Morpheus, 2) install the software dependencies and setup the testbed to run all the experiments, 3) the scripts that can be used to perform some of the experiments presented in the paper, and 4) the scripts to generate the plots based on the obtained results. For more information, please refer to https://github.com/Morpheus-compiler/Morpheus 
Type Of Technology Software 
Year Produced 2021 
Open Source License? Yes  
Impact It is too early at the moment to result in notable impact. 
URL https://zenodo.org/record/5830832
 
Description Guest Lecture at Purdue University 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact I delivered a Guest lecture at Purdue University (CS 536). The purpose was to increase awareness of students on the performance implications around end-host networking and how programmable hardware can be used in this situation. The lecture sparked questions and a good discussion afterwards, with many students interested in the subject.
Year(s) Of Engagement Activity 2021
 
Description Presentation at Brown University 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact I presented the role of programmable hardware in the context of network monitoring. The talk sparked questions and a lengthy discussion afterwards
Year(s) Of Engagement Activity 2021
URL https://systems.cs.brown.edu/sysread/2021-10-08%20%7C%20Gianni%20Antichi/
 
Description Presentation at Cisco Switzerland 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact I presented the role of programmable hardware in the context of network monitoring. The talk sparked questions and a lengthy discussion afterwards
Year(s) Of Engagement Activity 2021
 
Description Presentation at Compute-First Networking (Dagstuhl) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact I presented the role of programmable hardware in the context of network monitoring. The talk sparked questions and a lengthy discussion afterwards
Year(s) Of Engagement Activity 2021
URL https://www.dagstuhl.de/en/program/calendar/semhp/?semnr=21243
 
Description Presentation at Huawei Research Centre (Munich) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact I presented the role of programmable hardware in the context of network monitoring. The talk sparked questions and a lenghty discussion afterwards
Year(s) Of Engagement Activity 2020
 
Description Presentation at University of Trento 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact I presented the role of programmable hardware in the context of network monitoring. The talk sparked questions and a lengthy discussion afterwards
Year(s) Of Engagement Activity 2021