Efficient Cross-Domain DSL Development for Exascale
Lead Research Organisation:
University of Edinburgh
Department Name: Sch of Informatics
Abstract
Developing scientific software, for example for climate modeling or medical research, is a highly challenging task. Domain scientists are often deeply involved in low-level programming details just to make their code run sufficiently fast. These tedious, but important, optimization steps significantly reduce the productivity of scientists.
Domain specific languages (DSLs) revolutionize the productivity of domain scientists by enabling them to focus on scientific questions rather than making their code run fast. Sophisticated DSL compilers automatically generate high-performance code from domain-specific high-level problem descriptions.
While there are individual successes, the existing landscape of DSLs is scattered and the reuse of software components in DSL compiler implementations is limited as traditionally DSL compilers are built in isolation. This results in high development costs of new DSLs and prevents many DSLs from ever achieving a level of maturity and sustainability that enables uptake by the scientific community.
This project revolutionizes the design of DSL compiler implementations by leveraging the breadth and cross-industry support of the MLIR compiler and Python ecosystems. Python is the tool of choice for application developers in many domains, such as machine learning, data science, and - we believe - an important component of the future of High Performance Computing software. This project establishes MLIR as a common representation for code at multiple levels of abstraction in DSL compiler development. DSLs embedded in various host languages, including Python and Fortran, will be easily built on top of MLIR. Instead of building DSL compilers as isolated monolithic towers, our research will build a toolbox that enables developers to build DSLs using a rich ecosystem of shared intermediate representations IRs and optimizations.
This project evaluates, drives, and demonstrates the DSL design toolbox to build the next generation of DSLs for Seismic and Climate Modelling as well as Medical imaging. These will share common software components and make them available for other DSLs. An extensive evaluation will show the scalability of DSL software towards exascale.
Finally, this project investigates how future disruptors, including artificial intelligence, data science, and on-demand HPC-as-a-service, will shape and influence the next generations of high performance software. This project will work towards deeply integrating modern interactive data analytics and machine learning methods from the Python ecosystem with high-performance scientific code.
Domain specific languages (DSLs) revolutionize the productivity of domain scientists by enabling them to focus on scientific questions rather than making their code run fast. Sophisticated DSL compilers automatically generate high-performance code from domain-specific high-level problem descriptions.
While there are individual successes, the existing landscape of DSLs is scattered and the reuse of software components in DSL compiler implementations is limited as traditionally DSL compilers are built in isolation. This results in high development costs of new DSLs and prevents many DSLs from ever achieving a level of maturity and sustainability that enables uptake by the scientific community.
This project revolutionizes the design of DSL compiler implementations by leveraging the breadth and cross-industry support of the MLIR compiler and Python ecosystems. Python is the tool of choice for application developers in many domains, such as machine learning, data science, and - we believe - an important component of the future of High Performance Computing software. This project establishes MLIR as a common representation for code at multiple levels of abstraction in DSL compiler development. DSLs embedded in various host languages, including Python and Fortran, will be easily built on top of MLIR. Instead of building DSL compilers as isolated monolithic towers, our research will build a toolbox that enables developers to build DSLs using a rich ecosystem of shared intermediate representations IRs and optimizations.
This project evaluates, drives, and demonstrates the DSL design toolbox to build the next generation of DSLs for Seismic and Climate Modelling as well as Medical imaging. These will share common software components and make them available for other DSLs. An extensive evaluation will show the scalability of DSL software towards exascale.
Finally, this project investigates how future disruptors, including artificial intelligence, data science, and on-demand HPC-as-a-service, will shape and influence the next generations of high performance software. This project will work towards deeply integrating modern interactive data analytics and machine learning methods from the Python ecosystem with high-performance scientific code.
Organisations
Publications
Rodriguez-Canal G
(2023)
Stencil-HMLS: A multi-layered approach to the automatic optimisation of stencil codes on FPGA
Rodriguez-Canal G
(2023)
Fortran High-Level Synthesis: Reducing the Barriers to Accelerating HPC Codes on FPGAs
Mathieu Fehr
IRDL: An IR Definition Language for SSA Compilers
Mathieu Fehr
(2024)
Sidekick compilation with xDSL
| Description | In the xDSL project, we developed compiler technology that makes it easier for domain experts to build domain-specific compilers that connect closely with the MLIR deep-learning compiler stack. xDSL is written entirely in Python, is easy to install, and is easy to use. With xDSL, we demonstrated that both the Devito and the Psyclone DSL compilers for seismic imaging and climate modeling can share large parts of their compilation stack. To establish such a joint stack, we developed abstractions for MPI-based communication that have been used in the joint stack and have subsequently been contributed to the MLIR compiler ecosystem. Our MPI stack is the first HPC-specific abstraction in the MLIR community and will enable close collaboration across ML and computational science communities. Similarly, early experience has shown that a connection with the MLIR compiler framework allowed for the integration of high-level synthesis tools to use FPGAs as novel acceleration target. We also trained the UK and international science and industry community in compiler technology and made many compiler concepts easily accessible by offering a Python-native interactive compiler environment that is easily accessible by domain experts. As a result, new projects used our xDSL compiler framework to build domain-specific compilers for tensors or new backends for the RISCV architecture. |
| Exploitation Route | The xDSL project is meanwhile being used by various projects to build compiler backends, domain-specific compilers, and accessibility tools for DSL engineers. |
| Sectors | Digital/Communication/Information Technologies (including Software) |
| URL | https://www.xdsl.dev |
| Description | # Bringing HPC and Deep learning closer together We connected the ML and HPC community closer by introducing first-class MPI support to the MLIR ML compiler. Industry users like NVIDIA, Intel, and NextSilicon have already shown interest in using our abstraction. # We facilitated collaborations between academia and industry By organizing events such as the CGO conference in Edinburgh, contributing to conferences such as Supercomputing and the LLVM developer meetings in the US and Europe, and organizing workshops and outreach activities, we enabled technology exchanges and shared technology development between industry and academia. |
| Sector | Digital/Communication/Information Technologies (including Software) |
| Impact Types | Economic |
| Description | Smart and secure ultra-low power processors for Edge AI |
| Amount | € 9,391,000 (EUR) |
| Funding ID | 101070374 |
| Organisation | European Commission |
| Sector | Public |
| Country | European Union (EU) |
| Start | 11/2023 |
| End | 10/2026 |
| Description | CONVOLVE |
| Organisation | Eindhoven University of Technology |
| Country | Netherlands |
| Sector | Academic/University |
| PI Contribution | We became a partner in the CONVOLVE.eu project, where we are leading the compiler work package. We will be using the xDSL framework to build a DNN stack for new low-power embedded hardware. |
| Collaborator Contribution | Our partners are (a) application providers who build DNN stacks and (b) ASIC designers. They will both be using an xDSL based compiler stack to compile their applications to the ASICs designed in CONVOLVE. |
| Impact | One publication: - CONVOLVE: Smart and seamless design of smart edge processors |
| Start Year | 2022 |
| Description | Superoptimization for MLIR |
| Organisation | University of Utah |
| Country | United States |
| Sector | Academic/University |
| PI Contribution | My research team built an smt-based formal semantic model for MLIR as well as toolings for translation validation and superoptimization. |
| Collaborator Contribution | Our partners built static analysis tools for our SMT-based formal semantic model. They also consulted on the design of superoptimizers and offered datasets for our superoptimizer. |
| Impact | This collaborations contributed to our xDSL software artefact. |
| Start Year | 2023 |
| Title | Auto extraction of stencils from Fortran and optimisation |
| Description | Automatic extraction of stencils and then optimisation of these via the xDSL stencil dialect, enabling significant serial speed up of applicable Fortran codes using Flang and auto-optimisation on GPUs and distributed memory. This is the software component of the "Fortran performance optimisation and auto-parallelisation by leveraging MLIR-based domain specific abstractions in Flang" paper that has been associated with this record |
| Type Of Technology | Software |
| Year Produced | 2023 |
| Open Source License? | Yes |
| Impact | The paper "Fortran performance optimisation and auto-parallelisation by leveraging MLIR-based domain specific abstractions in Flang" |
| URL | https://github.com/xdslproject/ftn |
| Title | Devito xDSL integration |
| Description | Integration of the Devito DSL with xDSL, enabling Devito to use MLIR and optimising aspects for the toolchain |
| Type Of Technology | Software |
| Year Produced | 2023 |
| Open Source License? | Yes |
| Impact | A full stack Devito execution using MLIR, which has enabled execution of various benchmarks and codes. |
| URL | https://github.com/xdslproject/devito |
| Title | MPI Dialect abstraction for MLIR |
| Description | We contributed an abstraction for MPI message passing for the MLIR compiler stack. |
| Type Of Technology | Software |
| Year Produced | 2024 |
| Impact | This abstraction is the first distributed-memory HPC abstraction that has been embedded into the LLVM compiler community and forms the foundation for the xDSL compiler stack. It will make it possible to target MPI systems from production compilers such as flang, clang, and MLIR-based domain-specific compilers. |
| URL | https://mlir.llvm.org/docs/Dialects/MPI/ |
| Title | Project website and developer zone |
| Description | The main project website, which we view as a landing place for DSL developers to use as a hub for DSL ecosystem, documentation, tutorials etc. Whilst some of these activities will be further matured as the research progresses, crucially the framework and software has been set up such that we can easily add these components and keep them up to date as activities progress. |
| Type Of Technology | Webtool/Application |
| Year Produced | 2021 |
| Impact | As a landing page we have already used it to generate interest and as a focal area for the project. |
| URL | https://xdsl.dev/ |
| Title | Prototype full-stack PSyclone-MLIR intergration |
| Description | This is a research prototype for integrating PSyclone with xDSL (and then MLIR), enabling the sharing of infrastructure and tooling. It is currently on-going but usable and available to users. |
| Type Of Technology | Webtool/Application |
| Year Produced | 2023 |
| Open Source License? | Yes |
| Impact | We are currently using this within the xDSL project as an evaluator but are coordinating closely with the developers of PSyclone, STFC, to ensure that as the project progresses we can deliver appropriate impact for them |
| Title | xdsl |
| Description | A modern SSA-based compiler implemented in Python |
| Type Of Technology | Webtool/Application |
| Year Produced | 2021 |
| Open Source License? | Yes |
| Impact | We taught a compiler lecture based on xdsl which resulted in many students picking up xDSL |
| URL | https://github.com/xdslproject/xdsl |
| Description | Birds of a Feather (BoF) session at SC21 |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Birds of a feather session at SC21, which is the main world-wide supercomputing conference held in St. Louis USA November 2021. With a world-wide audience we split the hour session based on a short talk, three concurrent break-out sessions, and then a summary session. Idea was to explore consolidation of DSL ecosystems, as focused on the xDSL project, and start to build a global network around this. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://xdsl.dev/2021-11-10-bof-sc21/ |
| Description | Co-organisation of Code Generation and Optimisation (CGO) conference |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | This project is co-organising the CGO conference which is in Edinburgh March 2 - 6, 2024. The world leading conference on the interface of hardware and software on a wide range of optimization and code generation techniques and related issues, it demonstrates the importance of Edinburgh in hosting it and impact of the xDSL project that has been heavily involved in the organisation. |
| Year(s) Of Engagement Activity | 2024 |
| URL | https://conf.researchr.org/home/cgo-2024 |
| Description | DSL Workshop at EuroPar |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | We organised a workshop at the EuroPar conference (held in Glasgow in August 2022) around DSLs for HPC. There was a keynote talk and four accepted papers published in proceedings with LNCS. |
| Year(s) Of Engagement Activity | 2022 |
| URL | https://xdsl.dev/europar/ |
| Description | DSL keynote at IWOMP |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Co-I of project gave keynote talk at International Workshop on OpenMP (IWOMP) conference about DSLs, describing ecosystem consolidation as per the xDSL project. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://xdsl.dev/2021-09-14-iwomp-keynote/ |
| Description | Invited talk at Flang biweekly developer meeting on MPI dialect |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | An invited talk at the Flang bi-weekly developer meeting around the MPI dialect that we have developed in xDSL and wish to merge into MLIR. This was as a preliminary step to the ODM presentation (that we have also added as a record) but very important as it enabled us to gain initial feedback and garner support. |
| Year(s) Of Engagement Activity | 2023 |
| Description | Invited talk at Flang biweekly user group meeting on use of FIR dialect |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | An invited talk at the Flang biweekly developers meeting around the work we are doing with the FIR MLIR dialect for optimisation of parallel codes |
| Year(s) Of Engagement Activity | 2023 |
| Description | Keynote at ExHET PPoPP workshop |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Keynote talk at the ExHET PPoPP workshop |
| Year(s) Of Engagement Activity | 2024 |
| URL | https://ornl.github.io/events/exhet2024/ |
| Description | Led the ExCALIBUR BoF at SC22 |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | We organised and led a BoF session at SuperComputing (SC) 2022 in Dallas around the ExCALIBUR programme and projects. As part of this presented about xDSL. |
| Year(s) Of Engagement Activity | 2022 |
| Description | MLIR Open Developer Meeting (ODM) presentation on MPI dialect |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | The MLIR Open Developer Meetings (ODMs) are a key way in which the community discusses new developments and ideas. We were invited to lead an ODM on the MPI dialect that we have proposed for MLIR, discussing in detail the reason for this and why it is beneficial for MLIR. This is a key aspect of our project, as it enables us to bring HPC abstractions into main-line MLIR after prototyping them in the xDSL tool that has been developed. |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://mlir.llvm.org/OpenMeetings/2023-11-02-MPI-Dialect.pdf |
| Description | Mini-symposium at PASC22 around DSLs for HPC |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | We organised and led a two hour mini-symposium at the PASC conference (held in Basel Switzerland in 2022) around DSLs for HPC. With three invited speakers, one from the Met Office, another from Google, and the third from Washington State University, a diverse set of experiences and ideas were communicated on the topic before going into a panel discussion. |
| Year(s) Of Engagement Activity | 2022 |
| Description | Organisation of the Compiler Social Cambridge (Feb 2024) |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Industry/Business |
| Results and Impact | Our Cambridge Compiler Social brought together the regional compiler design community and interested stakeholders to explore joint open-source activities and learn about the latest MLIR/xDSL ecosystem advances. The key idea of this event was to create a space for exploring ideas, practicing the latest tools, and establishing connections among the stakeholders. Our event was attended by industry (AMD, Intel, ARM, Google Deepmind, Xilinx, Huawei), postgraduate students, professors, and undergraduate students. Attendees traveled from Cambridge, London, and Oxford. The program included: - 1 hour lecture by Tobias Grosser - 5-10 roundtables for discussion on EDA tools, domain-specific compilation, compiler backends, DSLs - 1h workshop on xDSL and its use for interactive high-performance code generation (40-50 participants) The overall event lasted for 4-5 hours. After the incredible uptake of this event, we plan a follow-up event in May. |
| Year(s) Of Engagement Activity | 2024 |
| URL | https://grosser.science/compiler-social-24-feb/ |
| Description | Panellist at ExHET workshop |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Panellist at the ExHET PPoPP workshop |
| Year(s) Of Engagement Activity | 2024 |
| URL | https://ornl.github.io/events/exhet2024/ |
| Description | Presentations at the LLVM Developers Meeting 2024 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Industry/Business |
| Results and Impact | Tobias research group disseminated their research around xDSL at the LLVM developers meeting, the largest open-source compiler conference in the LLVM ecosystem with over 600 attendees. We gave overall 11 talks, which is likely the largest number of talks ever given by a single research group. The highlights: - Our talk on an MPI MLIR dialect. It made the case for including MPI as a first-class abstraction into the compilation stack - xdsl-run: An interpreter for the xDSL compiler stack - Using MLIR from Python: We promote xDSL in general An SMT dialect for assigning semantics to MLIR dialects, Mathieu Fehr (PhD Student) https://llvm.swoogo.com/2023devmtg/session/1767414/an-smt-dialect-for-assigning-semantics-to-mlir- dialects (Correctly) Extending Dominance to MLIR Regions, Siddharth Bhat (PhD Student) https://llvm.swoogo.com/2023devmtg/session/1767406/correctly-extending-dominance-to-mlir-regions MLIR Side Effect Modelling, Siddharth Bhat (PhD Student) https://llvm.swoogo.com/2023devmtg/session/1767441/quick-talks An MPI Dialect for MLIR, Anton Lydike (Research Assistant) https://llvm.swoogo.com/2023devmtg/session/1767441/quick-talks Modernizing types and attributes (short talk) - Mathieu Fehr (PhD Student) https://llvm.swoogo.com/2023devmtg/session/1619586/mlir-workshop xdsl-run: an interpreter for MLIR (short talk), Sasha Lopoukhine (PhD Student) https://llvm.swoogo.com/2023devmtg/session/1619586/mlir-workshop Using MLIR from Python, Mathieu Fehr (PhD Student) https://llvm.swoogo.com/2023devmtg/session/1619586/mlir-workshop Building & Standardizing an Ecosystem for Encrypted Computation with MLIR, Alexander Viand (former visiting PhD student and current collaborator) https://llvm.swoogo.com/2023devmtg/session/1767408/building-standardizing-an-ecosystem-for- encrypted-computation-with-mlir Arcilator: Fast and Cycle-Accurate Hardware Simulation in CIRCT, Martin Erhart (former BSc student and current collaborator) https://llvm.swoogo.com/2023devmtg/session/1767452/arcilator-fast-and-cycle-accurate-hardware- simulation-in-circt Generalized mem2reg for MLIR and how to use it, Theo Degioanni (former visiting MSc Student and current collaborator) https://llvm.swoogo.com/2023devmtg/session/1767440/generalized-mem2reg-for-mlir-and-how-to-use-it MLIR Bufferization: From Tensors to MemRefs, Martin Erhart (former BSc student and current collaborator) https://llvm.swoogo.com/2023devmtg/session/1767436/mlir-bufferization-from-tensors-to-memrefs |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://llvm.swoogo.com/2023devmtg |
| Description | Tutorial with the Convolve EU project |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | A tutorial around how to use our xDSL software with the Convolve EU project who wish to use it for their work around efficient hardware generation |
| Year(s) Of Engagement Activity | 2023 |
| Description | User hackathon |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | A hackathon held in Edinburgh in January 2023, where we had around 15 in-person attendees and 6 virtual attendees. Engagement with users was the key objective, both in terms of helping them use xDSL but also enhancing the core framework. In person attendees from the UK and Switzerland. Virtual attendees from Germany and India. |
| Year(s) Of Engagement Activity | 2023 |
| Description | xDSL Compiler Workshop (Feb/Mar 2024) |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | The xDSL Compiler workshop was a three-day workshop on compiler design with xDSL. We invited key stakeholders from within the UK and beyond to discuss compiler design for AI and domain-specific compilation. We had industry participation from AMD and Jabra as postgraduate students from ETH Zurich, TU Vienna and the University of Utah (USA), as well as numerous students from the UK. The agenda covered the xDSL compiler toolchain, IREE and deep-learning compilers, HPC, and several open brainstorming and hacking sessions. |
| Year(s) Of Engagement Activity | 2024 |
| Description | xDSL MLIR hackathon |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | A week long xDSL/MLIR hackathon that culminated in an MLIR workshop on 9th of May in Edinburgh. The idea of the hackathon was to bring together those who are using, an those who are interested in using, xDSL with xDSL developers to further enhance the ecosystem and support new, novel use-cases. We also spent time working on the core HPC dialects of xDSL, with enhancements made especially around the halo and MPI dialects in order to ensure that these provide good performance. The hackathon culminated on Tuesday the 9th of May with an MLIR workshop organised by xDSL. We had a wide variety of speakers, from those involved in the xDSL project developing dialects and DSLs atop the technology, to those who are heavily leveraging MLIR and interested in the productivity enhancements provided by xDSL. The workshop culminated in round table discussions, where a series of topics were discussed in 30 minute sessions and the 40 or-so workshop participants selected the topic most relevant to them. These ranged from new vectorisation standards, to improved C frontends. |
| Year(s) Of Engagement Activity | 2023 |