DiRAC SMP Facility: Revealing the Structure of the Universe
Lead Research Organisation:
University of Cambridge
Department Name: Applied Maths and Theoretical Physics
Abstract
This proposal aims to address key questions about the fundamental structure of the Universe and the origin and nature of the galaxies, stars and planets within it. This proposal is to enhance the STFC DiRAC Facility which provides the primary computational platform for UK particle physicists, cosmologists and astrophysicists. The proposal will fund the DiRAC SMP Node, a highly flexible shared-memory (SMP) system with up to 16TB of globally addressable memory, which will be the largest such system in Europe and the first of its kind in the world to have Many Integrated Core accelerators. This HPC system will aim to advance the study of the early universe, the cosmic microwave sky and extra-solar planets. The unique SMP capabilities of this flexible architecture will be available to the wider community of DiRAC users.
This proposal will advance our understanding in three key scientific areas:
A. Science exploitation of the Cosmic Microwave Sky and Large-scale Structure surveys:
The cosmic microwave background (CMB) remains the premier source for cosmological information. Planck satellite data, in which many COSMOS consortium members have leading roles, dramatically supersedes the previous WMAP satellite. The new SMP node will be actively used to open up new frontiers in CMB science which will constrain fundamental cosmology. With key Planck data releases scheduled for 2013 and 2014, the proposed SMP upgrade would be timed strategically to leverage the proprietary Planck data, maximizing science exploitation by UK members prior to release. Complementary work on large-scale structure will encompass a range of topics, from using surveys to constrain fundamental properties of the Universe, to understanding the hierarchical formation of galaxies and uncovering their detailed properties.
B. Observational consequences of the Early Universe.
The COSMOS consortium has pioneered the use of lattice field theory simulations to understand the physics of non-linear phenomena in the early universe. Such epochs are expected in most cosmological models derived from fundamental theories such as the unification of forces, cosmic defects, extra dimensions, and string and M-theory. The challenge using the new SMP node will be to identify and calculate the observable consequences of these theories, which can range from signatures in the cosmic microwave background or the large-scale structure of the universe through to the production of dark matter or primordial gravitational waves.
C. Extra-solar planets and their atmospheres.
The DiRAC SMP node will also support key UK research in extra-solar planets by the Miracle consortium. This is concerned with the observation and characterisation of exoplanets and their atmospheres, developing key numerical codes which are vital to this international endeavour. The new SMP system will allow much more information to be extracted from spectroscopic data from exoplanet environments. This will help us answer some of the oldest questions in science such as: Are there worlds beyond our solar system? Are they numerous or rare? How many of them have the right conditions for life?
This proposal will advance our understanding in three key scientific areas:
A. Science exploitation of the Cosmic Microwave Sky and Large-scale Structure surveys:
The cosmic microwave background (CMB) remains the premier source for cosmological information. Planck satellite data, in which many COSMOS consortium members have leading roles, dramatically supersedes the previous WMAP satellite. The new SMP node will be actively used to open up new frontiers in CMB science which will constrain fundamental cosmology. With key Planck data releases scheduled for 2013 and 2014, the proposed SMP upgrade would be timed strategically to leverage the proprietary Planck data, maximizing science exploitation by UK members prior to release. Complementary work on large-scale structure will encompass a range of topics, from using surveys to constrain fundamental properties of the Universe, to understanding the hierarchical formation of galaxies and uncovering their detailed properties.
B. Observational consequences of the Early Universe.
The COSMOS consortium has pioneered the use of lattice field theory simulations to understand the physics of non-linear phenomena in the early universe. Such epochs are expected in most cosmological models derived from fundamental theories such as the unification of forces, cosmic defects, extra dimensions, and string and M-theory. The challenge using the new SMP node will be to identify and calculate the observable consequences of these theories, which can range from signatures in the cosmic microwave background or the large-scale structure of the universe through to the production of dark matter or primordial gravitational waves.
C. Extra-solar planets and their atmospheres.
The DiRAC SMP node will also support key UK research in extra-solar planets by the Miracle consortium. This is concerned with the observation and characterisation of exoplanets and their atmospheres, developing key numerical codes which are vital to this international endeavour. The new SMP system will allow much more information to be extracted from spectroscopic data from exoplanet environments. This will help us answer some of the oldest questions in science such as: Are there worlds beyond our solar system? Are they numerous or rare? How many of them have the right conditions for life?
Planned Impact
The DiRAC SMP facility will be a state-of-the-art supercomputer system which at delivery will be:
- Largest shared-memory single-image system in Europe.
- First Intel MIC-enabled SMP system in the world (i.e. Many Integrated Core technology)
From the perspective of UK cosmology and astrophysics research communities, the DiRAC SMP node will be an internationally competitive facility showcasing the next generation of highly flexible user-friendly HPC technologies.
The primary purpose of the DiRAC SMP facility is to enable UK researchers to maintain international
leadership in the investigation of the origin and structure of the Universe. This entails simulation of innovative theoretical ideas and comparison with vast new observational data sets, notably
exploiting maps of the cosmic microwave sky and large galaxy surveys, as well as spectral data from extra-solar planets and their atmospheres.
This work has a significant economic and social impact through (i) training in high performance computing and (ii) collaboration and interaction with HPC technology developers and (iii) public understanding of science and technology. The SMP will have a large user base of 30-50 new users annually (mainly PDRAs and graduate students) and it will maintain a special emphasis on user accessibility and education, ensuring maximum benefit to UK science and subsequent employers. We will also work with commerce, research design labs and other academic areas to produce new IT (hardware and software) solutions, and we will collaborate with leading IT vendors to test and design workload management systems, compilers and performance tools.
DiRAC SMP node users have a high public profile and will continue an active programme of outreach through open days, print media, internet, radio and television. Cosmology and exoplanets are areas of major public interest.
- Largest shared-memory single-image system in Europe.
- First Intel MIC-enabled SMP system in the world (i.e. Many Integrated Core technology)
From the perspective of UK cosmology and astrophysics research communities, the DiRAC SMP node will be an internationally competitive facility showcasing the next generation of highly flexible user-friendly HPC technologies.
The primary purpose of the DiRAC SMP facility is to enable UK researchers to maintain international
leadership in the investigation of the origin and structure of the Universe. This entails simulation of innovative theoretical ideas and comparison with vast new observational data sets, notably
exploiting maps of the cosmic microwave sky and large galaxy surveys, as well as spectral data from extra-solar planets and their atmospheres.
This work has a significant economic and social impact through (i) training in high performance computing and (ii) collaboration and interaction with HPC technology developers and (iii) public understanding of science and technology. The SMP will have a large user base of 30-50 new users annually (mainly PDRAs and graduate students) and it will maintain a special emphasis on user accessibility and education, ensuring maximum benefit to UK science and subsequent employers. We will also work with commerce, research design labs and other academic areas to produce new IT (hardware and software) solutions, and we will collaborate with leading IT vendors to test and design workload management systems, compilers and performance tools.
DiRAC SMP node users have a high public profile and will continue an active programme of outreach through open days, print media, internet, radio and television. Cosmology and exoplanets are areas of major public interest.
People |
ORCID iD |
Edward Shellard (Principal Investigator) |
Publications
Berti E
(2016)
Numerical relativity and high energy physics: Recent developments
in International Journal of Modern Physics D
Berti E
(2015)
Testing general relativity with present and future astrophysical observations
in Classical and Quantum Gravity
Berti E
(2013)
Numerical simulations of single and binary black holes in scalar-tensor theories: Circumventing the no-hair theorem
in Physical Review D
Bethke L
(2013)
Anisotropies in the gravitational wave background from preheating.
in Physical review letters
Bethke L
(2014)
On the anisotropy of the gravitational wave background from massless preheating
in Journal of Cosmology and Astroparticle Physics
Biagetti M
(2017)
Verifying the consistency relation for the scale-dependent bias from local primordial non-Gaussianity
in Monthly Notices of the Royal Astronomical Society
Bignell R
(2023)
Charm baryons at finite temperature on anisotropic lattices
Bilimogga P
(2022)
Using eagle simulations to study the effect of observational constraints on the determination of H i asymmetries in galaxies
in Monthly Notices of the Royal Astronomical Society
Bird S
(2012)
Massive neutrinos and the non-linear matter power spectrum The matter power with neutrinos
in Monthly Notices of the Royal Astronomical Society
Bisbas T
(2015)
torus-3dpdr: a self-consistent code treating three-dimensional photoionization and photodissociation regions
in Monthly Notices of the Royal Astronomical Society
Bisbas T
(2014)
A photodissociation region study of NGC 4038
in Monthly Notices of the Royal Astronomical Society
Bisbas T
(2015)
starbench: the D-type expansion of an H ii region
in Monthly Notices of the Royal Astronomical Society
Blondin S
(2022)
StaNdaRT: a repository of standardised test models and outputs for supernova radiative transfer
in Astronomy & Astrophysics
Boera E
(2014)
The thermal history of the intergalactic medium down to redshift z = 1.5: a new curvature measurement
in Monthly Notices of the Royal Astronomical Society
Bolton J
(2017)
The Sherwood simulation suite: overview and data comparisons with the Lyman a forest at redshifts 2 = z = 5
in Monthly Notices of the Royal Astronomical Society
Bolton J
(2014)
A consistent determination of the temperature of the intergalactic medium at redshift = 2.4
in Monthly Notices of the Royal Astronomical Society
Bolton J
(2022)
Limits on non-canonical heating and turbulence in the intergalactic medium from the low redshift Lyman a forest
in Monthly Notices of the Royal Astronomical Society
Bolton J
(2013)
On the rapid demise of Ly a emitters at redshift z ? 7 due to the increasing incidence of optically thick absorption systems
in Monthly Notices of the Royal Astronomical Society
Bolton JS
(2022)
Comparison of Low-Redshift Lyman-a Forest Observations to Hydrodynamical Simulations with Dark Photon Dark Matter.
in Physical review letters
Booth R
(2019)
Characterizing gravito-turbulence in 3D: turbulent properties and stability against fragmentation
in Monthly Notices of the Royal Astronomical Society
Booth RA
(2016)
Collision velocity of dust grains in self-gravitating protoplanetary discs.
in Monthly notices of the Royal Astronomical Society
Borrow J
(2022)
Sphenix : smoothed particle hydrodynamics for the next generation of galaxy formation simulations
in Monthly Notices of the Royal Astronomical Society
Borukhovetskaya A
(2022)
Galactic tides and the Crater II dwarf spheroidal: a challenge to LCDM?
in Monthly Notices of the Royal Astronomical Society
Borukhovetskaya A
(2022)
The tidal evolution of the Fornax dwarf spheroidal and its globular clusters
in Monthly Notices of the Royal Astronomical Society
Bosman S
(2022)
Hydrogen reionization ends by z = 5.3: Lyman-a optical depth measured by the XQR-30 sample
in Monthly Notices of the Royal Astronomical Society
Bower R
(2020)
Numerical convergence of hydrodynamical simulations of galaxy formation: the abundance and internal structure of galaxies and their cold dark matter haloes
in Monthly Notices of the Royal Astronomical Society
Boyle P
(2023)
Simulating rare kaon decays K + ? p + l + l - using domain wall lattice QCD with physical light quark masses
in Physical Review D
Boyle P
(2020)
Latest Results on Lattice Calculation Concerning K ? p l + l - Decays
in Journal of Physics: Conference Series
Boyle P
(2023)
Isospin-breaking corrections to light-meson leptonic decays from lattice simulations at physical quark masses
in Journal of High Energy Physics
Bozorgnia N
(2020)
The dark matter component of the Gaia radially anisotropic substructure
in Journal of Cosmology and Astroparticle Physics
Brady R
(2024)
Numerical Equivalence of Diabatic and Adiabatic Representations in Diatomic Molecules
in Journal of Chemical Theory and Computation
Brady RP
(2022)
An ab initio study of the rovibronic spectrum of sulphur monoxide (SO): diabatic vs. adiabatic representation.
in Physical chemistry chemical physics : PCCP
Briggs J
(2016)
Separable projection integrals for higher-order correlators of the cosmic microwave sky: Acceleration by factors exceeding 100
in Journal of Computational Physics
Brook C
(2013)
The Lowest Metallicity Stars in the LMC: Clues from MaGICC Simulations
in Publications of the Astronomical Society of Australia
Brook C
(2014)
MaGICC baryon cycle: the enrichment history of simulated disc galaxies
in Monthly Notices of the Royal Astronomical Society
Brook C
(2012)
Thin disc, thick disc and halo in a simulated galaxy Disc galaxy decomposition
in Monthly Notices of the Royal Astronomical Society
Brook CB
(2016)
The different baryonic Tully-Fisher relations at low masses.
in Monthly notices of the Royal Astronomical Society
Brown S
(2022)
Towards a universal model for the density profiles of dark matter haloes
in Monthly Notices of the Royal Astronomical Society
Brown S
(2020)
Connecting the structure of dark matter haloes to the primordial power spectrum
in Monthly Notices of the Royal Astronomical Society
Bruni M
(2014)
Computing general-relativistic effects from Newtonian N-body simulations: Frame dragging in the post-Friedmann approach
in Physical Review D
Buie E
(2020)
Interpreting Observations of Absorption Lines in the Circumgalactic Medium with a Turbulent Medium
in The Astrophysical Journal
Buividovich P
(2023)
Real-time simulations of quantum spin chains: Density of states and reweighting approaches
in Physical Review B
Buividovich P
(2022)
Quantum chaos in supersymmetric quantum mechanics: An exact diagonalization study
in Physical Review D
Description | High energy grazing collisions of black holes |
Amount | $48,207 (USD) |
Funding ID | NSF-XSEDE Grant No. PHY-090003 |
Organisation | National Science Foundation (NSF) |
Sector | Public |
Country | United States |
Start | 01/2017 |
End | 12/2017 |
Description | NSF XSEDE Grant PHY-090003 |
Amount | $59,225 (USD) |
Organisation | National Science Foundation (NSF) |
Sector | Public |
Country | United States |
Start | 01/2014 |
End | 12/2014 |
Description | Probing fundamental physics with gravity |
Amount | € 1 (EUR) |
Funding ID | RACE grant Tier-0 PPFPWG |
Organisation | European Commission |
Sector | Public |
Country | European Union (EU) |
Start | 12/2016 |
End | 11/2017 |
Description | Intel/SGI/COSMOS Collaboration |
Organisation | Intel Corporation |
Country | United States |
Sector | Private |
PI Contribution | The present relationship is a longstanding collaboration and partnership in high performance computing with SGI since 1997 and with Intel since 2003. On the basis of taking delivery of the largest deployment of Xeon Phi processors in a single-image shared memory system, we have been testing the architecture and system stack while developing codes to exploit this new many-integrated core (MIC) technology. We influenced design of the initial "MG" blade accommodating the Xeon Phi within the SGI UV2, and we have worked with Intel to improve their Xeon Phi offload implementation. |
Collaborator Contribution | Our partner SGI has contributed half the salary GPB£18000 so that COSMOS could employ a parallel programmer, James Briggs. Intel have contributed 50% of an Intel employee, John Pennycook, to assist with development work and to act as a direct link to Intel software engineering teams. SGI has also contributed substantial software consultancy support through Cheng Liao which has led to new scalable SMP applications. |
Impact | There have been many positive outcomes from this collaboration, notably the Xeon Phi acceleration of important COSMOS consortium codes, including Camb (main CMB cosmological parameter estimation code), WALLS (early universe field theory code), MODAL (non-Gaussianity code), etc. A major development has been the implementation of a scalable shared-memory eigensolver capable of decomposing dense matrices up to 1M x 1M; this has important applications for the TROVE pipeline, determining exoplanet spectral lines (and elsewhere). Work from this collaboration was reported at Supercomputing 2013, held in Denver, November 2013. |
Start Year | 2013 |
Description | Intel/SGI/COSMOS Collaboration |
Organisation | Silicon Graphics Inc |
Country | United States |
Sector | Private |
PI Contribution | The present relationship is a longstanding collaboration and partnership in high performance computing with SGI since 1997 and with Intel since 2003. On the basis of taking delivery of the largest deployment of Xeon Phi processors in a single-image shared memory system, we have been testing the architecture and system stack while developing codes to exploit this new many-integrated core (MIC) technology. We influenced design of the initial "MG" blade accommodating the Xeon Phi within the SGI UV2, and we have worked with Intel to improve their Xeon Phi offload implementation. |
Collaborator Contribution | Our partner SGI has contributed half the salary GPB£18000 so that COSMOS could employ a parallel programmer, James Briggs. Intel have contributed 50% of an Intel employee, John Pennycook, to assist with development work and to act as a direct link to Intel software engineering teams. SGI has also contributed substantial software consultancy support through Cheng Liao which has led to new scalable SMP applications. |
Impact | There have been many positive outcomes from this collaboration, notably the Xeon Phi acceleration of important COSMOS consortium codes, including Camb (main CMB cosmological parameter estimation code), WALLS (early universe field theory code), MODAL (non-Gaussianity code), etc. A major development has been the implementation of a scalable shared-memory eigensolver capable of decomposing dense matrices up to 1M x 1M; this has important applications for the TROVE pipeline, determining exoplanet spectral lines (and elsewhere). Work from this collaboration was reported at Supercomputing 2013, held in Denver, November 2013. |
Start Year | 2013 |
Description | 10th Rencontres du Vietnam |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | 10th Rencontres du Vietnam Very High Energy Phenomena in the Universe Quy Nhon, Vietnam, 8 August 2014 Talk title: "Numerical simulations of coalescing binaries" |
Year(s) Of Engagement Activity | 2014 |
Description | 3rd Sant Cugat Forum on Astrophysics |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | 3rd Sant Cugat Forum on Astrophysics Sant Cugat, Spain, 25 April 2014 Talk title: "Gravitational Recoil and Astrophysical impact" |
Year(s) Of Engagement Activity | 2014 |
Description | 6th form talk (Chelmsford) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Schools |
Results and Impact | October 2017: Ulrich Sperhake gave a talk to a 6th form college in Chelmsford. Subject: The direct detection of gravitational waves by LIGO and the insights already obtained from this new window on the Universe as well as the enormous potential for our search for answers to the most profound questions in astrophysics and fundamental physics. |
Year(s) Of Engagement Activity | 2017 |
Description | Comenius |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | John D. Barrow gave four talks on cosmology to Comenius (Netherlands) University group, March and September 2014, with two more to come in November 2014 See above |
Year(s) Of Engagement Activity | 2014 |
Description | Foligno |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | John D. Barrow gave two talks at Foligno Science Festival, Italy, April 2014 See above |
Year(s) Of Engagement Activity | 2014 |
Description | IMO (Cape Town) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | IMO Celebrity Lecture, The Maths of Entire Universes, International Mathematics Olympiad, U Cape Town, 2014 The International Mathematical Olympiad (IMO) is the World Championship Mathematics Competition for High School students and is held annually in a different country. |
Year(s) Of Engagement Activity | 2014 |
Description | IOP Engineering Society (Chelmsford) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Professional Practitioners |
Results and Impact | October 2017: Ulrich Sperhake gave a talk to the IOP Engineering Society in Chelmsford. Subject: The direct detection of gravitational waves by LIGO and the insights already obtained from this new window on the Universe as well as the enormous potential for our search for answers to the most profound questions in astrophysics and fundamental physics. |
Year(s) Of Engagement Activity | 2017 |
Description | Lecture (Isaac Newton Institute) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Annual Correspondents Day Lecture, Isaac Newton Institute, Cambridge, 2014 The lecture was attended by representatives from EPSRC, BBSRC, ESRC, NERC and STFC. The lecture is important in maintaining and increasing the value of the Institute to the entire UK mathematical community |
Year(s) Of Engagement Activity | 2014 |
Description | New Perspectives on Cosmology |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Yi Wang gave a talk entitled "CMB Anomalies in the Light of BICEP2" at the IAS Workshop, "New Perspectives on Cosmology," 23 May 2014 at the Hong Kong University of Science and Technology During last few years we have witnessed rapid developments in cosmology, both in theory and experiments. Especially, with the recent PLANCK data our understanding on the universe is facing even more refinements than ever. Moreover, with the planned number of future experiments on the cosmic microwave background, large scale structure, gravitational waves and so on, our view on cosmology and gravity will be further developed continuously. The aim of the workshop was to promote stimulating informal and intensive discussions among the participants and to initiate possible collaborations. |
Year(s) Of Engagement Activity | 2014 |
URL | http://ias.ust.hk/workshop/cosmology/2014/index.html |
Description | Pordenone |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | John D. Barrow gave a talk and a formal interview at Pordenone Literary Festival, Italy, September 2014 See above |
Year(s) Of Engagement Activity | 2014 |
URL | http://www.oppureblog.com/john-barrow-ci-racconta-cose-linfinito/ |
Description | Rollins |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | John D. Barrow gave a talk at Rollins College Florida, "Origin and Evolution of the Universe," October 2014 See above |
Year(s) Of Engagement Activity | 2014 |
Description | Science Festival 2013 |
Form Of Engagement Activity | Participation in an open day or visit at my research institution |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Public/other audiences |
Results and Impact | Substantial effort was made during the Cambridge Science Festival to publicise the role of the COSMOS supercomputer at the Centre for Mathematical Sciences, where hundreds of members of the general public visited on Saturday 23 March 2013. Group tours were offered every half hour which proved a particular highlight for the Open Day. The role of COSMOS for key science exploitation of the Planck satellite (announced on 21st March with considerable media coverage) was also emphasised. Considerable interest shown by members of the public. |
Year(s) Of Engagement Activity | 2013 |
Description | Vice Chancellor, Cape Town |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | John D. Barrow gave a talk in Vice Chancellor's series, University of Cape Town, "Origin and Evolution of the Universe," July 2014 See above |
Year(s) Of Engagement Activity | 2014 |