DiRAC 2.5 Operations 2017-2020

Lead Research Organisation: University of Leicester
Department Name: Physics and Astronomy

Abstract

Physicists across the astronomy, nuclear and particle physics communities are focussed
on understanding how the Universe works at a very fundamental level. The distance scales
with which they work vary by 50 orders of magnitude from the smallest distances probed
by experiments at the Large Hadron Collider, deep within the atomic
nucleus, to the largest scale galaxy clusters discovered out in space. The Science challenges,
however, are linked through questions such as: How did the Universe begin and how is it evolving?
and What are the fundamental constituents and fabric of the Universe and how do they interact?

Progress requires new astronomical observations and experimental data but also
new theoretical insights. Theoretical understanding comes increasingly from large-scale
computations that allow us to confront the consequences of our theories very accurately
with the data or allow us to interrogate the data in detail to extract information that has
impact on our theories. These computations test the fastest computers that we have and
push the boundaries of technology in this sector. They also provide an excellent
environment for training students in state-of-the-art techniques for code optimisation and
data mining and visualisation.

The DiRAC-2.5 project builds on the success of the DiRAC HPC facility and will provide the resources needed
to support cutting edge research during 2017 in all areas of science supported by STFC.

In addition to the existing DiRAC-2 services, from April 2017 DiRAC-2.5 will provide:

1) A factor 2 increase in the computational power of the DiRAC supercomputer at the University of Durham, which is designed for simulations requiring large amounts of computer memory. The enhanced system will be used to:
(i) simulate the merger of pairs of black holes which generate gravitational waves such as those recently discovered by the LIGO consortium;
(ii) perform the most realistic simulations to date of the formation and evolution of galaxies in the Universe
(iii) carry out detailed simulations of the interior of the sun and of planetary interiors.

2) A new High Performance Computer at Cambridge whose particular architecture is well suited to the theoretical
problems that we want to tackle that utilise large amounts of data, either as input or
being generated at intermediate stages of our calculations. Two key challenges
that we will tackle are those of
(i) improving our understanding of the Milky Way through analysis of new data from the European
Space Agency's GAIA satellite and
(ii) improving the potential of experiments at CERN's Large Hadron Collider for discovery
of new physics by increasing the accuracy of theoretical predictions for rare processes involving the
fundamental constituents of matter known as quarks.

3) An additional 3500 compute cores on the DiRAC Complexity supercomputer at Leicester which will make it possible to
carry out simulations of some of the most complex physical situation in the Universe. These include:
(i) the formation of stars in clusters - for the first time it will be possible to follow the formation of stars many times more massive than the sun;
(ii) the accretion of gas onto supermassive black holes, the most efficient means of extracting energy from matter and the engine
which drives galaxy formation and evolution.

4) A team of three research software engineers who will help DiRAC researchers to ensure their scientific codes to extract
the best possible performance from the hardware components of the DiRAC clusters. These highly skilled programmers will
increase the effective computational power of the DiRAC facility during 2017.

Planned Impact

The expected impact of the DiRAC 2.5 HPC facility is fully described in the attached pathways to impact document and includes:

1) Disseminating best practice in High Performance Computing software engineering throughout the theoretical Particle Physics, Astronomy and Nuclear physics communities in the UK as well as to industry partners.

2) Working on co-design projects with industry partners to improve future generations of hardware and software.

3) Development of new techniques in the area of High Performance Data Analytics which will benefit industry partners and researchers in other fields such as biomedicine, biology, engineering, economics and social science, and the natural environment who can use this new technology to improve research outcomes in their areas.

4) Share best practice on the design and operation of distributed HPC facilities with UK National e-Infrastructure partners.

5) Training of the next generation of research scientists of physical scientists to tackle problems effectively on state-of-the-art of High Performance Computing facilities. Such skills are much in demand from high-tech industry.

6) Engagement with the general public to promote interest in science, and to explain how our ability to solve complex problems using the latest computer technology leads to new scientific capabilities/insights. Engagement of this kind also naturally encourages the uptake of STEM subjects in schools.

Publications

10 25 50
 
Description Many new discoveries about the formation and evolution of galaxies, star formation and planet formation have been made possible by this award.
Exploitation Route Many international collaborative projects are supported by the HPC resources provided by DiRAC.
Sectors Aerospace

Defence and Marine

Creative Economy

Digital/Communication/Information Technologies (including Software)

Education

Healthcare

Transport

URL http://www.dirac.ac.uk
 
Title Bayesian evidence for the tensor-to-scalar ratio r and neutrino masses m_nu: Effects of uniform vs logarithmic priors (supplementary inference products) 
Description These are the nested sampling inference products and input files that were used to compute results for arXiv:2102.11511. Example plotting scripts (as .ipynb or as .html files) and figures from the papers are included to demonstrate usage.   Filename conventions: lcdm: Concordance cosmological model called \(\Lambda\mathrm{CDM}\) (without extension this assumes \(r=0\) and a single massive neutrino with mass \(m_\nu=0.06\,\mathrm{eV}\)). _r: \(\Lambda\mathrm{CDM}\) with variable tensor-to-scalar ratio \(r\). _nu: \(\Lambda\mathrm{CDM}\) with three massive neutrinos, sampling over the lightest neutrino mass \(m_\mathrm{light}\) and the squared mass splittings \(\delta m^2\) and \(\Delta m^2\). mcmc: Cobaya's Markov Chain Monte Carlo Metropolis sampler.https://github.com/CobayaSampler/cobaya/releases/tag/v3.0.2 pc#d###: PolyChord run with #d repeats per parameter block (where d is the number of parameters in that block) and with ### live points.https://github.com/PolyChord/PolyChordLite/releases/tag/1.17.1 _class: theory code CLASS.https://github.com/lesgourg/class_public/releases/tag/v2.9.4 _p18_TTTEEElowTE_SZ: Planck 2018 TT,TE,EE+lowl+lowE data.https://pla.esac.esa.int/pla/#cosmology _nufit50: NuFIT 5.0 data.http://www.nu-fit.org/?q=node/228 _NH and _IH: normal and inverted neutrino hierarchy. _logr##: logarithmic sampling of tensor-to-scalar ratio \(r\) with lower log bound given .by log10r=-##. _mdD: sampling over the lightest neutrino mass \(m_\mathrm{light}\) and the squared mass splittings \(\delta m^2\) and \(\Delta m^2\) (medium and heavy neutrino mass are derived parameters) with mass units in eV. _logmdD##: logarithmic (instead of uniform) sampling of the lightest neutrino mass \(m_\mathrm{light}\) with lower log bound given by log10mlight=-##.   Datasets used for the nested sampling runs: Planck 2018 TT,TE,EE+lowl+lowE: https://pla.esac.esa.int/pla/#cosmology NuFIT 5.0: http://www.nu-fit.org/?q=node/228   Software used: Cobaya: https://github.com/CobayaSampler/cobaya/releases/tag/v3.0.2 CLASS: https://github.com/lesgourg/class_public/releases/tag/v2.9.4 PolyChord: https://github.com/PolyChord/PolyChordLite/releases/tag/1.17.1 Anesthetic: https://github.com/lukashergt/anesthetic/tree/138299739544e888cc318746be087c898f1aff15   For more details see Cobaya's (https://cobaya.readthedocs.io/en/latest/index.html) and Anesthetic's (https://anesthetic.readthedocs.io/en/latest/) documentation. 
Type Of Material Database/Collection of data 
Year Produced 2021 
Provided To Others? Yes  
URL https://zenodo.org/doi/10.5281/zenodo.4556359
 
Title Dataset for "A steeply-inclined trajectory for the Chicxulub impact" 
Description Data files for 5 timesteps from each simulation. File name convention is A_v_t.npz where time is in seconds (or the string "final"). Each file contains several cell-based fields (pressure, temperature, specific internal energy, density), tracer fields (peak tracer pressure, x,y,z locations) and grid information (nodal and cell-centred coordinates). For an example of how to access all that information, see the "Timestep" class at the top of the "plot_frame.py" python script. Python script "plot_frame.py" will create a figure similar to the panels in Figures 2 and 3 in the paper. Use the flags -a, -V and -t to set the desired impact angle, impact velocity and time. iSALE3D input files for the 8 simulations can be found in inputfiles.tgz Postprocessing python scripts can be found in postprocessing.tgz 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? Yes  
URL https://zenodo.org/record/3667832
 
Title First-order phase transitions in Yang-Mills theories and the density of state method---data and analysis code release 
Description Data release for paper: Lucini, B., Mason, D., Piai, M., Rinaldi, E., & Vadacchino, D. (2023). First-order phase transitions in Yang-Mills theories and the density of state method. arXiv preprint arXiv:2305.07463. This data release comprises of: Importance sampling results: Input and output files for PureGauge file of HiRep (https://github.com/claudiopica/HiRep) and csv files contains analysis of results. LLR results: Input files for LLR_HB for a modified version of HiRep for the heat bath LLR algorthim with umbrella sampling (https://github.com/dave452/Hirep-LLR-SU) and some csv files containing analysis of output. Analysis code within the LLRAnalysis.zip, it contains the code and the conda environment. 
Type Of Material Database/Collection of data 
Year Produced 2023 
Provided To Others? Yes  
URL https://zenodo.org/record/8124749
 
Title First-order phase transitions in Yang-Mills theories and the density of state method---data and analysis code release 
Description Data release for paper: Lucini, B., Mason, D., Piai, M., Rinaldi, E., & Vadacchino, D. (2023). First-order phase transitions in Yang-Mills theories and the density of state method. arXiv preprint arXiv:2305.07463. This data release comprises of: Importance sampling results: Input and output files for PureGauge file of HiRep (https://github.com/claudiopica/HiRep) and csv files contains analysis of results. LLR results: Input files for LLR_HB for a modified version of HiRep for the heat bath LLR algorthim with umbrella sampling (https://github.com/dave452/Hirep-LLR-SU) and some csv files containing analysis of output. Analysis code within the LLRAnalysis.zip, it contains the code and the conda environment. 
Type Of Material Database/Collection of data 
Year Produced 2023 
Provided To Others? Yes  
URL https://zenodo.org/record/8124748
 
Title Flyby-induced misalignments in planet-hosting discs 
Description Data to recreate the published plots/replicate most results. Codes used are publicly available or on request (Phantom). Specifically, files included: - analysis files for Figure 1- relative misalignment files for all simulations (Figure 2 and 3)- relative misalignment files for Figure 4- simulation dump files for Figure 5- analysis files for Figure A1- initial simulation dump files for all 10 simulations (listed in Table 1)- python scripts to make all figures- reference *.log and *.in file from simulation R1 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? Yes  
URL https://leicester.figshare.com/articles/dataset/Flyby-induced_misalignments_in_planet-hosting_discs/...
 
Title Flyby-induced misalignments in planet-hosting discs 
Description Data to recreate the published plots/replicate most results. Codes used are publicly available or on request (Phantom). Specifically, files included: - analysis files for Figure 1- relative misalignment files for all simulations (Figure 2 and 3)- relative misalignment files for Figure 4- simulation dump files for Figure 5- analysis files for Figure A1- initial simulation dump files for all 10 simulations (listed in Table 1)- python scripts to make all figures- reference *.log and *.in file from simulation R1 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? Yes  
URL https://leicester.figshare.com/articles/dataset/Flyby-induced_misalignments_in_planet-hosting_discs/...
 
Title Inference products for "Finite inflation in curved space" 
Description These are the MCMC and nested sampling inference products and input files that were used to compute results for the paper "Finite inflation in cuved space" by L. T. Hergt, F. J. Agocs, W. J. Handley, M. P. Hobson, and A. N. Lasenby from 2022. Example plotting scripts (as \(\texttt{.ipynb}\) or as \(\texttt{.html}\) files) and figures from the paper are included to demonstrate usage.   We used the following python packages for the genertion of MCMC and nested sampling chains: Package Version anesthetic 2.0.0b12 classy 2.9.4 cobaya 3.0.4 GetDist 1.3.3 primpy 2.3.6 pyoscode 1.0.4 pypolychord 1.20.0                     Filename conventions: \(\texttt{mcmc}\): MCMC run \(\texttt{pcs#d####}\): PolyChord run (in synchronous mode) with \(\texttt{#d}\) repeats per parameter block (where \(\texttt{d}\) is the number of parameters in that block) and with \(\texttt{####}\) live points. \(\texttt{_cl_hf}\): Using Boltzmann theory code CLASS with nonlinearities code halofit. \(\texttt{_p18}\): Using Planck 2018 CMB data. \(\texttt{_TTTEEE}\): Using the high-l TTTEEE likelihood. \(\texttt{_TTTEEElite}\): Using the lite version of the high-l TTTEEE likelihood. \(\texttt{_lowl_lowE}\): Using the low-l likelihoods for temperature and E-modes. \(\texttt{_BK15}\): Using data from the 2015 observing season of Bicep2 and the Keck Array. \(\texttt{lcdm}\): Concordance cosmological model called LCDM (standard 6 cosmological sampling parameters, no tensor perturbations, zero spatial curvature) \(\texttt{_r}\): Extension with a variable tensor-to-scalar ratio \(r\). \(\texttt{_omegak}\): Extension with a variable curvature density parameter \(\Omega_K \). \(\texttt{_H0}\): Sampling over \(H_0\) instead of \(\theta_\mathrm{s}\). \(\texttt{_omegakh2}\): Extension with a variable curvature density parameter, but sampling over \(H_0\) instead of \(\theta_\mathrm{s}\) and over \(\omega_K\equiv\Omega_Kh^2\) instead of \(\Omega_K \). \(\texttt{_mn2}\): Using a quadratic monomial potential for the computation of the primordial universe. \(\texttt{_nat}\): Using the natural inflation potential for the computation of the primordial universe. \(\texttt{_stb}\): Using the Starobinsky potential for the computation of the primordial universe. \(\texttt{_AsfoH}\): Using the primordial sampling parameters {`logA_SR`, `N_star`, `f_i`, `omega_K`, `H0`}. \(\texttt{_perm}\): Assuming a permissive reheating scenario. 
Type Of Material Database/Collection of data 
Year Produced 2022 
Provided To Others? Yes  
URL https://zenodo.org/doi/10.5281/zenodo.6547871
 
Title MWC 480 ALMA image 
Description VizieR online Data Catalogue associated with article published in journal Astronomy & Astrophysics with title 'Ring structure in the MWC 480 disk revealed by ALMA.' (bibcode: 2019A&A...622A..75L) 
Type Of Material Database/Collection of data 
Year Produced 2019 
Provided To Others? Yes  
URL https://cdsarc.cds.unistra.fr/viz-bin/cat/J/A+A/622/A75
 
Title Rocking shadows in broken circumbinary discs 
Description Contains reference data and scripts for Nealon et al. 2020b. Specifically: - Jupyter notebooks to make all figures in the paper as well as movie (https://youtu.be/jGYLuEx-I78) - First dumpfile for simulation, *.in file and first *.log file for reference - Dumpfiles to make Figure 1 - Analysis outputs to make Figure 2 - Script to make schematic visualisation used in talks 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? Yes  
URL https://leicester.figshare.com/articles/dataset/Rocking_shadows_in_broken_circumbinary_discs/1341740...
 
Title Rocking shadows in broken circumbinary discs 
Description Contains reference data and scripts for Nealon et al. 2020b. Specifically: - Jupyter notebooks to make all figures in the paper as well as movie (https://youtu.be/jGYLuEx-I78) - First dumpfile for simulation, *.in file and first *.log file for reference - Dumpfiles to make Figure 1 - Analysis outputs to make Figure 2 - Script to make schematic visualisation used in talks 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? Yes  
URL https://leicester.figshare.com/articles/dataset/Rocking_shadows_in_broken_circumbinary_discs/1341740...
 
Title Scattered light shadows in warped protoplanetary discs 
Description Data to recreate the published plots/replicate most results. Codes used are publicly available or on request (Phantom and MCFOST). Specifically, files included: - simulation dump file for Figure 1- fits file for Figure 2 + python script to generate- fits files and analysis script to generate Figure 3- script with fitting routine for Figure 4- fits files in order to generate Figure 5 (using previous fitting routine)- script and fits file to generate Figure 6- analysis routines to generate Figure 7 and 8- Mathematica notebook to generate curves in Figure 9 
Type Of Material Database/Collection of data 
Year Produced 2021 
Provided To Others? Yes  
URL https://leicester.figshare.com/articles/dataset/Scattered_light_shadows_in_warped_protoplanetary_dis...
 
Title Scattered light shadows in warped protoplanetary discs 
Description Data to recreate the published plots/replicate most results. Codes used are publicly available or on request (Phantom and MCFOST). Specifically, files included: - simulation dump file for Figure 1- fits file for Figure 2 + python script to generate- fits files and analysis script to generate Figure 3- script with fitting routine for Figure 4- fits files in order to generate Figure 5 (using previous fitting routine)- script and fits file to generate Figure 6- analysis routines to generate Figure 7 and 8- Mathematica notebook to generate curves in Figure 9 
Type Of Material Database/Collection of data 
Year Produced 2021 
Provided To Others? Yes  
URL https://leicester.figshare.com/articles/dataset/Scattered_light_shadows_in_warped_protoplanetary_dis...
 
Title Three-dimensional magnetic reconnection in particle-in-cell simulations of anisotropic plasma turbulence (Simulation Data) 
Description This folder contains the output of the following simulation: We use the explicit Plasma Simulation Code (PSC, Germaschewski et al.2016) to simulate eight anisotropic counter-propagating Alfvén waves in an ion-electron plasma. The anisotropy of the initial fluctuation is set up according to the theory of critical balance by Sridhar & Goldreich (1994) and Goldreich & Sridhar (1995) at the small scale end of the inertial range: \(k_{\parallel} d_{i} = C (|k_{\perp}|d_{i})^{2/3}\), where \(C= 10^{-4/3}\). The normalization parameters are the speed of light \(c = 1\), the vacuum permittivity \(\epsilon_{0} = 1\), the magnetic permeability \(\mu_{0} = 1\), the Boltzmann constant \(k_{b}=1\), the elementary charge \(q=1\), the ion mass \(m_{i}=1\), the density of ions and electrons \(n_{i}=n_{e}=1\) and the ion inertial length \(d_{i}=c/\omega_{pi}\) where \(\omega_{pi}=\sqrt{n_{i}q^{2}/m_{i}\epsilon_{0}}\) is the ion plasma frequency. We set \(\beta_{s,\parallel}=1\) and \(T_{s,\parallel}/T_{s,\perp}=1\), where \(\beta_{s,\parallel}=2 n_s \mu_{0} k_{B}T_{s,\parallel}/B_{0}^{2}\) is the ratio between the plasma pressure parallel to the background magnetic field \(\mathbf{B}_{0}\) and the magnetic pressure and $T_{s,\parallel}$ is the parallel temperature. The magnetic field is normalised to \(B_{0}=V_{A}/c\), where \(V_{A}=B_{0} / \sqrt{\mu_{0}n_{i}m_{i}}\) is the ion Alfvén speed. We use 100 particles per cell (100 ions and 100 electrons), a mass ratio of \(m_{i}/m_{e} = 100\) so that \(d_e = 0.1 d_{i}\) where \(m_{e}\) is the electron mass and \(d_{e}\) is the electron inertial length. The simulation box size is \(L_{x} \times L_{y} \times L_{z} = 24d_{i}\times24d_{i}\times125d_{i}\) and the spatial resolution is \(\Delta x =\Delta y = \Delta z = 0.06d_{i}\). We use a time step \(\Delta t =0.06/ \omega_{pi}\). In our normalisation, the Debye length \(\lambda_{D}=d_{i}\sqrt{\beta_{i}/2}V_{A}/c\) defines the minimum spatial distance that needs to be resolve in the simulation and \(\lambda_D=0.07d_i\). This output corresponds to \(t=120 \omega_{pi}\). These data were produced using the Data Intensive at Leicester (DIaL) facility provided by the DiRAC project dp126 "Identifying and Quantifying the Role of Magnetic Reconnection in Space Plasma Turbulence". 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? Yes  
URL https://zenodo.org/record/4313309
 
Title First-order phase transitions in Yang-Mills theories and the density of state method --- HiRep LLR Code v1.0.0 
Description The is repository contains an old version of HiRep (https://github.com/claudiopica/HiRep) modified for the heatbath LLR algorthim. A version of this based on a newer version of HiRep is in progress. This repository contains the code for the LLR method used in the paper: Lucini, B., Mason, D., Piai, M., Rinaldi, E., & Vadacchino, D. (2023). First-order phase transitions in Yang-Mills theories and the density of state method. arXiv preprint arXiv:2305.07463. 
Type Of Technology Software 
Year Produced 2023 
URL https://zenodo.org/record/8134755