Dendro-GR

What is Dendro-GR ?

Dendro-GR is a highly-scalable framework that targets problems of interest to the numerical relativity and broader astrophysics communities. This framework combines a parallel octree-refined adaptive mesh (i.e., Dendro libarary) with a wavelet adaptive multiresolution and a physics module to solve the Einstein equations of general relativity. The goal of this work is to perform advanced, massively parallel numerical simulations of binary black holes with mass ratios on the order of 100:1. These studies will be used to generate waveforms as used in LIGO data analysis and to calibrate semi-analytical approximate methods. Our framework consists of a distributed memory octree-based adaptive meshing framework in conjunction with a node-local code generator. Dendro-GR framework achieve excellent performance and scalability on modern leadership architectures. Dendro-GR also has tested strong scalability up to 8 A100s and weak scaling up to 229,376 x86 cores on the Texas Advanced Computing Center’s Frontera system. To the best of our knowledge, Dendro-GR work is the first highly scalable, adaptive, multi-GPU numerical relativity code performing binary black hole mergers.

Get Dendro-GR

Dendro framework is an open-source scalable octree algorithms suite, designed to solve partial differential equations using Galerkin, finite difference, finite volume discretization methods. If you are interested in using Dendro please let us know how can we help. You can clone the repository using ,

Code generation dependancies

pip3 install –user sympy numpy numba git+https://github.com/moble/quaternion git+https://github.com/moble/spherical_functions cogapp quadpy

How to build Dendro-GR ?

To build Dendro-5.0, you need following externeral packages,

To build the code please use the following commands.

$cd <path to root source dir >
$ mkdir build
$ cd build
$ cmake ../
$ make bssnSolverCtx bssnSolverCUDA tpid -j4

Singularity container

The singularity container definition file is provided in the repository under the folder container. The following command can be used to build the Dendro-GR container which installs all the required dependencies and compile the Dendro-GR code.

sudo singularity build --sandbox dgr-cuda dgr.def
singularity run dgr-cuda dgr.def

Running experiemnts

Binary mergers and GWs

$ ./BSSN_GR/tpid q1.par.json <number of threads to use>
$ mpirun -np <number of GPUs> ./BSSN_GR/bssnSolverCUDA q1.par.json 1

GPU and CPU experiments

Additional scripts files for conducted experiments are presented in the <source dir>/BSSN GR/experiment_scripts/ls6 folder.

Running experiments with Singularity

If you built Dendro-GR with Singularity, the following command can be used to launch the main BSSN solver and other benchmarks.

singularity exec dgr-cuda \
sc22-dgr/build_gpu/BSSN_GR/./bssnSolverCtx sc22-dgr/build_gpu/q1.par.json 1

BSSNOK formulation

Dendro-GR consists of sympy based code generation framework (SympyGR) that supports efficient code generaton for both CPUs and GPUs. You can find the sympy BSSNOK file here. Numerical relativity application can be quite overwhelmed for a newcomer for Dendro-GR, hence we provide a simpler NLSigma computation here. Note that the overall workflow of the computational is ideantial to the solving the GR equation but with much simpler set of equations.

Simple Example: Nonlinear Sigma Model (NLSigma)

NlSigma folder consists of simple, non lineat wave equation with adaptive mesh refinement (AMR). You can copy the parameter file from NLSigma/par folder and simply run mpirun -np 8 ./NLSigma/nlsmSolver nlsm.par.json, on your lattop to large supercomputer with higher resolution.

nlsm nlsm nlsm nlsm

You can write the equations in symbolic python which generate the C compute kernel. Look at nlsm.py

import dendro
from sympy import *
###############################################################
#  initialize
###############################################################
r = symbols('r')
# declare functions
chi = dendro.scalar("chi","[pp]")
phi = dendro.scalar("phi","[pp]")
d = dendro.set_first_derivative('grad')    # first argument is direction
d2s = dendro.set_second_derivative('grad2')  # first 2 arguments are directions
d2 = dendro.d2

###############################################################
#  evolution equations
###############################################################

phi_rhs = sum( d2(i,i,chi)  for i in dendro.e_i ) - sin(2*chi)/r**2
chi_rhs = phi

###############################################################
#  evolution equations
###############################################################
outs = [phi_rhs, chi_rhs]
vnames = ['phi_rhs', 'chi_rhs']
dendro.generate(outs, vnames, '[pp]')

Which generate the code,

// Dendro: }

Parameters for NLSigma

Galerkin approximations (i.e., Finite element approaces):

Dendro supports finite element computations, currently Continous Galerkin (CG) Methods, but in future we will support Discontinous Galerkin (DG) Methods. All FEM related code are in FEM folder, you can write, allplications using the feMatrix.h and feVector.h classes, to represent left and right hand sidesof the variational formulations. oda.h contains the interface class which enables easy integration with petsc matrix based and matrix free methods.

Simple Laplace equation using FEM can be found in FEM/examples/src/heatEq.cpp file.

Publications