Parallelization
Introduction
MPTRAC combines MPI, OpenMP, and OpenACC. MPI supports two ways to distribute work across ranks:
- Independent simulations: Each rank runs a separate simulation selected from a list of working directories. The ranks do not exchange particles. This is the established MPI ensemble mode.
- Spatial domain decomposition (DD): The ranks share one simulation. Each rank owns a geographical subdomain and exchanges particles with other ranks as they move between subdomains.
OpenMP distributes particle calculations among CPU threads within each rank. OpenACC can run those calculations on a GPU. Either MPI strategy can be combined with OpenMP and OpenACC, so multiple ranks may use the CPUs or GPUs of one or more compute nodes.
Parallelization strategy of MPTRAC
Independent simulations
Build trac with MPI=1 and use DD=0 in the control files. The
directory list passed to trac contains one working directory per
simulation. MPI ranks take entries from this list in round-robin order;
a rank may process several simulations sequentially. Each simulation
has its own initial particles, control parameters, and meteorological
data. No particle exchange takes place between these simulations.
If the executable was also built with DD=1, set the runtime control
parameter DD=0 explicitly for independent simulations. Otherwise,
DD is enabled by default when that executable starts with more than one
MPI rank.
Spatial domain decomposition
Build trac with MPI=1 DD=1 and set DD=1 in the control files.
Divide one simulation into zonal and meridional subdomains using
DD_SUBDOMAINS_ZONAL and DD_SUBDOMAINS_MERIDIONAL. Start exactly one
MPI rank per subdomain, with one working-directory entry per rank in
the directory list. Each rank reads its local meteorological region
with halos and owns the particles currently in that region. Particles
crossing a subdomain boundary are sent to their new owner by MPI.
DD requires at least two subdomains and a latitude/longitude
meteorological grid; radioactive deposition is not supported.
For NetCDF input, MET_NC_READ_MODE=0 (default) lets each DD rank read
its own extended subdomain. MET_NC_READ_MODE=1 instead assigns each
complete horizontal field at one vertical level to one reader rank and
redistributes the local slices with MPI. It uses the same DD windows for
inner and periodic longitude halos, including their corners. The first
implementation supports MPI-DD NetCDF files with dimensions
(time, level, lat, lon) for 3D fields and (time, lat, lon) for surface
fields; it rejects other conventions. Each reader rank temporarily needs
one global float32 horizontal field plus a buffer for the largest local
slice. All ranks keep one local slice. No global 3D field is allocated.
For example, a 40,000 × 20,000 field needs about 3.2 GB for the global
buffer alone. With zonal DD, the present grid geometry expects a periodic
longitude grid without a duplicate 360° endpoint; the duplicate-endpoint
form fails the existing regular-spacing check in both read modes.
The domain-decomposed met input is instrumented with the standard timer
mechanism: TIMER_MET_READ (NetCDF reads), TIMER_MET_PACK (CPU packing),
TIMER_MET_MPI (MPI redistribution), and TIMER_MET_CONVERT (conversion
into the destination buffers) are printed with the regular timer output at
LOGLEV=1, together with the general timers TIMER_GET_MET (file open,
grid, setup, close), TIMER_READ_MET_SURFACE, and
TIMER_READ_MET_NC_LEVELS. These component times
are not synchronized elapsed times and should be compared alongside the
existing total input timer.
DD_EXCHANGE_MODE=0 (the default) uses the sort/push exchange path;
DD_SORT_DT controls periodic sorting and defaults to 1800 s.
DD_EXCHANGE_MODE=1 classifies particles, packs outgoing particles, and
compacts retained particles without sorting. It also removes particles
with invalid subdomain indices. In mode 1, DD_SORT_DT does not trigger
DD exchange sorting. The general SORT_DT independently controls optional
spatial particle sorting in the physics driver for either exchange mode.
Both modes use the same subdomain assignment and MPI particle communication.
With OpenACC, mode 1 performs classification, packing, and local
compaction on the GPU. The active particle and cache arrays remain on
the device; only outgoing particle records are copied to the host for
the current MPI exchange, and received records are copied to the GPU.
The iso_var quantity and the three uvwp cache components follow particles
between ranks. uvwp is included in the MPI particle datatype and restored
to the cache after each exchange. Particle files remain unchanged; uvwp
is an internal state variable.
The Fortran wrapper supports both DD exchange modes when built with
MPI=1 DD=1 and a matching MPI Fortran compiler (MPIFC).
OpenMP and OpenACC within each rank
OpenMP is enabled in MPTRAC builds and parallelizes particle calculations within an MPI rank. Its threads share that rank's meteorological data. Different MPI ranks have separate data, even when they run on the same node. The OpenMP scalability of MPTRAC is discussed by Rößler et al. (2018).
Build with GPU=1 to offload suitable calculations to GPUs through
OpenACC. For example, four ranks on a four-GPU node can run four
independent simulations or four subdomains of one DD simulation. The
choice of ranks, OpenMP threads, and GPUs depends on the problem size
and the target system.
Supercomputers in Jülich
The JUWELS Cluster module has 2271 standard compute nodes, each with 48 CPU cores and 96 GiB of memory. Simultaneous multithreading provides up to 96 hardware threads per node. The JUWELS Booster module has 936 compute nodes, each with 48 CPU cores, 512 GiB of memory, and four NVIDIA A100 GPUs. See the JUWELS hardware configuration for details.
Examples of job scripts and resource requests are available in the JUWELS batch-system documentation.
Alternatively, MPTRAC has also been used on the JURECA-DC system.