From 88a14b8e251a95c2dec56cee77b60b1a5a31d80d Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Thu, 1 Oct 2020 09:23:05 -0700 Subject: Doc: LLNL Setups (#1394) * [Draft] Doc: Lassen (LLNL) Document installation and usage on Lassen (LLNL). * [Draft] Doc: Quartz (LLNL) Document installation and usage on Quartz (LLNL). --- Docs/source/running_cpp/platforms.rst | 58 +++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) (limited to 'Docs/source/running_cpp') diff --git a/Docs/source/running_cpp/platforms.rst b/Docs/source/running_cpp/platforms.rst index 2a3ead381..925eacccd 100644 --- a/Docs/source/running_cpp/platforms.rst +++ b/Docs/source/running_cpp/platforms.rst @@ -131,3 +131,61 @@ parameters provided good performance: * **Sixteen `64x64x64` grids per MPI rank** (with default tiling in WarpX, this results in ~49 tiles per OpenMP thread) + +.. _running-cpp-lassen: + +Running on Lassen at LLNL +------------------------- + +.. _running-cpp-lassen-V100-GPUs: + +V100 GPUs +^^^^^^^^^ + +The batch script below can be used to run a WarpX simulation on 2 nodes on the supercomputer Lassen at LLNL. +Replace descriptions between chevrons ``<>`` by relevant values, for instance ```` could be ``plasma_mirror_inputs``. +Note that the only option so far is to run with one MPI rank per GPU. + +.. literalinclude:: ../../../Tools/BatchScripts/batch_lassen.sh + :language: bash + +To run a simulation, copy the lines above to a file ``batch_lassen.sh`` and run +:: + + bsub batch_lassen.sh + +to submit the job. + +For a 3D simulation with a few (1-4) particles per cell using FDTD Maxwell +solver on V100 GPUs for a well load-balanced problem (in our case laser +wakefield acceleration simulation in a boosted frame in the quasi-linear +regime), the following set of parameters provided good performance: + +* ``amr.max_grid_size=256`` and ``amr.blocking_factor=128``. + +* **One MPI rank per GPU** (e.g., 4 MPI ranks for the 4 GPUs on each Lassen + node) + +* **Two `128x128x128` grids per GPU**, or **one `128x128x256` grid per GPU**. + + +Running on Quartz at LLNL +------------------------- + +.. _running-cpp-quartz-CPUs: + +Intel Xeon E5-2695 v4 CPUs +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The batch script below can be used to run a WarpX simulation on 2 nodes on the supercomputer Quartz at LLNL. +Replace descriptions between chevrons ``<>`` by relevant values, for instance ```` could be ``plasma_mirror_inputs``. + +.. literalinclude:: ../../../Tools/BatchScripts/batch_quartz.sh + :language: bash + +To run a simulation, copy the lines above to a file ``batch_quartz.sh`` and run +:: + + sbatch batch_quartz.sh + +to submit the job. -- cgit v1.2.3