Default: abort_on_out_of_gpu_memory = 1 (#1164)

* Default: abort_on_out_of_gpu_memory = 1

Change the default input parameter from AMReX
`amrex.abort_on_out_of_gpu_memory` from false (`0`) to true (`1`).

We set this by default to avoid that users experience super-slow GPU
runs when exceeding GPU memory. In such a case, users should explicitly
set this to option.

In my optinion, this is only an intermediate solution since what we
actually want on out-of-GPU memory events should be:
- finish current simulation step and cause a load balance or
- trigger a checkpoint and shut down cleanly
  - then the user can manually restart with more resources

We want to address the opposite case, user under-utilizes a GPU, with
a warning for now.

Ref.:
- https://amrex-codes.github.io/amrex/docs_html/GPU.html#inputs-parameters

* abort_on_out_of_gpu_memory: review

Add review comments.

Co-authored-by: L. Diana Amorim <LDianaAmorim@lbl.gov>

Co-authored-by: L. Diana Amorim <LDianaAmorim@lbl.gov>

1 files changed, 8 insertions, 2 deletions

diff --git a/Docs/source/running_cpp/parameters.rst b/Docs/source/running_cpp/parameters.rst
index 4859f6034..ff08967d7 100644
--- a/Docs/source/running_cpp/parameters.rst
+++ b/Docs/source/running_cpp/parameters.rst
@@ -45,7 +45,7 @@ Overall simulation parameters
     printed to standard output. Currently only works if the Lorentz boost and
     the moving window are along the z direction.
 
-* ``warpx.verbose`` (`0` or `1`)
+* ``warpx.verbose`` (``0`` or ``1``; default is ``1`` for true)
     Controls how much information is printed to the terminal, when running WarpX.
 
 * ``warpx.random_seed`` (`string` or `int` > 0) optional
@@ -61,7 +61,7 @@ Overall simulation parameters
     one should not expect to obtain the same random numbers,
     even if a fixed ``warpx.random_seed`` is provided.
 
-* ``warpx.do_electrostatic`` (`0` or `1`; default is `0`)
+* ``warpx.do_electrostatic`` (``0`` or ``1``; default is ``0`` for false)
     Run WarpX in electrostatic mode. Instead of updating the fields
     at each iteration with the full Maxwell equations, the fields are
     instead recomputed at each iteration from the (relativistic) Poisson
@@ -69,6 +69,12 @@ Overall simulation parameters
     electromagnetic effects (e.g. propagation of radiation, lasers, etc.)
     are not captured.
 
+* ``amrex.abort_on_out_of_gpu_memory``  (``0`` or ``1``; default is ``1`` for true)
+    When running on GPUs, memory that does not fit on the device will be automatically swapped to host memory when this option is set to ``0``.
+    This will cause severe performance drops.
+    Note that even with this set to ``1`` WarpX will not catch all out-of-memory events yet when operating close to maximum device memory.
+    `Please also see the documentation in AMReX <https://amrex-codes.github.io/amrex/docs_html/GPU.html#inputs-parameters>`_.
+
 .. _running-cpp-parameters-box:
 
 Setting up the field mesh