diff options
| author |  Axel Huebl <axel.huebl@plasma.ninja>
| 2023-08-28 15:38:08 -0700 |
|---|---|---|
| committer |  GitHub <noreply@github.com>
| 2023-08-28 15:38:08 -0700 |
| commit | 5ebaa976b34e6421b98382268bb0d0b41cce85e8 (patch) | |
| tree | a276f2f4e68a6b9be9a9b420d313c79cc2eae89e /Tools/machines/perlmutter-nersc/perlmutter_gpu.sbatch | |
| parent | a6b73ef9d6565f24ca97f2e9851daa16c2a484a4 (diff) | |
| download | WarpX-5ebaa976b34e6421b98382268bb0d0b41cce85e8.tar.gz WarpX-5ebaa976b34e6421b98382268bb0d0b41cce85e8.tar.zst WarpX-5ebaa976b34e6421b98382268bb0d0b41cce85e8.zip | |
Docs: Perlmutter (NERSC) Update (#4223)
Update slurm changes and pinning.
Diffstat (limited to 'Tools/machines/perlmutter-nersc/perlmutter_gpu.sbatch')
| -rw-r--r-- | Tools/machines/perlmutter-nersc/perlmutter_gpu.sbatch | 10 |
1 files changed, 6 insertions, 4 deletions
diff --git a/Tools/machines/perlmutter-nersc/perlmutter_gpu.sbatch b/Tools/machines/perlmutter-nersc/perlmutter_gpu.sbatch index 8432b2eb9..873fd3035 100644 --- a/Tools/machines/perlmutter-nersc/perlmutter_gpu.sbatch +++ b/Tools/machines/perlmutter-nersc/perlmutter_gpu.sbatch @@ -17,24 +17,26 @@ # A100 80GB (256 nodes) #S BATCH -C gpu&hbm80g #SBATCH --exclusive +# ideally single:1, but NERSC cgroups issue #SBATCH --gpu-bind=none +#SBATCH --ntasks-per-node=4 #SBATCH --gpus-per-node=4 #SBATCH -o WarpX.o%j #SBATCH -e WarpX.e%j # executable & inputs file or python interpreter & PICMI script here EXE=./warpx -INPUTS=inputs_small +INPUTS=inputs # pin to closest NIC to GPU export MPICH_OFI_NIC_POLICY=GPU # threads for OpenMP and threaded compressors per MPI rank +# note: 16 avoids hyperthreading (32 virtual cores, 16 physical) export SRUN_CPUS_PER_TASK=16 -# depends on https://github.com/ECP-WarpX/WarpX/issues/2009 -#GPU_AWARE_MPI="amrex.the_arena_is_managed=0 amrex.use_gpu_aware_mpi=1" -GPU_AWARE_MPI="" +# GPU-aware MPI optimizations +GPU_AWARE_MPI="amrex.use_gpu_aware_mpi=1" # CUDA visible devices are ordered inverse to local task IDs # Reference: nvidia-smi topo -m |