Merge `FillBoundary` and `NodalSync` (`OneStep_nosub`, `OneStep_sub1`) (#3013)

* Add argument for synchronizing nodal points

* Add the option to overridesync in FillBoundaryE / FillBoundaryB

* Remove explicit synchronization

* Cleanup calls to nodal synchronization

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Improve const Correctness

* Cleaning

* Merge NodalSync and FillBoundary First

Restarting the implementation from scratch.
To-do: remove redundant FillBoundary calls.

* Merge Only for OneStep_nosub and OneStep_sub1

* Fix Checksums of Python_wrappers

* Define sync_nodal_points as WarpX Private Member Variable

* Keep Calls to NodalSync for B

* Keep Calls to NodalSync for F

* Reset Benchmark of reduced_diags_single_precision

* OneStep_sub1: Remove All Calls to NodalSync

* OneStep_nosub: Remove All Calls to NodalSync

* Reset Benchmark of divb_cleaning_3d

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Edoardo Zoni <ezoni@lbl.gov>

1 files changed, 13 insertions, 3 deletions

diff --git a/Source/Parallelization/WarpXCommUtil.cpp b/Source/Parallelization/WarpXCommUtil.cpp
index d395e7628..104aa5aae 100644
--- a/Source/Parallelization/WarpXCommUtil.cpp
+++ b/Source/Parallelization/WarpXCommUtil.cpp
@@ -93,7 +93,8 @@ void FillBoundary (amrex::MultiFab& mf, const amrex::Periodicity& period)
 
 void FillBoundary (amrex::MultiFab&          mf,
                    amrex::IntVect            ng,
-                   const amrex::Periodicity& period)
+                   const amrex::Periodicity& period,
+                   const bool                nodal_sync)
 {
     BL_PROFILE("WarpXCommUtil::FillBoundary");
 
@@ -106,13 +107,22 @@ void FillBoundary (amrex::MultiFab&          mf,
 
         mixedCopy(mf_tmp, mf, 0, 0, mf.nComp(), mf.nGrowVect());
 
-        mf_tmp.FillBoundary(ng, period);
+        if (nodal_sync) {
+            mf_tmp.FillBoundaryAndSync(0, mf.nComp(), ng, period);
+        } else {
+            mf_tmp.FillBoundary(ng, period);
+        }
 
         mixedCopy(mf, mf_tmp, 0, 0, mf.nComp(), mf.nGrowVect());
     }
     else
     {
-        mf.FillBoundary(ng, period);
+
+        if (nodal_sync) {
+            mf.FillBoundaryAndSync(0, mf.nComp(), ng, period);
+        } else {
+            mf.FillBoundary(ng, period);
+        }
     }
 }