Averaged Galilean PSATD (#869)

* Read Galilean velocity

* Prepare structures for Galilean solver

* Started implementing Galilean equations

* Analytical limits for X1, X2, X3, X4 coefficients added

* Slight changes added

* Added Galilean position pusher

* Scale galilean velocity

* Remove unneeded Abort

* Fix Galilean pusher

* Allocate Theta2 array

* Fix definition of coefficients

* Increase guard cells for Galilean

* Add guard cell in particle exchange

* Added modifications for PICSAR galilean branch

* Averaged coefficients added

* Type corrected

* v_gal added to warpx_current_deposition

* v_gal added to WarpXParticleContainer.H

* Bug fixed  - update particle x-position over one time step

* Fix issues with merge from dev

* Averaged fileds allocated on fine patch (Ex)

* Preparation for merging dev into galilean.

* remove TABs by hand

* Removed a tab.

* Redeclared v_galilean as a vector & related changed

* Added an automated test

* Moved v_galilean inside WarpX constructor

* Added analysis script for the automated test

* Changed name of the automated test to galilean_psatd

* Added InitializeSpectralCoefficients method

* Removed temporary comments

* Averaged fields added to FiledGather

* Added infinite order k[i]

* Setting kx_mod( kz_mod) = 0 for index = nx/2(-nz/2)

* Adding galilean shift

* Implemented galilean shift

* Changed method's name from GalileanShift to ShiftGalileanBoundary

* Added doxygen string for ShiftGalileanBoundary

* Removed never used method LowerCornerWithCentering

* Removed temporary comments

* Removed dt as a variable from DepositCharge method and its dependencies

* Changing type of v_galilean from amrex::Vector to amrex::Array

* Changed back ng_fft to be equal nox_fft

* Changed v_galilean's type from amrex::vector to amrex::array

* Removed type

* Removed temporary comments

* Added flag 'do_time_averaging' to swith from unaveraged to averaged Galilean PSATD

* Updated doxygen

* Small fix on using 'do_time_averaging' flag

* Remove some commented Print statements

* [skip ci] Further cleanup

* Fix compilation

* Guard cells update of the averaged E,B fields

* Corrected included header file accordingly

* Removed EOL

* Removed EOL

* Corrected path of the included header file

* Updated choice of the spectral solver.

* Cleanup.

* use amrex::exp instead of std::exp

* no backward FFT for avg fields if avg is off

* Need to shift avg fields in MoveWindow

* Further cleaning

* Added 2D automated test for averaged PSATD

* Added automated 2D and 3D tests for averaged PSATD

* Removed comments

* Added specifications for averaged Galilean PSATD tests.

* Bug fixed - do FillBoundary of the averaged fields only if averaged is activated

* Do shiftMF of the averaged fields only if fft_do_time_averaging=true

* Add checksum json benchmarks for averaged Galilean PSATD

* Add missing indentation

* Add missing indentation

* Updated automated analysis script

* Updated diags period

* Fixed bug: no backtransform of the averaged fields when WARPX_DIM_RZ is set

* Fixed an uninitialized variable

* Clean-up

* Changed permissions for analysis_avg_*d.py

* Compactified variables initialization via conditional assignment

* Initialized averaged E_avg, B_avg fields

* Updated automated test for 2D averaged galilean PSATD

* Updated benchmark for 2D averaged galilean PSATD

* Updated automated 2D averaged PSATD test & benchmark

* Updated automated 3D averaged PSATD test & benchmark

* Fixed typo

* Updated benchmark averaged_galilean_3d_psatd

* Initialize the averaged fields only if the averaged Galilean PSATD algorithm is enabled

* Encreased time step for automated 3D averaged Galilean PSATD (to be equal c*dt=dz while dz/dx=3)

* Updated 3D automated test: specified standard deviations `.ux_th, .uy_th, .uz_th` along each direction

* Typo from the previous commit: enabled the  averaged algorithm

* Trying out another input parameters for 3D averaged automated test with v_gal = v_plasma

* Update benchmark for previously added 3D averaged automated test

* Update 2D averaged automated test and corresponding benchmark

* Revert "Updated diags period"

This reverts commit 7334729b05e96589e020c981efdb430ca095991d.

* Cleanup: removed unwanted comment.

* Added descriptions of the 2D and 3D automated tests for the averaged Galilean PSATD.

* Updated  value calculated via standard Galilean PSATD.

* Encreased relative error tolerance for 3D automated test for the averaged Galilean PSATD.

* Removed to avoid duplication since it already specified in the corresponding 2D/3D input scripts.

* Removed unwanted empty lines

* Added spaces after function's names

* Removed unwanted empty line

* Removed

* Fixed indentation

* Cleanup: removed #include <math.h>

* Cleanup: removed empty lines and fixed indentation

* Added 'AvgGalileanAlgorithm.cpp' to 'CMakeLists.txt'

Co-authored-by: Remi Lehe <remi.lehe@normalesup.org>
Co-authored-by: MaxThevenet <mthevenet@lbl.gov>

1 files changed, 125 insertions, 0 deletions

diff --git a/Source/Parallelization/WarpXComm.cpp b/Source/Parallelization/WarpXComm.cpp
index 95ed848f1..79bcce2f4 100644
--- a/Source/Parallelization/WarpXComm.cpp
+++ b/Source/Parallelization/WarpXComm.cpp
@@ -353,6 +353,25 @@ WarpX::FillBoundaryF (IntVect ng)
 }
 
 void
+WarpX::FillBoundaryB_avg (IntVect ng, IntVect ng_extra_fine)
+{
+    for (int lev = 0; lev <= finest_level; ++lev)
+    {
+        FillBoundaryB_avg(lev, ng, ng_extra_fine);
+    }
+}
+
+void
+WarpX::FillBoundaryE_avg (IntVect ng, IntVect ng_extra_fine)
+{
+    for (int lev = 0; lev <= finest_level; ++lev)
+    {
+        FillBoundaryE_avg(lev, ng, ng_extra_fine);
+    }
+}
+
+
+void
 WarpX::FillBoundaryE(int lev, IntVect ng, IntVect ng_extra_fine)
 {
     FillBoundaryE(lev, PatchType::fine, ng+ng_extra_fine);
@@ -475,6 +494,112 @@ WarpX::FillBoundaryB (int lev, PatchType patch_type, IntVect ng)
 }
 
 void
+WarpX::FillBoundaryE_avg(int lev, IntVect ng, IntVect ng_extra_fine)
+{
+    FillBoundaryE_avg(lev, PatchType::fine, ng+ng_extra_fine);
+    if (lev > 0) FillBoundaryE_avg(lev, PatchType::coarse, ng);
+}
+
+void
+WarpX::FillBoundaryE_avg (int lev, PatchType patch_type, IntVect ng)
+{
+    if (patch_type == PatchType::fine)
+    {
+        if (do_pml && pml[lev]->ok())
+         {
+            amrex::Abort("Averaged Galilean PSATD with PML is not yet implemented");
+         }
+
+        const auto& period = Geom(lev).periodicity();
+        if ( safe_guard_cells ){
+            Vector<MultiFab*> mf{Efield_avg_fp[lev][0].get(),Efield_avg_fp[lev][1].get(),Efield_avg_fp[lev][2].get()};
+            amrex::FillBoundary(mf, period);
+        } else {
+            AMREX_ALWAYS_ASSERT_WITH_MESSAGE(
+                ng <= Efield_avg_fp[lev][0]->nGrowVect(),
+                "Error: in FillBoundaryE_avg, requested more guard cells than allocated");
+            Efield_avg_fp[lev][0]->FillBoundary(ng, period);
+            Efield_avg_fp[lev][1]->FillBoundary(ng, period);
+            Efield_avg_fp[lev][2]->FillBoundary(ng, period);
+        }
+    }
+    else if (patch_type == PatchType::coarse)
+    {
+        if (do_pml && pml[lev]->ok())
+         {
+            amrex::Abort("Averaged Galilean PSATD with PML is not yet implemented");
+         }
+
+        const auto& cperiod = Geom(lev-1).periodicity();
+        if ( safe_guard_cells ) {
+            Vector<MultiFab*> mf{Efield_avg_cp[lev][0].get(),Efield_avg_cp[lev][1].get(),Efield_avg_cp[lev][2].get()};
+            amrex::FillBoundary(mf, cperiod);
+
+        } else {
+            AMREX_ALWAYS_ASSERT_WITH_MESSAGE(
+                ng <= Efield_avg_cp[lev][0]->nGrowVect(),
+                "Error: in FillBoundaryE, requested more guard cells than allocated");
+            Efield_avg_cp[lev][0]->FillBoundary(ng, cperiod);
+            Efield_avg_cp[lev][1]->FillBoundary(ng, cperiod);
+            Efield_avg_cp[lev][2]->FillBoundary(ng, cperiod);
+        }
+    }
+}
+
+
+void
+WarpX::FillBoundaryB_avg (int lev, IntVect ng, IntVect ng_extra_fine)
+{
+    FillBoundaryB_avg(lev, PatchType::fine, ng + ng_extra_fine);
+    if (lev > 0) FillBoundaryB_avg(lev, PatchType::coarse, ng);
+}
+
+void
+WarpX::FillBoundaryB_avg (int lev, PatchType patch_type, IntVect ng)
+{
+    if (patch_type == PatchType::fine)
+    {
+        if (do_pml && pml[lev]->ok())
+          {
+            amrex::Abort("Averaged Galilean PSATD with PML is not yet implemented");
+          }
+        const auto& period = Geom(lev).periodicity();
+        if ( safe_guard_cells ) {
+            Vector<MultiFab*> mf{Bfield_avg_fp[lev][0].get(),Bfield_avg_fp[lev][1].get(),Bfield_avg_fp[lev][2].get()};
+            amrex::FillBoundary(mf, period);
+        } else {
+            AMREX_ALWAYS_ASSERT_WITH_MESSAGE(
+                ng <= Bfield_fp[lev][0]->nGrowVect(),
+                "Error: in FillBoundaryB, requested more guard cells than allocated");
+            Bfield_avg_fp[lev][0]->FillBoundary(ng, period);
+            Bfield_avg_fp[lev][1]->FillBoundary(ng, period);
+            Bfield_avg_fp[lev][2]->FillBoundary(ng, period);
+        }
+    }
+    else if (patch_type == PatchType::coarse)
+    {
+        if (do_pml && pml[lev]->ok())
+          {
+            amrex::Abort("Averaged Galilean PSATD with PML is not yet implemented");
+          }
+
+        const auto& cperiod = Geom(lev-1).periodicity();
+        if ( safe_guard_cells ){
+            Vector<MultiFab*> mf{Bfield_avg_cp[lev][0].get(),Bfield_avg_cp[lev][1].get(),Bfield_avg_cp[lev][2].get()};
+            amrex::FillBoundary(mf, cperiod);
+        } else {
+            AMREX_ALWAYS_ASSERT_WITH_MESSAGE(
+                ng <= Bfield_avg_cp[lev][0]->nGrowVect(),
+                "Error: in FillBoundaryB_avg, requested more guard cells than allocated");
+            Bfield_avg_cp[lev][0]->FillBoundary(ng, cperiod);
+            Bfield_avg_cp[lev][1]->FillBoundary(ng, cperiod);
+            Bfield_avg_cp[lev][2]->FillBoundary(ng, cperiod);
+        }
+    }
+}
+
+
+void
 WarpX::FillBoundaryF (int lev, IntVect ng)
 {
     FillBoundaryF(lev, PatchType::fine, ng);