ECT conformal solver (#1923)

* adding base implementation of the conformal solver

* adding some preprocessor directives

* qualifying the isnan's correctly

* getting rid of some unused fields

* computing S_stab on the fly

* using empty in length check

* Source/FieldSolver/FiniteDifferenceSolver/EvolveB.cpp

* replaced a looponcpu with a parallelfor

* trying to not pass lending/borrowing info by reference in evolveB

* passing data using dataPtr

* converting inds into a device vector

* simplifying some +=

* extracting the inds BaseFabs as DataPtr

* Revert "extracting the inds BaseFabs as DataPtr"

This reverts commit bc709d2fcaa347c119514de79a3f57169a05af65.

* implementing new way ogf handling cell extensions (gpu compatible)

* fixing some white spaces

* removed a typo

* made a function static

* tidying up

* tidying up

* getting rid of the rho vector

* moving the rho update to evolveE

* refectoring the cell extension

* small chenge in input parameters

* updating WarpX.H

* bug fix

* fixing another bug

* rotating the cube

* updated gitignore

* using the conformal soler in the cube test

* fixing the signature of a function

* trying to fix the setVal problem

* Revert "trying to fix the setVal problem"

This reverts commit c7d0e5e3709b730ff570183b2a6df5f587ca4640.

* trying to fix the setVal problem

* cleaning some comments

* removing an abort in device code

* Including geometric information in the external field initializer

* cleaned up the test

* moving the geometric data initialization before the external fields initialization

* changing way of saving info about intruded cells

* fixed some white spaces

* Using S_mod instead of S_red and S_enl

* substituting a std::vector with amrex::Array1D

* bug fix in the uint8_t -> coords maps

* Condensed ect cell info into one single MultiFab

* fixing some includes

* fixing some more includes

* fixed a typo

* making some functions available in gpu code

* using tilebox instead of validbox for the geometry initialization

* communicating the geometric info in the guard cells

* fixing the guard cells initialization for ect

* fixing an unused parameter

* fixing an unused parameter

* ignored some unused variables when EB is not supported

* cleaning up

* cleaning up

* ignored some unused variables when EB is not supported

* evolving rho just for ect

* handling some more unused variables

* removing some white spaces

* adding the rotated cube test

* removed some white spaces

* change cells into faces

* small fix in unused parameters

* fixed a comment

* for safety for now just initialize the geometric data when lev==maxLevel

* adding some preprocessor directives to isolate EB code

* updating Makefile stuff

* Improving the rotated cube test

* bug fix in mesh refinement

* fixing boundary conditions in rotated cube test

* deactivating MPI in rotated cube test

* activating mpi in test

* improving outputs of cells extension

* improved the documentation

* Update Source/EmbeddedBoundary/WarpXInitEB.cpp

Co-authored-by: Weiqun Zhang <WeiqunZhang@lbl.gov>

* Update Source/EmbeddedBoundary/WarpXInitEB.cpp

Co-authored-by: Weiqun Zhang <WeiqunZhang@lbl.gov>

* Removed some obsolete isnan's

* undone change to gitignore

* adding some missing AMREX_GPU_DEVICEs

* Changing some amrex::Real(0) into 0.

Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com>

* Making some variables const

Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com>

* Adding amrex:: to some ignore_unused

Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com>

* Enhancing readability of some parallelfor's

Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com>

* Removed some unecessary includes

* Fixing some tags

Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com>

* Fixing the function CountExtFaces

* added a comment for Rhofield

* Fixed a typo in CountExtCells

* trying to remove accesses to private members in ComputeEdgeLengths

* making some functions public

* undoing some useless changes

* undoing some useless changes

* adding some AMREX_GPU_DEVICEs

* adding some unused variables

* adding some AMREX_GPU_DEVICEs to fix warnings

* fixing relative error

* Making several variables const

Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com>

* Fixed a comment

Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com>

* changing some double to amrex::Real

* removing commented lines

* changing some double to amrex::Real

* removing commented lines

* inizialing nelems_x,y,z to avoid a warning

* Fixing number of cells

* Adding missing analysis routine in rotated cube test

Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com>

* Making some variables constexpr

Co-authored-by: Weiqun Zhang <WeiqunZhang@lbl.gov>

* Improving some reduce operations

Co-authored-by: Weiqun Zhang <WeiqunZhang@lbl.gov>

* reading time directly from the output

* fixed a few data types

* fixing another int

* Replacing or->||, and->&&, not->!

* Adding license info

* Adding a white space

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* removed some unused imports

* Moving the doxygen documentation

* Including AMReX_LayoutData.H

* removing some useless parameters

* adding ect solver to the doc

* Removing some useless reductions

* Small change for consistency

* Handling the resizing of arrays

* Handling correctly the resing of arrays

* Fixing some whitespaces

* Fixing an indentation

* Improving a comment

* Removed a useless initialization

* Renamed Rhofield to ECTRhofield

* Renamed Rhofield to ECTRhofield

* Added some if conditions to isolate some EB related code

* Improved some comments

Co-authored-by: Remi Lehe <remi.lehe@normalesup.org>

* Made some functions not member of WarpX anymore

* Isolated some EB-only code

* Isolated some EB-only code

* Using the square brackets to access vectors

* Ignoring some unused variables

* Bug fix in initialization

* Removed a redundant if

* Using enumeration for connectivity and improved names

* Added a comment

* added a few comments

* Removed some useless template parameters

* Printing some info only in verbose mode

* Revert "Removed some useless template parameters"

This reverts commit 2c527980e6872c0212767c27f00e2b53ecbcfd0a.

* Fixed a bug with the neighbours enumeration

* Initializing geom data alsoo in the ghost cells

Co-authored-by: Weiqun Zhang <WeiqunZhang@lbl.gov>
Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com>
Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
Co-authored-by: Remi Lehe <remi.lehe@normalesup.org>

1 files changed, 21 insertions, 4 deletions

diff --git a/Source/FieldSolver/FiniteDifferenceSolver/EvolveE.cpp b/Source/FieldSolver/FiniteDifferenceSolver/EvolveE.cpp
index 241992539..ac19984cd 100644
--- a/Source/FieldSolver/FiniteDifferenceSolver/EvolveE.cpp
+++ b/Source/FieldSolver/FiniteDifferenceSolver/EvolveE.cpp
@@ -49,11 +49,21 @@ void FiniteDifferenceSolver::EvolveE (
     std::array< std::unique_ptr<amrex::MultiFab>, 3 > const& Bfield,
     std::array< std::unique_ptr<amrex::MultiFab>, 3 > const& Jfield,
     std::array< std::unique_ptr<amrex::MultiFab>, 3 > const& edge_lengths,
+    std::array< std::unique_ptr<amrex::MultiFab>, 3 > const& face_areas,
+    std::array< std::unique_ptr<amrex::MultiFab>, 3 >& ECTRhofield,
     std::unique_ptr<amrex::MultiFab> const& Ffield,
     int lev, amrex::Real const dt ) {
 
-   // Select algorithm (The choice of algorithm is a runtime option,
-   // but we compile code for each algorithm, using templates)
+#ifdef AMREX_USE_EB
+    if (m_fdtd_algo != MaxwellSolverAlgo::ECT) {
+        amrex::ignore_unused(face_areas, ECTRhofield);
+    }
+#else
+    amrex::ignore_unused(face_areas, ECTRhofield);
+#endif
+
+    // Select algorithm (The choice of algorithm is a runtime option,
+    // but we compile code for each algorithm, using templates)
 #ifdef WARPX_DIM_RZ
     if (m_fdtd_algo == MaxwellSolverAlgo::Yee){
         ignore_unused(edge_lengths);
@@ -63,10 +73,14 @@ void FiniteDifferenceSolver::EvolveE (
 
         EvolveECartesian <CartesianNodalAlgorithm> ( Efield, Bfield, Jfield, edge_lengths, Ffield, lev, dt );
 
-    } else if (m_fdtd_algo == MaxwellSolverAlgo::Yee) {
+    } else if (m_fdtd_algo == MaxwellSolverAlgo::Yee || m_fdtd_algo == MaxwellSolverAlgo::ECT) {
 
         EvolveECartesian <CartesianYeeAlgorithm> ( Efield, Bfield, Jfield, edge_lengths, Ffield, lev, dt );
-
+#ifdef AMREX_USE_EB
+        if (m_fdtd_algo == MaxwellSolverAlgo::ECT) {
+            EvolveRhoCartesianECT(Efield, edge_lengths, face_areas, ECTRhofield, lev);
+        }
+#endif
     } else if (m_fdtd_algo == MaxwellSolverAlgo::CKC) {
 
         EvolveECartesian <CartesianCKCAlgorithm> ( Efield, Bfield, Jfield, edge_lengths, Ffield, lev, dt );
@@ -157,13 +171,16 @@ void FiniteDifferenceSolver::EvolveECartesian (
                 // Skip field push if this cell is fully covered by embedded boundaries
                 if (ly(i,j,k) <= 0) return;
 #endif
+
                 Ey(i, j, k) += c2 * dt * (
                     - T_Algo::DownwardDx(Bz, coefs_x, n_coefs_x, i, j, k)
                     + T_Algo::DownwardDz(Bx, coefs_z, n_coefs_z, i, j, k)
                     - PhysConst::mu0 * jy(i, j, k) );
+
             },
 
             [=] AMREX_GPU_DEVICE (int i, int j, int k){
+
 #ifdef AMREX_USE_EB
                 // Skip field push if this cell is fully covered by embedded boundaries
                 if (lz(i,j,k) <= 0) return;