Particle Buffer for Backtransformed Diagnostics (#1898)

* move BTD call before Redistribute

* define particle buffer, BTD particle functor, particle filter

* select particles on the slice

* add particle functors

* add Lorentz Transform

* fix conflict

* add call to loretnz operator

* storing Particles in buffer

* This is WIP. Added particle buffers, filled them, sent them for flush with number already flushed. Does not work for multiple flushes. Crashes for OpenPMD.

* trailing endif after rebase

* adding print statements and not flushing particles in OPENPMD if numpart is 0

* last timestep flush is ensured at the end of evolve loop

* fix bug in declaring uy uy new and computing uzp

* set particle Geom, BA, and DMAP for particle flush with plotfile

* set Particle BA Geom DMAP for particle buffer and no BTD transform for force flush

* separate compute and pack from flush

* Fix Typo: resizeable -> resizable

Fixes HDF5 BTD particle output.

* new class for plotfile particles for BTD

* copy particle_H and DATA and Header. some WIP print statements

* Merge plotfile

* clean print statements

* fix warning message

* struct declaration in header, fix warning

* doxygen comments and copyright

* clean print statements

* fix eol and override function warning

* tile data

* fix output species array size bug

* fix access for particle buffer size

* clean and move time-update

* add cur_time update back

* remove cur time update which was called twice

* dont access particles flushed already for full diagnostics

* cur time must be updated for RigidInjection BTD CI test to pass

* temporarily move call to BTD

* updating time and calling BTD before movewindow

* cleanup

* reset benchmarks and analysis script

* clean and add comments

* fix particle box array, geom, dmap

* reset benchmarks for multi_J rz and ElectrostaticSphereEB_mixedBC

* wip commit

* wip commit

* add SI conversion

* abort for openpmd bp backend if species is selected. Also write particle output for BTD only if write_species is 1

* add documentation for aborting if adios is used with openpmd and add other BTD input parameters

* Apply suggestions from code review

commit Axel's suggestions from review

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* use bool instead of int

* fix doxygen format

* using h5 as backend in example test to ensure consistency with abort for particle output.

* fix doxygen comment

* reset benchmark again for comoving_2d_psatd galilean_2d_psatd multi_J_rz_psatd

* reset benchmark for background_mcc

* self-review suggestions

* reset benchmarks. Update with last snapshot full info

* Axel's PR suggestions

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Axel's doxygen fix

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* add comments

* fix eol

* improved exception handling for stringsteam

* PR suggestions

* Axels' suggestions from code review

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Axel's suggestions :)

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* simplify logic

* suggestions from review (Axel/Reva)

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* variable name change for clarity

* if num particles in tmp array is 0, return

* Use new BTD inputs to set up BTD for particles in the corresponding particle container

* unused var

* fix logic error

* speciesID undefined

* separate particle and field buffer calls and initialization for BTD. Data common to both are initialized separately

* rename variable so it does not ghost existing varname

* add more comments

* Assert that fields are on

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* redundant function definition

* unused variable

* unused variable zp

* 1D 2D 3D definition

* fix else

* Apply suggestions from code review

Add Prabhat's suggestion

Co-authored-by: Prabhat Kumar <89051199+prkkumar@users.noreply.github.com>

* missing semicolon and ignore xp yp for 1D

* resetting benchmarks for boosted sims and mcc sim

* temporarily changing tolerance since the relative difference for momentum_z is 3.68e-3 and the current tolerance is 2.5e-3

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Prabhat Kumar <89051199+prkkumar@users.noreply.github.com>

1 files changed, 8 insertions, 4 deletions

diff --git a/Source/Particles/PhysicalParticleContainer.cpp b/Source/Particles/PhysicalParticleContainer.cpp
index c669d976a..6a001717a 100644
--- a/Source/Particles/PhysicalParticleContainer.cpp
+++ b/Source/Particles/PhysicalParticleContainer.cpp
@@ -1605,7 +1605,8 @@ PhysicalParticleContainer::Evolve (int lev,
 
     bool has_buffer = cEx || cjx;
 
-    if (WarpX::do_back_transformed_diagnostics && do_back_transformed_diagnostics)
+    if ( (WarpX::do_back_transformed_diagnostics && do_back_transformed_diagnostics) ||
+         (m_do_back_transformed_particles) )
     {
         for (WarpXParIter pti(*this, lev); pti.isValid(); ++pti)
         {
@@ -2287,6 +2288,8 @@ PhysicalParticleContainer::GetParticleSlice (
                   uzpold = tmp_particle_data[lev][index][TmpIdx::uzold].dataPtr();
 
                 const long np = pti.numParticles();
+                amrex::Print() << " np old BTD " << np << "\n";
+                amrex::Print() << " tmp particle size : " << tmp_particle_data.size() << "\n";
 
                 Real uzfrm = -WarpX::gamma_boost*WarpX::beta_boost*PhysConst::c;
                 Real inv_c2 = 1.0/PhysConst::c/PhysConst::c;
@@ -2505,9 +2508,10 @@ PhysicalParticleContainer::PushPX (WarpXParIter& pti,
     ParticleReal* const AMREX_RESTRICT uz = attribs[PIdx::uz].dataPtr() + offset;
 
     auto copyAttribs = CopyParticleAttribs(pti, tmp_particle_data, offset);
-    int do_copy = (WarpX::do_back_transformed_diagnostics &&
-                          do_back_transformed_diagnostics &&
-                   (a_dt_type!=DtType::SecondHalf));
+    int do_copy = ( (WarpX::do_back_transformed_diagnostics
+                     && do_back_transformed_diagnostics
+                     && a_dt_type!=DtType::SecondHalf)
+                  || (m_do_back_transformed_particles && (a_dt_type!=DtType::SecondHalf)) );
 
     int* AMREX_RESTRICT ion_lev = nullptr;
     if (do_field_ionization) {