Particle Buffer for Backtransformed Diagnostics (#1898)

* move BTD call before Redistribute

* define particle buffer, BTD particle functor, particle filter

* select particles on the slice

* add particle functors

* add Lorentz Transform

* fix conflict

* add call to loretnz operator

* storing Particles in buffer

* This is WIP. Added particle buffers, filled them, sent them for flush with number already flushed. Does not work for multiple flushes. Crashes for OpenPMD.

* trailing endif after rebase

* adding print statements and not flushing particles in OPENPMD if numpart is 0

* last timestep flush is ensured at the end of evolve loop

* fix bug in declaring uy uy new and computing uzp

* set particle Geom, BA, and DMAP for particle flush with plotfile

* set Particle BA Geom DMAP for particle buffer and no BTD transform for force flush

* separate compute and pack from flush

* Fix Typo: resizeable -> resizable

Fixes HDF5 BTD particle output.

* new class for plotfile particles for BTD

* copy particle_H and DATA and Header. some WIP print statements

* Merge plotfile

* clean print statements

* fix warning message

* struct declaration in header, fix warning

* doxygen comments and copyright

* clean print statements

* fix eol and override function warning

* tile data

* fix output species array size bug

* fix access for particle buffer size

* clean and move time-update

* add cur_time update back

* remove cur time update which was called twice

* dont access particles flushed already for full diagnostics

* cur time must be updated for RigidInjection BTD CI test to pass

* temporarily move call to BTD

* updating time and calling BTD before movewindow

* cleanup

* reset benchmarks and analysis script

* clean and add comments

* fix particle box array, geom, dmap

* reset benchmarks for multi_J rz and ElectrostaticSphereEB_mixedBC

* wip commit

* wip commit

* add SI conversion

* abort for openpmd bp backend if species is selected. Also write particle output for BTD only if write_species is 1

* add documentation for aborting if adios is used with openpmd and add other BTD input parameters

* Apply suggestions from code review

commit Axel's suggestions from review

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* use bool instead of int

* fix doxygen format

* using h5 as backend in example test to ensure consistency with abort for particle output.

* fix doxygen comment

* reset benchmark again for comoving_2d_psatd galilean_2d_psatd multi_J_rz_psatd

* reset benchmark for background_mcc

* self-review suggestions

* reset benchmarks. Update with last snapshot full info

* Axel's PR suggestions

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Axel's doxygen fix

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* add comments

* fix eol

* improved exception handling for stringsteam

* PR suggestions

* Axels' suggestions from code review

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Axel's suggestions :)

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* simplify logic

* suggestions from review (Axel/Reva)

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* variable name change for clarity

* if num particles in tmp array is 0, return

* Use new BTD inputs to set up BTD for particles in the corresponding particle container

* unused var

* fix logic error

* speciesID undefined

* separate particle and field buffer calls and initialization for BTD. Data common to both are initialized separately

* rename variable so it does not ghost existing varname

* add more comments

* Assert that fields are on

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* redundant function definition

* unused variable

* unused variable zp

* 1D 2D 3D definition

* fix else

* Apply suggestions from code review

Add Prabhat's suggestion

Co-authored-by: Prabhat Kumar <89051199+prkkumar@users.noreply.github.com>

* missing semicolon and ignore xp yp for 1D

* resetting benchmarks for boosted sims and mcc sim

* temporarily changing tolerance since the relative difference for momentum_z is 3.68e-3 and the current tolerance is 2.5e-3

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Prabhat Kumar <89051199+prkkumar@users.noreply.github.com>

1 files changed, 185 insertions, 0 deletions

diff --git a/Source/Diagnostics/BTD_Plotfile_Header_Impl.cpp b/Source/Diagnostics/BTD_Plotfile_Header_Impl.cpp
index 6b02760c4..bf9f8e877 100644
--- a/Source/Diagnostics/BTD_Plotfile_Header_Impl.cpp
+++ b/Source/Diagnostics/BTD_Plotfile_Header_Impl.cpp
@@ -1,3 +1,9 @@
+/* Copyright 2021 Revathi Jambunathan
+ *
+ * This file is part of WarpX.
+ *
+ * License: BSD-3-Clause-LBNL
+ */
 #include "BTD_Plotfile_Header_Impl.H"
 
 #include "WarpX.H"
@@ -341,3 +347,182 @@ BTDMultiFabHeaderImpl::CopyVec(amrex::Vector<amrex::Real>& dst,
         dst[i] = src[i];
     }
 }
+
+
+BTDSpeciesHeaderImpl::BTDSpeciesHeaderImpl (std::string const & Headerfile_path, std::string const& species_name)
+    : m_Header_path(Headerfile_path), m_species_name(species_name)
+{
+
+}
+
+void
+BTDSpeciesHeaderImpl::ReadHeader ()
+{
+    amrex::Vector<char> HeaderCharPtr;
+    amrex::Long fileLength(0), fileLengthPadded(0);
+    std::ifstream iss;
+
+    iss.open(m_Header_path.c_str(), std::ios::in);
+    iss.seekg(0, std::ios::end);
+    fileLength = static_cast<std::streamoff>(iss.tellg());
+    iss.seekg(0, std::ios::beg);
+
+    fileLengthPadded = fileLength + 1;
+    HeaderCharPtr.resize(fileLengthPadded);
+    iss.read(HeaderCharPtr.dataPtr(), fileLength);
+    iss.close();
+    HeaderCharPtr[fileLength] = '\0';
+
+    std::istringstream is(HeaderCharPtr.dataPtr(), std::istringstream::in);
+    is.exceptions(std::ios_base::failbit | std::ios_base::badbit);
+
+    is >> m_file_version;
+    is >> m_spacedim;
+    is >> m_num_output_real;
+    m_real_comp_names.resize(m_num_output_real);
+    for (int i = 0; i < m_real_comp_names.size(); ++i) {
+        is >> m_real_comp_names[i];
+    }
+    is >> m_num_output_int;
+    for (int i = 0; i < m_int_comp_names.size(); ++i) {
+        is >> m_int_comp_names[i];
+    }
+    is >> m_is_checkpoint;
+    is >> m_total_particles;
+    is >> m_nextid;
+    is >> m_finestLevel;
+    m_particleBoxArray_size.resize(m_finestLevel+1);
+    m_which_data.resize(m_finestLevel+1);
+    m_particles_per_box.resize(m_finestLevel+1);
+    m_offset_per_box.resize(m_finestLevel+1);
+    for (int lev = 0; lev <= m_finestLevel; ++lev) {
+        is >> m_particleBoxArray_size[lev];
+        m_which_data[lev].resize(m_particleBoxArray_size[lev]);
+        m_particles_per_box[lev].resize(m_particleBoxArray_size[lev]);
+        m_offset_per_box[lev].resize(m_particleBoxArray_size[lev]);
+        for (int i = 0; i < m_particleBoxArray_size[lev]; ++i) {
+            is >> m_which_data[lev][i] >> m_particles_per_box[lev][i] >> m_offset_per_box[lev][i];
+        }
+    }
+
+}
+
+void
+BTDSpeciesHeaderImpl::WriteHeader ()
+{
+    if (amrex::FileExists(m_Header_path)) {
+        amrex::FileSystem::Remove(m_Header_path);
+    }
+    std::ofstream HeaderFile;
+    HeaderFile.open(m_Header_path.c_str(), std::ofstream::out |
+                                           std::ofstream::trunc |
+                                           std::ofstream::binary);
+    if ( !HeaderFile.good()) amrex::FileOpenFailed(m_Header_path);
+
+    HeaderFile.precision(17);
+
+    // File Version
+    HeaderFile << m_file_version << '\n';
+    // spacedim
+    HeaderFile << m_spacedim << '\n';
+    // number of real components output
+    HeaderFile << m_num_output_real << '\n';
+    for (int i = 0; i < m_num_output_real; ++i) {
+        HeaderFile << m_real_comp_names[i] << '\n';
+    }
+    HeaderFile << m_num_output_int << '\n';
+    for (int i = 0; i < m_num_output_int; ++i) {
+        HeaderFile << m_int_comp_names[i] << '\n';
+    }
+    HeaderFile << m_is_checkpoint << '\n';
+    HeaderFile << m_total_particles << '\n';
+    HeaderFile << m_nextid << '\n';
+    HeaderFile << m_finestLevel << '\n';
+    for (int lev = 0; lev <= m_finestLevel; ++lev) {
+        HeaderFile << m_particleBoxArray_size[lev] << '\n';
+        for (int i = 0; i < m_particleBoxArray_size[lev]; ++i) {
+            HeaderFile << m_which_data[lev][i] << ' ' << m_particles_per_box[lev][i] << ' ' << m_offset_per_box[lev][i] << '\n';
+        }
+    }
+}
+
+void
+BTDSpeciesHeaderImpl::set_DataIndex(const int lev, const int box_id, const int data_index)
+{
+    m_which_data[lev][box_id] = data_index;
+}
+
+void
+BTDSpeciesHeaderImpl::AppendParticleInfoForNewBox ( const int data_index,
+                                                    const int particles_per_box,
+                                                    const int offset)
+{
+    m_which_data[m_finestLevel].resize(m_particleBoxArray_size[m_finestLevel]);
+    m_particles_per_box[m_finestLevel].resize(m_particleBoxArray_size[m_finestLevel]);
+    m_offset_per_box[m_finestLevel].resize(m_particleBoxArray_size[m_finestLevel]);
+
+    const int last_boxId = m_particleBoxArray_size[m_finestLevel] - 1;
+    m_which_data[m_finestLevel][last_boxId] = data_index;
+    m_particles_per_box[m_finestLevel][last_boxId] = particles_per_box;
+    m_offset_per_box[m_finestLevel][last_boxId] = offset;
+
+}
+
+BTDParticleDataHeaderImpl::BTDParticleDataHeaderImpl (std::string const & Headerfile_path)
+    : m_Header_path(Headerfile_path)
+{
+
+}
+
+void
+BTDParticleDataHeaderImpl::ReadHeader ()
+{
+    // Read existing fab Header first
+    amrex::Vector<char> HeaderCharPtr;
+    amrex::Long fileLength(0), fileLengthPadded(0);
+    std::ifstream iss;
+
+    iss.open(m_Header_path.c_str(), std::ios::in);
+    iss.seekg(0, std::ios::end);
+    fileLength = static_cast<std::streamoff>(iss.tellg());
+    iss.seekg(0, std::ios::beg);
+
+    fileLengthPadded = fileLength + 1;
+    HeaderCharPtr.resize(fileLengthPadded);
+    iss.read(HeaderCharPtr.dataPtr(), fileLength);
+    iss.close();
+    HeaderCharPtr[fileLength] = '\0';
+
+    std::istringstream is(HeaderCharPtr.dataPtr(), std::istringstream::in);
+    is.exceptions(std::ios_base::failbit | std::ios_base::badbit);
+
+
+    int in_hash;
+    int bl_ignore_max = 100000;
+
+    is.ignore(bl_ignore_max,'(') >> m_ba_size >> in_hash;
+    m_ba.resize(m_ba_size);
+    for (int ibox = 0; ibox < m_ba.size(); ++ibox) {
+        amrex::Box bx;
+        is >> bx;
+        m_ba.set(ibox, bx);
+    }
+    is.ignore(bl_ignore_max, ')');
+
+}
+
+void
+BTDParticleDataHeaderImpl::WriteHeader ()
+{
+    if (amrex::FileExists(m_Header_path)) {
+        amrex::FileSystem::Remove(m_Header_path);
+    }
+    std::ofstream HeaderFile;
+    HeaderFile.open(m_Header_path.c_str(), std::ofstream::out |
+                                           std::ofstream::trunc |
+                                           std::ofstream::binary);
+    if ( !HeaderFile.good()) amrex::FileOpenFailed(m_Header_path);
+
+    HeaderFile.precision(17);
+    m_ba.writeOn(HeaderFile);
+}