11 files changed, 421 insertions, 0 deletions
diff --git a/Docs/source/developers/amrex_basics.rst b/Docs/source/developers/amrex_basics.rst
new file mode 100644
index 000000000..577a6547b
--- /dev/null
+++ b/Docs/source/developers/amrex_basics.rst
@@ -0,0 +1,32 @@
+.. _developers-amrex-basics:
+
+AMReX basics (excessively basic)
+================================
+
+WarpX is built on the Adaptive Mesh Refinement (AMR) library `AMReX <https://github.com/AMReX-Codes/amrex>`__. This section provides a very sporadic description of the main AMReX classes and concepts relevant for WarpX, that can serve as a reminder. Please read the AMReX basics `doc page <https://amrex-codes.github.io/amrex/docs_html/Basics.html>`__, of which this section is largely inspired.
+
+* ``amrex::Box``: Dimension-dependent lower and upper indices defining a rectangular volume in 3D (or surface in 2D) in the index space. ``Box`` is a lightweight meta-data class, with useful member functions.
+
+* ``amrex::BoxArray``: Collection of ``Box`` on a single AMR level. The information of which MPI rank owns which ``Box`` in a ``BoxArray`` is in ``DistributionMapping``.
+
+* ``amrex::FArrayBox``: Fortran-ordered array of floating-point ``amrex::Real`` elements defined on a ``Box``. A ``FArrayBox`` can represent scalar data or vector data, with ``ncomp`` components.
+
+* ``amrex::MultiFab``: Collection of `FAB` (= ``FArrayBox``) on a single AMR level, distributed over MPI ranks. The concept of `ghost cells` is defined at the ``MultiFab`` level.
+
+* ``amrex::ParticleContainer``: A collection of particles, typically for particles of a physical species. Particles in a ``ParticleContainer`` are organized per ``Box``. Particles in a ``Box`` are organized per tile (this feature is off when running on GPU). Particles within a tile are stored in several structures, each being contiguous in memory: (i) an Array-Of-Struct (AoS) (often called `data`, they are the 3D position, the particle ID and the index of the CPU owning the particle), where the Struct is an ``amrex::Particle`` and (ii) Struct-Of-Arrays (SoA) for extra variables (often called ``attribs``, in WarpX they are the momentum, field on particle etc.).
+
+The simulation domain is decomposed in several ``Box``, and each MPI rank owns (and performs operations on) the fields and particles defined on a few of these ``Box``, but has the metadata of all of them. For convenience, AMReX provides iterators, to easily iterate over all ``FArrayBox`` (or even tile-by-tile, optionally) in a ``MultiFab`` own by the MPI rank (``MFIter``), or over all particles in a ``ParticleContainer`` on a per-box basis (``ParIter``, or its derived class ``WarpXParIter``). These are respectively done in loops like:
+
+.. code-block:: cpp
+
+   // mf is a pointer to MultiFab
+   for ( amrex::MFIter mfi(mf, false); mfi.isValid(); ++mfi ) { ... }
+
+and
+
+.. code-block:: cpp
+
+   // *this is a pointer to a ParticleContainer
+   for (WarpXParIter pti(*this, lev); pti.isValid(); ++pti) { ... }
+
+When looping over ``FArrayBox`` in a ``MultiFab``, the iterator provides functions to retrieve the metadata of the ``Box`` on which the ``FAB`` is defined (``MFIter::box()``, ``MFIter::tilebox()`` or variations) or the particles defined on this ``Box`` (``ParIter::GetParticles()``).
diff --git a/Docs/source/developers/developers.rst b/Docs/source/developers/developers.rst
new file mode 100644
index 000000000..4eab0102a
--- /dev/null
+++ b/Docs/source/developers/developers.rst
@@ -0,0 +1,20 @@
+.. _developers:
+
+Developers documentation
+========================
+
+For general information on how to contribute to WarpX, including our ``git`` workflow and code practices, have a look at our `CONTRIBUTING.md <https://github.com/ECP-WarpX/WarpX/blob/dev/CONTRIBUTING.md>`__!
+
+.. toctree::
+   :maxdepth: 1
+
+   amrex_basics
+   repo_organization
+   fields
+   particles
+   initialization
+   diagnostics
+   moving_window
+   portability
+   profiling
+   documentation
diff --git a/Docs/source/developers/diagnostics.rst b/Docs/source/developers/diagnostics.rst
new file mode 100644
index 000000000..6945670b5
--- /dev/null
+++ b/Docs/source/developers/diagnostics.rst
@@ -0,0 +1,18 @@
+.. _developers-diagnostics:
+
+Diagnostics
+===========
+
+Regular Diagnostics (plotfiles)
+-------------------------------
+
+.. note::
+
+   Section empty!
+
+Back-Transformed Diagnostics
+----------------------------
+
+.. note::
+
+   Section empty!
diff --git a/Docs/source/developers/documentation.rst b/Docs/source/developers/documentation.rst
new file mode 100644
index 000000000..a6f06c7bd
--- /dev/null
+++ b/Docs/source/developers/documentation.rst
@@ -0,0 +1,48 @@
+.. _developers-docs:
+
+Documentation
+=============
+
+.. warning::
+
+   Needs info on how to install all pieces to compile the doc! Some basic info is in the ``CONTRIBUTING.md`` .
+
+Doxygen documentation
+---------------------
+
+WarpX uses a Doxygen documentation. Whenever you create a new class, please document it where it is declared (typically in the header file):
+
+.. code-block:: cpp
+
+   /**
+    * few-line description explaining the purpose of my_class.
+    *
+    * If you are kind enough, also quickly explain how things in my_class work.
+    * (typically a few more lines)
+    */
+   class my_class
+   { ... }
+
+Doxygen reads this docstring, so please be accurate with the syntax! See `Doxygen manual <http://www.doxygen.nl/manual/docblocks.html>`__ for more information. Similarly, please document functions when you declare them (typically in a header file) like:
+
+.. code-block:: cpp
+
+  /**
+   * few-line description explaining the purpose of my_function.
+   *
+   * \param[in,out] my_int a pointer to an integer variable on which
+   *                       my_function will operate.
+   */
+  void my_class::my_function(int* my_int);
+
+Breathe documentation
+---------------------
+
+Your Doxygen documentation is not only useful for people looking into the code, it is also part of the `WarpX online documentation <https://ecp-warpx.github.io>`__ based on `Sphinx <http://www.sphinx-doc.org/en/master/>`__! This is done using the Python module `Breathe <http://breathe.readthedocs.org>`__, that allows you to read Doxygen documentation dorectly in the source and include it in your Sphinx documentation, by calling Breathe functions. For instance, the following line will get the Doxygen documentation for ``WarpXParticleContainer`` in ``Source/Particles/WarpXParticleContainer.H`` and include it to the html page generated by Sphinx:
+
+  .. doxygenclass:: WarpXParticleContainer
+
+Exhale documentation
+--------------------
+
+Very similar to Breathe, the Python module `exhale <https://exhale.readthedocs.io/en/latest/>`__ reads the full Doxygen documentation and renders it in ` rst <https://en.wikipedia.org/wiki/ReStructuredText>`__ format, and is accessible from the main WarpX ReadTheDocs page.
diff --git a/Docs/source/developers/fields.rst b/Docs/source/developers/fields.rst
new file mode 100644
index 000000000..6f43246b8
--- /dev/null
+++ b/Docs/source/developers/fields.rst
@@ -0,0 +1,127 @@
+.. _developers-fields:
+
+Fields
+======
+
+.. note::
+
+   Add info on staggering and domain decomposition. Synchronize with section ``initialization``.
+
+The main fields are the electric field ``Efield``, the magnetic field ``Bfield``, the current density ``current`` and the charge density ``rho``. When a divergence-cleaner is used, we add another field ``F`` (containing :math:`\vec \nabla \cdot \vec E - \rho`).
+
+Due the AMR strategy used in WarpX (see section :ref:`Theory: AMR <theory-amr>` for a complete description), each field on a given refinement level ``lev`` (except for the coarsest ``0``) is defined on:
+
+* **the fine patch** (suffix ``_fp``, the actual resolution on ``lev``).
+
+* **the coarse patch** (suffix ``_cp``, same physical domain with the resolution of MR level ``lev-1``).
+
+* **the auxiliary grid** (suffix ``_aux``, same resolution as ``_fp``), from which the fields are gathered from the grids to particle positions. For this reason. only ``E`` and ``B`` are defined on this ``_aux`` grid (not the current density or charge density).
+
+* In some conditions, i.e., when buffers are used for the field gather (for numerical reasons), a **copy of E and B on the auxiliary grid** ``_aux`` **of the  level below** ``lev-1`` is stored in fields with suffix ``_cax`` (for `coarse aux`).
+
+As an example, the structures for the electric field are ``Efield_fp``, ``Efield_cp``, ``Efield_aux`` (and optionally ``Efield_cax``).
+
+Declaration
+-----------
+
+All the fields described above are public members of class ``WarpX``, defined in ``WarpX.H``. They are defined as an ``amrex::Vector`` (over MR levels) of ``std::array`` (for the 3 spatial components :math:`E_x`, :math:`E_y`, :math:`E_z`) of ``std::unique_ptr`` of ``amrex::MultiFab``, i.e.:
+
+  amrex::Vector<std::array< std::unique_ptr<amrex::MultiFab>, 3 > > Efield_fp;
+
+Hence, ``Ex`` on MR level ``lev`` is a pointer to an ``amrex::MultiFab``. The other fields are organized in the same way.
+
+Allocation and initialization
+-----------------------------
+
+The ``MultiFab`` constructor (for, e.g., ``Ex`` on level ``lev``) is called in ``WarpX::AllocLevelMFs``.
+
+By default, the ``MultiFab`` are set to ``0`` at initialization. They can be assigned a different value in ``WarpX::InitLevelData``.
+
+Field solver
+------------
+
+The field solver is performed in ``WarpX::EvolveE`` for the electric field and ``WarpX::EvolveB`` for the magnetic field, called from ``WarpX::OneStep_nosub`` in ``WarpX::EvolveEM``. This section describes the FDTD field push (the PSATD field push should be added later). It is implemented in ``Source/FieldSolver/WarpXPushFieldsEM.cpp``.
+
+As all cell-wise operation, the field push is done as follows (this is split in multiple functions in the actual implementation to aboid code duplication)
+:
+
+.. code-block:: cpp
+
+   // Loop over MR levels
+   for (int lev = 0; lev <= finest_level; ++lev) {
+      // Get pointer to MultiFab Ex on level lev
+      MultiFab* Ex = Efield_fp[lev][0].get();
+      // Loop over boxes (or tiles if not on GPU)
+      for ( MFIter mfi(*Ex, TilingIfNotGPU()); mfi.isValid(); ++mfi ) {
+          // Apply field solver on the FAB
+      }
+  }
+
+The innermost step ``// Apply field solver on the FAB`` could be done with 3 nested ``for`` loops for the 3 dimensions (in 3D). However, for portability reasons (see section :ref:`Developers: Portability <developers-portability>`), this is done in two steps: (i) extract AMReX data structures into plain-old-data simple structures, and (ii) call a general ``ParallelFor`` function (translated into nested loops on CPU or a kernel launch on GPU, for instance):
+
+.. code-block:: cpp
+
+  // Get Box corresponding to the current MFIter
+  const Box& tex  = mfi.tilebox(Ex_nodal_flag);
+  // Extract the FArrayBox into a simple structure, for portability
+  Array4<Real> const& Exfab = Ex->array(mfi);
+  // Loop over cells and perform stencil operation
+  amrex::ParallelFor(tex,
+      [=] AMREX_GPU_DEVICE (int j, int k, int l)
+      {
+          warpx_push_ex_yee(...);
+      }
+  );
+
+Function ``warpx_push_ex_yee`` performs the FDTD stencil operation on a single cell. It is implemented in ``Source/FieldSolver/WarpX_K.H`` (where ``_K`` stands for kernel).
+
+Guard cells exchanges
+---------------------
+
+Communications are mostly handled in ``Source/Parallelization/``.
+
+For E and B guard cell **exchanges**, the main functions are variants of ``amrex::FillBoundary(amrex::MultiFab, ...)`` (or ``amrex::MultiFab::FillBoundary(...)``) that fill guard cells of all ``amrex::FArrayBox`` in an ``amrex::MultiFab`` with valid cells of corresponding ``amrex::FArrayBox`` neighbors of the same ``amrex::MultiFab``. There are a number of ``FillBoundaryE``, ``FillBoundaryB`` etc. Under the hood, ``amrex::FillBoundary`` calls ``amrex::ParallelCopy``, which is also sometimes directly called in WarpX. Most calls a
+
+For the current density, the valid cells of neighboring ``MultiFabs`` are accumulated (added) rather than just copied. This is done using ``amrex::MultiFab::SumBoundary``, and mostly located in ``Source/Parallelization/WarpXSumGuardCells.H``.
+
+Interpolations for MR
+---------------------
+
+This is mostly implemented in ``Source/Parallelization``, see the following functions (you may complain to the authors if the documentation is empty)
+
+.. doxygenfunction:: WarpX::SyncCurrent
+
+.. doxygenfunction:: interpolateCurrentFineToCoarse
+
+.. doxygenfunction:: WarpX::RestrictCurrentFromFineToCoarsePatch
+
+.. doxygenfunction:: WarpX::AddCurrentFromFineLevelandSumBoundary
+
+Filter
+------
+
+General functions for filtering can be found in ``Source/Filter/``, where the main ``Filter`` class is defined (see below). All filters (so far there are two of them) in WarpX derive from this class.
+
+.. doxygenclass:: Filter
+
+Bilinear filter
+~~~~~~~~~~~~~~~
+
+The multi-pass bilinear filter (applied on the current density) is implemented in ``Source/Filter/``, and class ``WarpX`` holds an instance of this class in member variable ``WarpX::bilinear_filter``. For performance reasons (to avoid creating too many guard cells), this filter is directly applied in communication routines, see
+
+.. doxygenfunction:: WarpX::AddCurrentFromFineLevelandSumBoundary
+
+and
+
+.. doxygenfunction:: WarpX::ApplyFilterandSumBoundaryJ
+
+Godfrey's anti-NCI filter for FDTD simulations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This filter is applied on the electric and magnetic field (on the auxiliary grid) to suppress the Numerical Cherenkov Instability when running FDTD. It is implemented in ``Source/Filter/``, and there are two different stencils, one for ``Ex``, ``Ey`` and ``Bz`` and the other for ``Ez``, ``Bx`` and ``By``.
+
+.. doxygenclass:: NCIGodfreyFilter
+
+The class ``WarpX`` holds two corresponding instances of this class in member variables ``WarpX::nci_godfrey_filter_exeybz`` and ``WarpX::nci_godfrey_filter_bxbyez``. It is a 9-point stencil (is the ``z`` direction only), for which the coefficients are computed using tabulated values (depending on dz/dx) in ``Source/Utils/NCIGodfreyTables.H``, see variable ``table_nci_godfrey_galerkin_Ex_Ey_Bz``. The filter is applied in ``PhysicalParticleContainer::Evolve``, right after field gather and before particle push, see
+
+.. doxygenfunction:: PhysicalParticleContainer::applyNCIFilter
diff --git a/Docs/source/developers/initialization.rst b/Docs/source/developers/initialization.rst
new file mode 100644
index 000000000..79e94d4d2
--- /dev/null
+++ b/Docs/source/developers/initialization.rst
@@ -0,0 +1,22 @@
+.. _developers-initialization:
+
+Initialization
+==============
+
+.. note::
+   Section almost empty!!
+
+General simulation initialization
+---------------------------------
+
+Regular simulation
+~~~~~~~~~~~~~~~~~~
+
+Running in a boosted frame
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Field initialization
+--------------------
+
+Particle initialization
+-----------------------
diff --git a/Docs/source/developers/moving_window.rst b/Docs/source/developers/moving_window.rst
new file mode 100644
index 000000000..38e17c962
--- /dev/null
+++ b/Docs/source/developers/moving_window.rst
@@ -0,0 +1,8 @@
+.. _developers-moving-window:
+
+Moving Window
+=============
+
+.. note::
+
+   Section empty!
diff --git a/Docs/source/developers/particles.rst b/Docs/source/developers/particles.rst
new file mode 100644
index 000000000..edd817b29
--- /dev/null
+++ b/Docs/source/developers/particles.rst
@@ -0,0 +1,108 @@
+.. _developers-particles:
+
+Particles
+=========
+
+Particle containers
+-------------------
+
+Particle structures and functions are defined in ``Source/Particles/``. WarpX uses the ``Particle`` class from AMReX for single particles. An ensemble of particles (e.g., a plasma species, or laser particles) is stored as a ``WarpXParticleContainer`` (see description below) in a per-box (and even per-tile on CPU) basis.
+
+.. doxygenclass:: WarpXParticleContainer
+
+Physical species are stored in ``PhysicalParticleContainer``, that derives from ``WarpXParticleContainer``. In particular, the main function to advance all particles in a physical species is ``PhysicalParticleContainer::Evolve`` (see below).
+
+.. doxygenfunction:: PhysicalParticleContainer::Evolve
+   :outline:
+
+Finally, all particle species (physical plasma species ``PhysicalParticleContainer``, photon species ``PhotonParticleContainer`` or non-physical species ``LaserParticleContainer``) are stored in ``MultiParticleContainer``. The class ``WarpX`` holds one instance of ``MultiParticleContainer`` as a member variable, called ``WarpX::mypc`` (where `mypc` stands for "my particle containers"):
+
+.. doxygenclass:: MultiParticleContainer
+
+Loop over particles
+-------------------
+
+A typical loop over particles reads:
+
+.. code-block:: cpp
+
+  // pc is a std::unique_ptr<WarpXParticleContainer>
+  // Loop over MR levels
+  for (int lev = 0; lev <= finest_level; ++lev) {
+      // Loop over particles, box by box
+      for (WarpXParIter pti(*this, lev); pti.isValid(); ++pti) {
+          // Do something on particles
+          // [MY INNER LOOP]
+      }
+  }
+
+The innermost step ``[MY INNER LOOP]`` typically calls ``amrex::ParallelFor`` to perform operations on all particles in a portable way. For this reasons, the particle data needs to be converted in plain-old-data structures. The innermost loop in the code snippet above could look like:
+
+.. code-block:: cpp
+
+  // Get Array-Of-Struct particle data, also called data
+  // (x, y, z, id, cpu)
+  const auto& particles = pti.GetArrayOfStructs();
+  // Get Struct-Of-Array particle data, also called attribs
+  // (ux, uy, uz, w, Exp, Ey, Ez, Bx, By, Bz)
+  auto& attribs = pti.GetAttribs();
+  auto& Exp = attribs[PIdx::Ex];
+  // [...]
+  // Number of particles in this box
+  const long np = pti.numParticles();
+
+Link fields and particles?
+--------------------------
+
+In WarpX, the loop over boxes through a ``MultiFab`` iterator ``MFIter`` and the loop over boxes through a ``ParticleContainer`` iterator ``WarpXParIter`` are consistent.
+
+On a loop over boxes in a ``MultiFab`` (``MFIter``), it can be useful to access particle data on a GPU-friendly way. This can be done by:
+
+.. code-block:: cpp
+
+  // Index of grid (= box)
+  const int grid_id = mfi.index();
+  // Index of tile within the grid
+  const int tile_id = mfi.LocalTileIndex();
+  // Get GPU-friendly arrays of particle data
+  auto& ptile = GetParticles(lev)[std::make_pair(grid_id,tile_id)];
+  ParticleType* pp = particle_tile.GetArrayOfStructs()().data();
+  // Only need attribs (i.e., SoA data)
+  auto& soa = ptile.GetStructOfArrays();
+  // As an example, let's get the ux momentum
+  const ParticleReal * const AMREX_RESTRICT ux = soa.GetRealData(PIdx::ux).data();
+
+On a loop over particles it can be useful to access the fields on the box we are looping over (typically when we use both field and particle data on the same box, for field gather or current deposition for instance). This is done for instance by adding this snippet in ``[MY INNER LOOP]``:
+
+.. code-block:: cpp
+
+  // E is a reference to, say, WarpX::Efield_aux
+  // Get the Ex field on the grid
+  const FArrayBox& exfab = (*E[lev][0])[pti];
+  // Let's be generous and also get the underlying box (i.e., index info)
+  const Box& box = pti.validbox();
+
+Main functions
+--------------
+
+.. doxygenfunction:: PhysicalParticleContainer::FieldGather
+
+.. doxygenfunction:: PhysicalParticleContainer::PushPX
+
+.. doxygenfunction:: WarpXParticleContainer::DepositCurrent
+
+.. note::
+   The current deposition is used both by ``PhysicalParticleContainer`` and ``LaserParticleContainer``, so it is in the parent class ``WarpXParticleContainer``.
+
+Buffers
+-------
+
+To reduce numerical artifacts at the boundary of a mesh-refinement patch, WarpX has an option to use buffers: When particles evolve on the fine level, they gather from the coarse level (e.g., ``Efield_cax``, a copy of the ``aux`` data from the level below) if they are located on the fine level but fewer than ``WarpX::n_field_gather_buffer`` cells away from the coarse-patch boundary. Similarly, when particles evolve on the fine level, they deposit on the coarse level (e.g., ``Efield_cp``) if they are located on the fine level but fewer than ``WarpX::n_current_deposition_buffer`` cells away from the coarse-patch boundary.
+
+``WarpX::gather_buffer_masks`` and ``WarpX::current_buffer_masks`` contain masks indicating if a cell is in the interior of the fine-resolution patch or in the buffers. Then, particles depending on this mask in
+
+.. doxygenfunction:: PhysicalParticleContainer::PartitionParticlesInBuffers
+
+.. note::
+
+   Buffers are complex!
diff --git a/Docs/source/developers/portability.rst b/Docs/source/developers/portability.rst
new file mode 100644
index 000000000..c20d6f41f
--- /dev/null
+++ b/Docs/source/developers/portability.rst
@@ -0,0 +1,7 @@
+.. _developers-portability:
+
+Portability
+===========
+
+.. note::
+   Section empty!
diff --git a/Docs/source/developers/profiling.rst b/Docs/source/developers/profiling.rst
new file mode 100644
index 000000000..86c838e8d
--- /dev/null
+++ b/Docs/source/developers/profiling.rst
@@ -0,0 +1,7 @@
+.. _developers-profiling:
+
+Profiling
+=========
+
+.. note::
+   Section empty!
diff --git a/Docs/source/developers/repo_organization.rst b/Docs/source/developers/repo_organization.rst
new file mode 100644
index 000000000..495cc8fc7
--- /dev/null
+++ b/Docs/source/developers/repo_organization.rst
@@ -0,0 +1,24 @@
+.. _developers-repo-structure:
+
+WarpX structure
+===============
+
+Repo organization
+-----------------
+
+All the WarpX source code is located in ``Source/``. All sub-directories have a pretty straigtforward name. The PIC loop is part of the WarpX class, in function ``WarpX::EvolveEM`` implemented in ``Source/WarpXEvolveEM.cpp``. The core of the PIC loop (i.e., without diagnostics etc.) is in ``WarpX::OneStep_nosub`` (when subcycling is OFF) or ``WarpX::OneStep_sub1`` (when subcycling is ON, with method 1).
+
+Code organization
+-----------------
+
+The main WarpX class is WarpX, implemented in ``Source/WarpX.cpp``.
+
+Build system
+------------
+
+WarpX uses the AMReX build system. Each sub-folder contains a file ``Make.package`` with the names of header files and source files to include for the build.
+
+WarpX-specific vocabulary
+-------------------------
+
+- ``Evolve`` is a generic term to advance a quantity (this comes from AMReX). For instance, ``WarpX::EvolveE(dt)`` advances the electric field for duration ``dt``, ``PhysicalParticleContainer::Evolve(...)`` does field gather + particle push + current deposition for all particles in ``PhysicalParticleContainer``, and ``WarpX::EvolveEM`` is the central ``WarpX`` function that performs 1 PIC iteration.