Transition to pyAMReX (#3474)

* pyAMReX: Build System

* CI Updates (Changed Options)

* Callback modernization (#4)

* refactor callbacks.py
* added binding code in `pyWarpX.cpp` to add or remove keys from the callback dictionary
* minor PR cleanups

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Added Python level reference to fetch the multifabs (#3)

* pyAMReX: Build System

* Added Python level reference to Ex_aux

* Now uses the multifab map

* Fix typo

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Add initialization and finalize routines (#5)

A basic PICMI input file will now run to completion.

* Regression Tests: WarpX_PYTHON=ON

* Update Imports to nD pyAMReX

* IPO/LTO Control

Although pybind11 relies heavily on IPO/LTO to create low-latency,
small-binary bindings, some compilers will have troubles with that.

Thus, we add a compile-time option to optionally disable it when
needed.

* Fix: Link Legacy WarpXWrappers.cpp

* Wrap WarpX instance and start multi particle container

* Fix test Python_pass_mpi_comm
* Start wrapper for multiparticle container
* Add return policy

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Update fields to use MultiFabs directly

Remove EOL white space

Removed old routines accessing MultiFabs

Update to use "node_centered"

* Fix compilation with Python

* Update fields.py to use modified MultiFab tag names

* Remove incorrect, unused code

* Add function to extract the WarpX MultiParticleContainer

* Complete class WarpXParticleContainer

* Wrap functions getNprocs / getMyProc

* restore `install___` callback API - could remove later if we want but should maintain backward compatibility for now

* add `gett_new` and `getistep` functions wrappers; fix typos in `callbacks.py`; avoid error in getting `rho` from `fields.py`

* Update callback call and `getNproc`/`getMyProc` function

* Replace function add_n_particles

* Fix setitem in fields.py for 1d and 2d

* also update `gett_new()` in `_libwarpx.py` in case we want to keep that API

* added binding for `WarpXParIter` - needed to port `libwarpx.depositChargeDensity()` which is an ongoing effort

* Wrap function num_real_comp

* added binding for `TotalNumberOfParticles` and continue progress on enabling 1d MCC test to run

* add `SyncRho()` binding and create helper function in `libwarpx.depositChargeDensity` to manage scope of the particle iter

* Clean up issues in fields.py

* update bindings for `get_particle_structs`

* Fix setitem in fields.py

* switch order of initialization for particle container and particle iterator

* switch deposit_charge loop to C++ code; bind `ApplyInverseVolumeScalingToChargeDensity`

* move `WarpXParticleContainer.cpp` and `MultiParticleContainer.cpp` to new Particles folder

* added binding for `ParticleBoundaryBuffer`

* More fixes for fields.py

* Fix: Backtraces from Python

Add the Python executable name with an absolute path, so backtraces
in AMReX work. See linked AMReX issue for details.

* Cleaning

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix: Backtraces from Python Part II

Do not add Python script name - it confuses the AMReX ParmParser to
build its table.

* Fix: CMake Dependencies for Wheel

This fixes a racecondition during `pip_install`: it was possible
that not all dimensions where yet build from pybind before we
start packing them in the wheel for pip install.

* MCC Test: Install Callbacks before Run

Otherwise hangs in aquiring the gil during shutdown.

* addition of `Python/pywarpx/particle_containers.py` and various associated bindings

* Fix: CMake Superbuild w/ Shared AMReX

We MUST build AMReX as a shared (so/dll/dylib) library, otherwise
all the global state in it will cause split-brain situations, where
our Python modules operate on different stacks.

* add `clear_all()` to callbacks in order to remove all callbacks at finalize

* add `-DWarpX_PYTHON=ON` to CI tests that failed to build

* add `get_comp_index` and continue to port particle data bindings

* Add AMReX Module as `libwarpx_so.amr`

Attribute to pass through the already loaded AMReX module with the
matching dimensionality to the simulation.

* Fix for fields accounting for guard cells

* Fix handling of ghost cells in fields

* Update & Test: Particle Boundary Scraping

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* CI: Python Updates

- modernize Python setups
- drop CUDA 11.0 for good and go 11.3+ as documented already
```
Error #3246: Internal Compiler Error (codegen): "there was an error in verifying the lgenfe output!"
```

* CI: Python Updates (chmod)

* Add support for cupy in fields.py

* Add MultiFab reduction routines

* CI: CUDA 11.3 is <= Ubuntu 20.04

* changed `AddNParticles` to take `amrex::Vector` arguments

* setup.py: WarpX_PYTHON=ON

* update various 2d and rz tests with new APIs

* add `-DWarpX_PYTHON_IPO=OFF` to compile string for 2d and 3d Python CI tests to speed up linking

* CI: -DpyAMReX_IPO=OFF

* CI: -DpyAMReX_IPO=OFF

actually adding `=OFF`

* CI: Intel Python

* CI: macOS Python Executable

Ensure we always use the same `python3` executable, as specified
by the `PATH` priority.

* CMake: Python Multi-Config Build

Add support for multi-config generators, especially on Windows.

* __init__.py: Windows DLL Support

Python 3.8+ on Windows: DLL search paths for dependent
shared libraries
Refs.:
- https://github.com/python/cpython/issues/80266
- https://docs.python.org/3.8/library/os.html#os.add_dll_directory

* CI: pywarpx Update

our setup.py cannot install pyamrex yet as a dependency.

* ABLASTR: `ablastr/export.H`

Add a new header to export public globals that are not covered by
`WINDOWS_EXPORT_ALL_SYMBOLS`.
https://stackoverflow.com/questions/54560832/cmake-windows-export-all-symbols-does-not-cover-global-variables/54568678#54568678

* WarpX: EXPORT Globals in `.dll` files

WarpX still uses a lot of globals:
- `static` member variables
- `extern` global variables

These globals cannot be auto-exported with CMake's
`WINDOWS_EXPORT_ALL_SYMBOLS` helper and thus we need to mark them
manually for DLL export (and import) via the new ABLASTR
`ablastr/export.H` helper macros.

This starts to export symbols in the:
- WarpX and particle container classes
- callback hook database map
- ES solver

* CI: pywarpx Clang CXXFLAGS Down

Move CXXFLAGS (`-Werror ...`) down until deps are installed.

* GNUmake: Generate `ablastr/export.H`

* CMake: More Symbol Exports for Windows

* `WarpX-tests.ini`: Simplify Python no-IPO

Also avoids subtle differences in compilation that increase
compile time.

* Update PICMI_inputs_EB_API.py for embedded_boundary_python_API CI test

* Fix Python_magnetostatic_eb_3d

* Update: Python_restart_runtime_components

New Python APIs

* Windows: no dllimport for now

* CI: Skip `PYINSTALLOPTIONS` For Now

* CMake: Dependency Bump Min-Versions

for external packages picked up by `find_package`.

* Fix F and G_fp names in fields.py

* Tests: Disable `Python_pass_mpi_comm`

* Wrappers: Cleanup

* pyWarpX: Include Cleaning

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fields.py: Fix F and G Wrappers

Correct MultiFab names (w/o components).

* Remove unused in fields.py

* Windows: New Export Headers

- ABLASTR: `ablastr/export.H`
- WarpX: `Utils/export.H`

* removed `WarpInterface.py` since that functionality is now in `particle_containers.py`; removed parts of `WarpXWrappers.cpp` that have been ported to pyamrex

* CMake: Link OBJECT Target PRIVATE

* CMake: Remove OBJECT Target

Simplify and make `app` link `lib` (default: static). Remove OBJECT
target.

* Fix in fields.py for the components index

* Update get_particle_id/cpu

As implemented in pyAMReX with
https://github.com/AMReX-Codes/pyamrex/pull/165

* WarpX: Update for Private Constructor

* Import AMReX Before pyd DLL Call

Importing AMReX will add the `add_dll_directory` to a potentially
shared amrex DLL on Windows.

* Windows CI: Set PATH to amrex_Nd.dll

* CMake: AMReX_INSTALL After Python

In superbuild, Python can modify `AMReX_BUILD_SHARED_LIBS`.

* Clang Win CI: Manually Install requirements

Sporadic error is:
```
...
Installing collected packages: pyparsing, numpy, scipy, periodictable, picmistandard
ERROR: Could not install packages due to an OSError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\hostedtoolcache\\windows\\Python\\3.11.4\\x64\\Lib\\site-packages\\numpy\\polynomial\\__init__.py'
Consider using the `--user` option or check the permissions.
```

* Hopefully final fixes to fields.py

* Update getProbLo/getProbHi

* Set plasma length strength

Co-authored-by: Remi Lehe <remi.lehe@normalesup.org>

* Fix fields method to remove CodeQL notice

* Update Comments & Some Finalize

* Move: set_plasma_lens_strength

to MPC

---------

Co-authored-by: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com>
Co-authored-by: David Grote <dpgrote@lbl.gov>
Co-authored-by: Remi Lehe <remi.lehe@normalesup.org>
Co-authored-by: Dave Grote <grote1@llnl.gov>
Co-authored-by: Roelof Groenewald <regroenewald@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

1 files changed, 4 insertions, 197 deletions

diff --git a/Source/Python/WarpXWrappers.H b/Source/Python/WarpXWrappers.H
index cd1e41cf4..38ba8b10f 100644
--- a/Source/Python/WarpXWrappers.H
+++ b/Source/Python/WarpXWrappers.H
@@ -3,6 +3,10 @@
  *
  * This file is part of WarpX.
  *
+ * This file is a legacy file and will be removed soon.
+ * Please do NOT add new bindings here! Please see the other files
+ * in this directory for the new pybind11-based bindings!
+ *
  * License: BSD-3-Clause-LBNL
  */
 #ifndef WARPX_WRAPPERS_H_
@@ -30,104 +34,20 @@ extern "C" {
 
     int warpx_galerkin_interpolation();
 
-    int warpx_nComps();
-
-    int warpx_nCompsSpecies(const char* char_species_name);
-
-    int warpx_SpaceDim();
-
-    void amrex_init (int argc, char* argv[]);
-
     void amrex_init_with_inited_mpi (int argc, char* argv[], MPI_Comm mpicomm);
 
-    void amrex_finalize (int finalize_mpi);
-
-    void warpx_init ();
-
-    void warpx_finalize ();
-
     typedef void(*WARPX_CALLBACK_PY_FUNC_0)();
 
-    void warpx_set_callback_py (const char* char_callback_name,
-                                WARPX_CALLBACK_PY_FUNC_0 callback);
-    void warpx_clear_callback_py (const char* char_callback_name);
-
-    void warpx_evolve (int numsteps);  // -1 means the inputs parameter will be used.
-
-    void warpx_addNParticles(const char* char_species_name,
-                             int lenx,
-                             amrex::ParticleReal const * x,
-                             amrex::ParticleReal const * y,
-                             amrex::ParticleReal const * z,
-                             amrex::ParticleReal const * vx,
-                             amrex::ParticleReal const * vy,
-                             amrex::ParticleReal const * vz,
-                             const int nattr_real,
-                             amrex::ParticleReal const * attr_real,
-                             const int nattr_int,
-                             int const * attr_int,
-                             int uniqueparticles);
-
     void warpx_ConvertLabParamsToBoost();
 
     void warpx_ReadBCParams();
 
     void warpx_CheckGriddingForRZSpectral();
 
-    amrex::Real warpx_getProbLo(int dir);
-
-    amrex::Real warpx_getProbHi(int dir);
-
     amrex::Real warpx_getCellSize(int dir, int lev);
 
-    long warpx_getNumParticles(const char* char_species_name, const bool local);
-
-    void warpx_convert_id_to_long (amrex::Long* ids, const WarpXParticleContainer::ParticleType* pstructs, int size);
-
-    void warpx_convert_cpu_to_int (int* cpus, const WarpXParticleContainer::ParticleType* pstructs, int size);
-
-    amrex::ParticleReal** warpx_getParticleStructs(
-        const char* char_species_name, int lev, int* num_tiles,
-        int** particles_per_tile);
-
-    amrex::ParticleReal** warpx_getParticleArrays(
-        const char* char_species_name, const char* char_comp_name, int lev,
-        int* num_tiles, int** particles_per_tile);
-
-    int warpx_getParticleCompIndex(
-        const char* char_species_name, const char* char_comp_name);
-
-    void warpx_addRealComp(
-        const char* char_species_name, const char* char_comp_name, bool comm);
-
     amrex::Real warpx_sumParticleCharge(const char* char_species_name, const bool local);
 
-    int warpx_getParticleBoundaryBufferSize(const char* species_name, int boundary, bool local);
-
-    int** warpx_getParticleBoundaryBufferScrapedSteps(
-        const char* species_name, int boundary, int lev,
-        int* num_tiles, int** particles_per_tile);
-
-    amrex::ParticleReal** warpx_getParticleBoundaryBuffer(
-        const char* species_name, int boundary, int lev,
-        int* num_tiles, int** particles_per_tile, const char* comp_name);
-
-    amrex::ParticleReal** warpx_getParticleBoundaryBufferStructs(
-            const char* species_name, int boundary, int lev,
-            int* num_tiles, int** particles_per_tile);
-
-    void warpx_clearParticleBoundaryBuffer ();
-
-    /**
-     * \brief This function is used to deposit a given species' charge density
-     * in the rho_fp multifab which can then be accessed from python via
-     * pywarpx.fields.RhoFPWrapper()
-     *
-     * @param[in] species_name specifying the name of the species to deposit
-     * @param[in] lev mesh refinement level
-     */
-    void warpx_depositChargeDensity (const char* species_name, int lev);
-
   void warpx_ComputeDt ();
   void warpx_MoveWindow (int step, bool move_j);
 
@@ -135,7 +55,6 @@ extern "C" {
   void warpx_EvolveB (amrex::Real dt, DtType a_dt_type);
   void warpx_FillBoundaryE ();
   void warpx_FillBoundaryB ();
-  void warpx_SyncRho ();
   void warpx_SyncCurrent (
       const amrex::Vector<std::array<std::unique_ptr<amrex::MultiFab>,3>>& J_fp,
       const amrex::Vector<std::array<std::unique_ptr<amrex::MultiFab>,3>>& J_cp,
@@ -143,9 +62,7 @@ extern "C" {
   void warpx_UpdateAuxilaryData ();
   void warpx_PushParticlesandDepose (amrex::Real cur_time);
 
-  int warpx_getistep (int lev);
   void warpx_setistep (int lev, int ii);
-  amrex::Real warpx_gett_new (int lev);
   void warpx_sett_new (int lev, amrex::Real time);
   amrex::Real warpx_getdt (int lev);
 
@@ -154,118 +71,8 @@ extern "C" {
 
   int warpx_finestLevel ();
 
-  int warpx_getMyProc ();
-  int warpx_getNProcs ();
-
-  void warpx_setPotentialEB (const char * char_potential);
-
-  void warpx_setPlasmaLensStrength (const int i_lens, const amrex::Real strength_E, const amrex::Real strength_B);
-
   void mypc_Redistribute ();
 
-  amrex::Real** warpx_getEfield (int lev, int direction, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  amrex::Real** warpx_getEfieldCP (int lev, int direction, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  amrex::Real** warpx_getEfieldFP (int lev, int direction, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-
-  amrex::Real** warpx_getBfield (int lev, int direction, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  amrex::Real** warpx_getBfieldCP (int lev, int direction, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  amrex::Real** warpx_getBfieldFP (int lev, int direction, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-
-  amrex::Real** warpx_getCurrentDensity (int lev, int direction, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  amrex::Real** warpx_getCurrentDensityCP (int lev, int direction, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  amrex::Real** warpx_getCurrentDensityFP (int lev, int direction, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-
-  amrex::Real** warpx_getCurrentDensityFP_Ampere (int lev, int direction, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-
-  amrex::Real** warpx_getVectorPotentialFP (int lev, int direction, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-
-  int* warpx_getEfieldLoVects (int lev, int direction, int *return_size, int **ngrowvect);
-  int* warpx_getEfieldCPLoVects (int lev, int direction, int *return_size, int **ngrowvect);
-  int* warpx_getEfieldFPLoVects (int lev, int direction, int *return_size, int **ngrowvect);
-
-  int* warpx_getBfieldLoVects (int lev, int direction, int *return_size, int **ngrowvect);
-  int* warpx_getBfieldCPLoVects (int lev, int direction, int *return_size, int **ngrowvect);
-  int* warpx_getBfieldFPLoVects (int lev, int direction, int *return_size, int **ngrowvect);
-
-  int* warpx_getCurrentDensityLoVects (int lev, int direction, int *return_size, int **ngrowvect);
-  int* warpx_getCurrentDensityCPLoVects (int lev, int direction, int *return_size, int **ngrowvect);
-  int* warpx_getCurrentDensityFPLoVects (int lev, int direction, int *return_size, int **ngrowvect);
-
-  int* warpx_getVectorPotentialFPLoVects (int lev, int direction, int *return_size, int **ngrowvect);
-
-  amrex::Real** warpx_getEdgeLengths (int lev, int direction, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  int* warpx_getEdgeLengthsLoVects (int lev, int direction, int *return_size, int **ngrowvect);
-
-  amrex::Real** warpx_getFaceAreas (int lev, int direction, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  int* warpx_getFaceAreasLoVects (int lev, int direction, int *return_size, int **ngrowvect);
-
-  int* warpx_getEx_nodal_flag ();
-  int* warpx_getEy_nodal_flag ();
-  int* warpx_getEz_nodal_flag ();
-  int* warpx_getBx_nodal_flag ();
-  int* warpx_getBy_nodal_flag ();
-  int* warpx_getBz_nodal_flag ();
-  int* warpx_getAx_nodal_flag ();
-  int* warpx_getAy_nodal_flag ();
-  int* warpx_getAz_nodal_flag ();
-  int* warpx_getJx_nodal_flag ();
-  int* warpx_getJy_nodal_flag ();
-  int* warpx_getJz_nodal_flag ();
-  int* warpx_getRho_nodal_flag ();
-  int* warpx_getPhi_nodal_flag ();
-  int* warpx_getF_nodal_flag ();
-  int* warpx_getG_nodal_flag ();
-  int* warpx_get_edge_lengths_x_nodal_flag ();
-  int* warpx_get_edge_lengths_y_nodal_flag ();
-  int* warpx_get_edge_lengths_z_nodal_flag ();
-  int* warpx_get_face_areas_x_nodal_flag ();
-  int* warpx_get_face_areas_y_nodal_flag ();
-  int* warpx_get_face_areas_z_nodal_flag ();
-
-  amrex::Real** warpx_getChargeDensityCP (int lev, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  amrex::Real** warpx_getChargeDensityFP (int lev, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  int* warpx_getChargeDensityCPLoVects (int lev, int *return_size, int **ngrowvect);
-  int* warpx_getChargeDensityFPLoVects (int lev, int *return_size, int **ngrowvect);
-
-  amrex::Real** warpx_getPhiFP (int lev, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-
-  int* warpx_getPhiFPLoVects (int lev, int *return_size, int **ngrowvect);
-
-  amrex::Real** warpx_getFfieldCP (int lev, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  amrex::Real** warpx_getFfieldFP (int lev, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  int* warpx_getFfieldCPLoVects (int lev, int *return_size, int **ngrowvect);
-  int* warpx_getFfieldFPLoVects (int lev, int *return_size, int **ngrowvect);
-  amrex::Real** warpx_getGfieldCP (int lev, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  amrex::Real** warpx_getGfieldFP (int lev, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  int* warpx_getGfieldCPLoVects (int lev, int *return_size, int **ngrowvect);
-  int* warpx_getGfieldFPLoVects (int lev, int *return_size, int **ngrowvect);
-
-  amrex::Real** warpx_getEfieldCP_PML (int lev, int direction, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  amrex::Real** warpx_getEfieldFP_PML (int lev, int direction, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  amrex::Real** warpx_getBfieldCP_PML (int lev, int direction, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  amrex::Real** warpx_getBfieldFP_PML (int lev, int direction, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  amrex::Real** warpx_getCurrentDensityCP_PML (int lev, int direction, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  amrex::Real** warpx_getCurrentDensityFP_PML (int lev, int direction, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-
-  int* warpx_getEfieldCPLoVects_PML (int lev, int direction, int *return_size, int **ngrowvect);
-  int* warpx_getEfieldFPLoVects_PML (int lev, int direction, int *return_size, int **ngrowvect);
-  int* warpx_getBfieldCPLoVects_PML (int lev, int direction, int *return_size, int **ngrowvect);
-  int* warpx_getBfieldFPLoVects_PML (int lev, int direction, int *return_size, int **ngrowvect);
-  int* warpx_getCurrentDensityCPLoVects_PML (int lev, int direction, int *return_size, int **ngrowvect);
-  int* warpx_getCurrentDensityFPLoVects_PML (int lev, int direction, int *return_size, int **ngrowvect);
-
-  amrex::Real** warpx_getFfieldCP_PML (int lev, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  amrex::Real** warpx_getFfieldFP_PML (int lev, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  int* warpx_getFfieldCPLoVects_PML (int lev, int *return_size, int **ngrowvect);
-  int* warpx_getFfieldFPLoVects_PML (int lev, int *return_size, int **ngrowvect);
-  amrex::Real** warpx_getGfieldCP_PML (int lev, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  amrex::Real** warpx_getGfieldFP_PML (int lev, int *return_size, int *ncomps, int **ngrowvect, int **shapes);
-  int* warpx_getGfieldCPLoVects_PML (int lev, int *return_size, int **ngrowvect);
-  int* warpx_getGfieldFPLoVects_PML (int lev, int *return_size, int **ngrowvect);
-
-  int* warpx_getF_pml_nodal_flag ();
-  int* warpx_getG_pml_nodal_flag ();
-
 #ifdef __cplusplus
 }
 #endif