From 6c93d9fc13830d574c69ac7b166f5fbdb0809731 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Sat, 12 Aug 2023 11:17:38 -0700 Subject: Transition to pyAMReX (#3474) * pyAMReX: Build System * CI Updates (Changed Options) * Callback modernization (#4) * refactor callbacks.py * added binding code in `pyWarpX.cpp` to add or remove keys from the callback dictionary * minor PR cleanups Co-authored-by: Axel Huebl * Added Python level reference to fetch the multifabs (#3) * pyAMReX: Build System * Added Python level reference to Ex_aux * Now uses the multifab map * Fix typo Co-authored-by: Axel Huebl * Add initialization and finalize routines (#5) A basic PICMI input file will now run to completion. * Regression Tests: WarpX_PYTHON=ON * Update Imports to nD pyAMReX * IPO/LTO Control Although pybind11 relies heavily on IPO/LTO to create low-latency, small-binary bindings, some compilers will have troubles with that. Thus, we add a compile-time option to optionally disable it when needed. * Fix: Link Legacy WarpXWrappers.cpp * Wrap WarpX instance and start multi particle container * Fix test Python_pass_mpi_comm * Start wrapper for multiparticle container * Add return policy Co-authored-by: Axel Huebl * Update fields to use MultiFabs directly Remove EOL white space Removed old routines accessing MultiFabs Update to use "node_centered" * Fix compilation with Python * Update fields.py to use modified MultiFab tag names * Remove incorrect, unused code * Add function to extract the WarpX MultiParticleContainer * Complete class WarpXParticleContainer * Wrap functions getNprocs / getMyProc * restore `install___` callback API - could remove later if we want but should maintain backward compatibility for now * add `gett_new` and `getistep` functions wrappers; fix typos in `callbacks.py`; avoid error in getting `rho` from `fields.py` * Update callback call and `getNproc`/`getMyProc` function * Replace function add_n_particles * Fix setitem in fields.py for 1d and 2d * also update `gett_new()` in `_libwarpx.py` in case we want to keep that API * added binding for `WarpXParIter` - needed to port `libwarpx.depositChargeDensity()` which is an ongoing effort * Wrap function num_real_comp * added binding for `TotalNumberOfParticles` and continue progress on enabling 1d MCC test to run * add `SyncRho()` binding and create helper function in `libwarpx.depositChargeDensity` to manage scope of the particle iter * Clean up issues in fields.py * update bindings for `get_particle_structs` * Fix setitem in fields.py * switch order of initialization for particle container and particle iterator * switch deposit_charge loop to C++ code; bind `ApplyInverseVolumeScalingToChargeDensity` * move `WarpXParticleContainer.cpp` and `MultiParticleContainer.cpp` to new Particles folder * added binding for `ParticleBoundaryBuffer` * More fixes for fields.py * Fix: Backtraces from Python Add the Python executable name with an absolute path, so backtraces in AMReX work. See linked AMReX issue for details. * Cleaning * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix: Backtraces from Python Part II Do not add Python script name - it confuses the AMReX ParmParser to build its table. * Fix: CMake Dependencies for Wheel This fixes a racecondition during `pip_install`: it was possible that not all dimensions where yet build from pybind before we start packing them in the wheel for pip install. * MCC Test: Install Callbacks before Run Otherwise hangs in aquiring the gil during shutdown. * addition of `Python/pywarpx/particle_containers.py` and various associated bindings * Fix: CMake Superbuild w/ Shared AMReX We MUST build AMReX as a shared (so/dll/dylib) library, otherwise all the global state in it will cause split-brain situations, where our Python modules operate on different stacks. * add `clear_all()` to callbacks in order to remove all callbacks at finalize * add `-DWarpX_PYTHON=ON` to CI tests that failed to build * add `get_comp_index` and continue to port particle data bindings * Add AMReX Module as `libwarpx_so.amr` Attribute to pass through the already loaded AMReX module with the matching dimensionality to the simulation. * Fix for fields accounting for guard cells * Fix handling of ghost cells in fields * Update & Test: Particle Boundary Scraping * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * CI: Python Updates - modernize Python setups - drop CUDA 11.0 for good and go 11.3+ as documented already ``` Error #3246: Internal Compiler Error (codegen): "there was an error in verifying the lgenfe output!" ``` * CI: Python Updates (chmod) * Add support for cupy in fields.py * Add MultiFab reduction routines * CI: CUDA 11.3 is <= Ubuntu 20.04 * changed `AddNParticles` to take `amrex::Vector` arguments * setup.py: WarpX_PYTHON=ON * update various 2d and rz tests with new APIs * add `-DWarpX_PYTHON_IPO=OFF` to compile string for 2d and 3d Python CI tests to speed up linking * CI: -DpyAMReX_IPO=OFF * CI: -DpyAMReX_IPO=OFF actually adding `=OFF` * CI: Intel Python * CI: macOS Python Executable Ensure we always use the same `python3` executable, as specified by the `PATH` priority. * CMake: Python Multi-Config Build Add support for multi-config generators, especially on Windows. * __init__.py: Windows DLL Support Python 3.8+ on Windows: DLL search paths for dependent shared libraries Refs.: - https://github.com/python/cpython/issues/80266 - https://docs.python.org/3.8/library/os.html#os.add_dll_directory * CI: pywarpx Update our setup.py cannot install pyamrex yet as a dependency. * ABLASTR: `ablastr/export.H` Add a new header to export public globals that are not covered by `WINDOWS_EXPORT_ALL_SYMBOLS`. https://stackoverflow.com/questions/54560832/cmake-windows-export-all-symbols-does-not-cover-global-variables/54568678#54568678 * WarpX: EXPORT Globals in `.dll` files WarpX still uses a lot of globals: - `static` member variables - `extern` global variables These globals cannot be auto-exported with CMake's `WINDOWS_EXPORT_ALL_SYMBOLS` helper and thus we need to mark them manually for DLL export (and import) via the new ABLASTR `ablastr/export.H` helper macros. This starts to export symbols in the: - WarpX and particle container classes - callback hook database map - ES solver * CI: pywarpx Clang CXXFLAGS Down Move CXXFLAGS (`-Werror ...`) down until deps are installed. * GNUmake: Generate `ablastr/export.H` * CMake: More Symbol Exports for Windows * `WarpX-tests.ini`: Simplify Python no-IPO Also avoids subtle differences in compilation that increase compile time. * Update PICMI_inputs_EB_API.py for embedded_boundary_python_API CI test * Fix Python_magnetostatic_eb_3d * Update: Python_restart_runtime_components New Python APIs * Windows: no dllimport for now * CI: Skip `PYINSTALLOPTIONS` For Now * CMake: Dependency Bump Min-Versions for external packages picked up by `find_package`. * Fix F and G_fp names in fields.py * Tests: Disable `Python_pass_mpi_comm` * Wrappers: Cleanup * pyWarpX: Include Cleaning * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fields.py: Fix F and G Wrappers Correct MultiFab names (w/o components). * Remove unused in fields.py * Windows: New Export Headers - ABLASTR: `ablastr/export.H` - WarpX: `Utils/export.H` * removed `WarpInterface.py` since that functionality is now in `particle_containers.py`; removed parts of `WarpXWrappers.cpp` that have been ported to pyamrex * CMake: Link OBJECT Target PRIVATE * CMake: Remove OBJECT Target Simplify and make `app` link `lib` (default: static). Remove OBJECT target. * Fix in fields.py for the components index * Update get_particle_id/cpu As implemented in pyAMReX with https://github.com/AMReX-Codes/pyamrex/pull/165 * WarpX: Update for Private Constructor * Import AMReX Before pyd DLL Call Importing AMReX will add the `add_dll_directory` to a potentially shared amrex DLL on Windows. * Windows CI: Set PATH to amrex_Nd.dll * CMake: AMReX_INSTALL After Python In superbuild, Python can modify `AMReX_BUILD_SHARED_LIBS`. * Clang Win CI: Manually Install requirements Sporadic error is: ``` ... Installing collected packages: pyparsing, numpy, scipy, periodictable, picmistandard ERROR: Could not install packages due to an OSError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\hostedtoolcache\\windows\\Python\\3.11.4\\x64\\Lib\\site-packages\\numpy\\polynomial\\__init__.py' Consider using the `--user` option or check the permissions. ``` * Hopefully final fixes to fields.py * Update getProbLo/getProbHi * Set plasma length strength Co-authored-by: Remi Lehe * Fix fields method to remove CodeQL notice * Update Comments & Some Finalize * Move: set_plasma_lens_strength to MPC --------- Co-authored-by: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> Co-authored-by: David Grote Co-authored-by: Remi Lehe Co-authored-by: Dave Grote Co-authored-by: Roelof Groenewald Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- Python/pywarpx/PGroup.py | 182 ----------------------------------------------- 1 file changed, 182 deletions(-) delete mode 100644 Python/pywarpx/PGroup.py (limited to 'Python/pywarpx/PGroup.py') diff --git a/Python/pywarpx/PGroup.py b/Python/pywarpx/PGroup.py deleted file mode 100644 index 0dcb664e5..000000000 --- a/Python/pywarpx/PGroup.py +++ /dev/null @@ -1,182 +0,0 @@ -# Copyright 2017-2019 David Grote -# -# This file is part of WarpX. -# -# License: BSD-3-Clause-LBNL - -import numpy as np - -from ._libwarpx import libwarpx -from ._picmi import constants - - -class PGroup(object): - """Implements a class that has the same API as a warp ParticleGroup instance. - """ - - def __init__(self, igroup, ispecie, level=0): - self.igroup = igroup - self.ispecie = ispecie - self.level = level - self.ns = 1 # Number of species - - self.gallot() - - def name(self): - return 'WarpXParticleGroup' - - def gallot(self): - self.lebcancel_pusher = 0 # turns on/off cancellation of E+VxB within V push (logical type) - self.lebcancel = 0 # turns on/off cancellation of E+VxB before V push (logical type) - - self.sm = np.zeros(self.ns) # Species mass [kg] - self.sq = np.zeros(self.ns) # Species charge [C] - self.sw = np.ones(self.ns) # Species weight, (real particles per simulation particles) - - self.sid = np.arange(self.ns, dtype=int) # Global species index for each species - self.ndts = np.ones(self.ns, dtype=int) # Stride for time step advance for each species - self.ldts = np.ones(self.ns, dtype=int) # (logical type) - self.lvdts = np.ones(self.ns, dtype=int) # (logical type) - self.iselfb = np.zeros(self.ns, dtype=int) # Group number for particles that are affected by - # their own magnetic field, using the 1/gamma**2 - # approximation. The correction is not applied to - # group number -1. - self.fselfb = np.zeros(self.ns) # The scaling factor, vz. - self.l_maps = np.zeros(self.ns) - self.dtscale = np.ones(self.ns) # Scale factor applied to time step size for each - # species. Only makes sense in steaday and and - # transverse slice modes. - - self.limplicit = np.zeros(self.ns, dtype=int) # Flags implicit particle species (logical type) - self.iimplicit = np.full(self.ns, -1, dtype=int) # Group number for implicit particles - self.ldoadvance = np.ones(self.ns, dtype=int) # Flags whether particles are time advanced (logical type) - self.lboundaries = np.ones(self.ns, dtype=int) # Flags whether boundary conditions need to be applied (logical type) - self.lparaxial = np.zeros(self.ns, dtype=int) # Flags to turn on/off paraxial approximation (logical type) - - self.zshift = np.zeros(self.ns) - self.gamma_ebcancel_max = np.ones(self.ns) # maximum value allowed for ExB cancellation - - # --- Temporary fix - gchange = gallot - - def allocated(self, name): - return True - - def addspecies(self): - pass - - def getnpid(self): - return libwarpx.get_nattr() - npid = property(getnpid) - - def getnps(self): - return np.array([len(self.xp)], dtype='l') - nps = property(getnps) - - def getins(self): - return np.ones(self.ns, dtype='l') - ins = property(getins) - - def getipmax(self): - return np.array([0, len(self.xp)], dtype='l') - ipmax = property(getipmax) - - def getnpmax(self): - return self.nps.sum() - npmax = property(getnpmax) - - def getxp(self): - return libwarpx.get_particle_x(self.ispecie, self.level)[self.igroup] - xp = property(getxp) - - def getyp(self): - return libwarpx.get_particle_y(self.ispecie, self.level)[self.igroup] - yp = property(getyp) - - def getrp(self): - return libwarpx.get_particle_r(self.ispecie, self.level)[self.igroup] - rp = property(getrp) - - def getzp(self): - return libwarpx.get_particle_z(self.ispecie, self.level)[self.igroup] - zp = property(getzp) - - def getuxp(self): - return libwarpx.get_particle_ux(self.ispecie, self.level)[self.igroup] - uxp = property(getuxp) - - def getuyp(self): - return libwarpx.get_particle_uy(self.ispecie, self.level)[self.igroup] - uyp = property(getuyp) - - def getuzp(self): - return libwarpx.get_particle_uz(self.ispecie, self.level)[self.igroup] - uzp = property(getuzp) - - def getw(self): - return libwarpx.get_particle_weight(self.ispecie, self.level)[self.igroup] - - def getpid(self, id): - pid = libwarpx.get_particle_arrays(self.ispecie, id, self.level)[self.igroup] - return np.array([pid]).T - - def getgaminv(self): - uxp = self.getuxp() - uyp = self.getuyp() - uzp = self.getuzp() - return np.sqrt(1. - (uxp**2 + uyp**2 + uzp**2)/constants.c**2) - gaminv = property(getgaminv) - - def getex(self): - raise Exception('Particle E fields not supported') - ex = property(getex) - - def getey(self): - raise Exception('Particle E fields not supported') - ey = property(getey) - - def getez(self): - raise Exception('Particle E fields not supported') - ez = property(getez) - - def getbx(self): - raise Exception('Particle B fields not supported') - bx = property(getbx) - - def getby(self): - raise Exception('Particle B fields not supported') - by = property(getby) - - def getbz(self): - raise Exception('Particle B fields not supported') - bz = property(getbz) - - def gettheta(self): - return libwarpx.get_particle_theta(self.ispecie, self.level)[self.igroup] - theta = property(gettheta) - -class PGroups(object): - def __init__(self, ispecie=0, level=0): - self.ispecie = ispecie - self.level = level - - def setuppgroups(self): - xall = libwarpx.get_particle_x(self.ispecie, self.level) - self.ngroups = len(xall) - - self._pgroups = [] - for igroup in range(self.ngroups): - self._pgroups.append(PGroup(igroup, self.ispecie, self.level)) - - def __iter__(self): - self.setuppgroups() - for igroup in range(self.ngroups): - yield self._pgroups[igroup] - - def __getitem__(self, key): - self.setuppgroups() - return self._pgroups[key] - - def __len__(self): - self.setuppgroups() - return len(self._pgroups) -- cgit v1.2.3