aboutsummaryrefslogtreecommitdiff
path: root/Source/Parallelization (follow)
AgeCommit message (Collapse)AuthorFilesLines
2021-05-03Implement div(B) Cleaning With FDTD (#1829)Gravatar Edoardo Zoni 3-1/+72
* Implement div(B) Cleaning With FDTD * Add CI Test * Clean Up
2021-04-21Fix LoadBalance + MR: Rebuild Particle Masks (#1904)Gravatar Axel Huebl 1-2/+6
In MR simulations, particle masks are used to determine if a fine-patch particle is in the field gather and/or current deposition buffer regions near the coarse/fine boundaries. This is needed because some particles deposit to / gather from the coarse level, since they are near the level boundaries. On regriding during load-balancing, we can avoid to communicate the `MultiFab`s for those masks (gather and deposit) if we locally re-build the mask in the new distribution mapping. We forgot to trigger the rebuild, which caused illegal memory accesses in the particle evolve (partition) down the road.
2021-04-20Fix logic error when load balancing with more than 1 level. (#1903)Gravatar Andrew Myers 1-2/+6
* Fix logic error when load balancing with more than 1 level. * 0 -> false
2021-04-19Divergence cleaning for PSATD in PML (#1600)Gravatar Edoardo Zoni 1-0/+4
* Add div(E)/div(B) cleaning options for PSATD in PMLs * Pass missing flags to spectral solver in PML * Duplicate MPI exchange and communication functions for G * Use separate parameters for div cleaning in PMLs * Add asserts for features that are not implemented * Do not need to duplicate MPI exchange functions for G * Add short documentation for new input parameters * Set new parameters true by default with PSATD solver * Add CI test for PML div cleaning with PSATD * Use new syntax <diag_name>.intervals in new input file * Reset benchmark of new CI test * Always synchronize nodal points of G MultiFab * Fix few warnings in 2D build * Update Benchmark of pml_psatd_dive_divb_cleaning * Improve Documentation of warpx.do_pml_dive_cleaning Co-authored-by: Neïl Zaim <49716072+NeilZaim@users.noreply.github.com> * Improve Documentation of warpx.do_pml_divb_cleaning Co-authored-by: Neïl Zaim <49716072+NeilZaim@users.noreply.github.com> * Improve Abort Message * Clean Up Co-authored-by: Neïl Zaim <49716072+NeilZaim@users.noreply.github.com>
2021-04-15Hybrid Galilean: Same Guard Cells in z as Nodal (#1887)Gravatar Edoardo Zoni 1-11/+25
* Galilean Staggered: Same Guard Cells in z as Nodal * Update Benchmarks of Hybrid Tests
2021-04-12Add nodal synchronization of E and B fields in the main grid (#1817)Gravatar Neïl Zaim 1-0/+66
* Add nodal synchronization of E and B fields in the main grid * Update benchmarks * Update value in pml analysis file and put synchronization between PSATD push and FillBoundary * Update PML benchmarks and put the synchronization back in its original position * Bogus change in benchmark to trigger CI again * Update benchmarks
2021-04-08Fix Out-Of-Bound Access In Some Interpolation Functions (#1885)Gravatar Edoardo Zoni 1-12/+48
2021-04-05Compile-time option to use `algo.load_balance_costs_update = gpuclock` (#1838)Gravatar Michael E Rowan 1-12/+20
* compile time option for gpuclock eol move directives eol spaces assert fix GPU clock: CMake option * GPUCLOCK: ON by default Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
2021-03-31Move KernelTimer to Parallelization (#1856)Gravatar Michael E Rowan 1-0/+75
2021-03-29Regrid spectral solver during load balance (#1815)Gravatar Michael E Rowan 1-8/+76
* add PSATD support when remaking level * eol * update_with_rho * Update WarpXRegrid.cpp * Upate WarpXRegrid.cpp * Update WarpXRegrid.cpp * WIP * Cleanup * Remove * Add function to allocate spectral solver * EOL * spaces * Remove comment * Update documentation * eol * Remove function arguments * fix * Move lines into function * eol * amrex::Real * args
2021-03-29Add option for finite-order centering of currents (nodal to staggered) (#1763)Gravatar Edoardo Zoni 2-82/+184
* Start adding centering of current * Implement arbitrary order centering and split inputs * No need to define a brand new interpolation function * Update input file of hybrid CI tests * Clean up * Clean up more * Fix bug and clean up * Use current centering in two existing CI tests * Update documentation * Move Calls To UpdateCurrentNodalToStag Into SyncCurrent * Add Doxygen For New Function UpdateCurrentNodalToStag * Add Doxygen For New Functions Used For Stencil Coefficients * Finite-Order Centering of Currents Not Implemented With MR
2021-03-18Use Clear Naming Standard for ParmParse Variables (#1809)Gravatar Edoardo Zoni 1-4/+4
2021-03-17Replaced almost all nGrow with nGrowVect (#1801)Gravatar David Grote 1-2/+2
2021-03-04Implement averaged algo on staggered grids & merge spectral classes (#1544)Gravatar Edoardo Zoni 1-12/+17
* Refactor and clean up some spectral classes * Abort when current correction or Vay deposition are not implemented * Implement general equations for averaged Galilean * Allocate averaged MultiFabs also when aux_is_nodal=1 and do_nodal=0 * Allocate +ngextra guard cells also for averaged MultiFabs * Make alias MultiFabs for averaged aux data * With averaging, interpolate from avg_fp (not fp) to aux * Fix some limits of the coefficients * Fix bug causing NaNs in spectral coefficients * Add 2D CI test with same analysis as nodal test * Add 3D CI test with same analysis as nodal test * Add limit that was not covered (knorm=0 && knorm_c!=0 && nu=0) * Allocate T2_coef only if Galilean algorithm is used * Allocate X4_coef only if Galilean algorithm is used * Remove extra ghost cell from 'avg_fp' MultiFabs
2021-03-04Remove extra ghost cell used with momentum-conserving gathering (#1758)Gravatar Edoardo Zoni 3-30/+16
2021-03-02CI: Always with FFTW/PSATD (#1587)Gravatar Axel Huebl 2-70/+69
* CI: Always with FFTW/PSATD Reduce build time by always building with FFTW and unifying PSATD tests. This also adds coverage for the case that we forget to make `#ifdef WARPX_USE_PSATD` runtime blocks, which can potentially be seen in FDTD runs now. * Remove worker with environment USE_WARPX_PSATD * Fix warpx_interp: Unconditional PSATD Fix the `warpx_interp` function to properly enable/disable PSATD logic based on runtime logic. Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com> Co-authored-by: Remi Lehe <remi.lehe@normalesup.org> Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com>
2021-02-25Finite-order field centering: few small optimizations (#1714)Gravatar Edoardo Zoni 1-49/+52
* Use re-ordered stencil coefficients to optimize interpolation * Few more small optimizations and clean-up
2021-02-12Fix bug in loop over boxes in WarpX::UpdateAuxilaryDataStagToNodal (#1691)Gravatar Edoardo Zoni 2-17/+23
* Clarify loop over boxes in WarpX::UpdateAuxilaryDataStagToNodal * Loop over fabbox instead and pad with zeros beyond ghost cells * Clean up names of source and destination arrays
2021-02-03Reduced diagnostic for load balance efficiency (#1673)Gravatar Michael E Rowan 1-0/+5
* load balance efficiency reduced diagnostic * eol * minor * eol * docs * Docs -1 explanation Add explanation about load balance efficiency before costs are recorded * eol
2021-02-02Refactor finite-order interpolation functions for momentum-conserving field ↵Gravatar Edoardo Zoni 2-303/+196
gathering (#1653) * Unify high-order Fornberg interpolation functions * Add Doxygen documentation for new function * Unify also FDTD linear interpolation functions * Improve new implementation * Compute stencil coefficients only at initialization * Small clean-up and optimization
2021-02-02Always synchronize nodal points of PML MultiFabs (#1669)Gravatar Edoardo Zoni 1-0/+35
* Always synchronize nodal points of PML MultiFabs * Reset benchmark for CI test pml_x_psatd * Use new separate functions NodalSyncPML
2021-01-29make sure we redefine the tmp particle tiles when we load balance. (#1658)Gravatar Andrew Myers 1-0/+1
2021-01-27Take time step into account to compute guard cells for J and rho (#1607)Gravatar Edoardo Zoni 2-8/+27
* Use IntVect for ng_J and ng_rho * Compute guard cells for J and rho based on dt * Reset some CI benchmarks * Fix rebase commit * Add back +1 cell for rho: fix remaining out-of-bound accesses * Simplify ASSERTS using new interface of amrex::numParticlesOutOfRange
2021-01-26Start Embedded Boundary development (#1641)Gravatar Weiqun Zhang 1-0/+8
* Start Embedded Boundary development Modify the build system for embedded boundary (EB) support. Currently EB is a compile time option that must be explicitly enabled. For GNU Make, one can enable it with `USE_EB=TRUE`, whereas for CMake, `-DAMReX_EB=ON`. Later we could decide to enable EB by default with all regular geometry. Add a simple geometry initialization function, WarpX::InitEB. By default, the geometry is all regular. A few basic types such as box, cylinder, plane, sphere, etc. are supported via ParmParse runtime parameters. See `amrex/Src/EB/AMReX_EB2.cpp` for more details. Later, we could build more complex geometry using constructive solid geometry (CSG). ( https://en.wikipedia.org/wiki/Constructive_solid_geometry ) See `amrex/Tutorials/EB/GeometryGeneration` for an example of CSG. There is also a STL ( https://en.wikipedia.org/wiki/STL_(file_format) ) approach under development in AMReX. Add a new member, m_factory, to WarpX class. This object can be used to obtain geometry information such as whether a cell is cut, edge centroids, etc. Currently we are not using these factories to build MultiFabs for field data to embed the geometry information into the data containers. We could do that later if it is needed or it makes things more convenient. Nevertheless, this should be sufficient to start the EB development of the field solver. It's not clear to me yet how many ghost cells are needed for the geometry information. It's currently one, and can be adjusted. In the future, when particle and embedded boundary interaction is considered, we can use `amrex::FillSignedDistance` function to obtain signed distance function on the nodes. With that information, one should be able to determine where and when a particle collides with the embedded boundary. * resize factory vector * CMake: Require AMReX_EB for embedded boundaries * Add AMReX_Config.H et al. Explicit includes are most robust to make sure defines are set with future refactorings. * rename Factory fieldFactory Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
2021-01-21Define: _OPENMP -> AMREX_USE_OMP (#1520)Gravatar Axel Huebl 1-5/+5
* Define: _OPENMP -> AMREX_USE_OMP Replace the define check of `_OPENMP` with the explicit backend control of `AMREX_USE_OMP` for parallel constructs. Doing so avoids that we accidentially turn on OpenMP, e.g. if a dependency pulls it in for linear algebra, I/O, etc. This can led to confusion if the user explicitly requested a serial build. Also, we might want to use OpenMP functionality here and there for auxiliary functions w/o having to use the AMReX OpenMP backend, i.e. because we compile for GPUs. * Add missing amrex::Gpu::notInLaunchRegion
2021-01-15Fix bug in momentum-conserving interpolation function with MR (#1624)Gravatar Edoardo Zoni 1-6/+11
* Fix bug in momentum-conserving interpolation function * Reset checksum benchmark for CI test momentum-conserving-gather
2021-01-12Remove unused functions "ExchangeWithPml..." (#1616)Gravatar Edoardo Zoni 1-38/+0
2021-01-12Fix some other warnings (#1618)Gravatar Luca Fedeli 1-1/+1
* fix warnings * fixed bug
2021-01-11Fix some warnings (#1612)Gravatar Luca Fedeli 1-82/+94
* fixed some warnings * add missing files * fixed bug * fix several other warnings * add missing include * fixed bug * fixed bug * fixed bug * fixed bug
2020-12-22Fix: PSATD + CUDA build (#1595)Gravatar Axel Huebl 1-6/+9
2020-12-21Generalize some interpolation functions for MR (#1585)Gravatar Edoardo Zoni 2-157/+65
* Pass refinement ratio as argument to interpolation functions * Use new general interpolation function
2020-12-21Fix: PSATD Comm. Runtime Switch (FDTD) (#1588)Gravatar Axel Huebl 2-71/+67
* Fix: PSATD Comm. Runtime Switch (FDTD) Fix a missing PSATD runtime conversion in communication. This switch could have potentially influenced FDTD simulations if PSATD was compiled but not used. Co-authored-by: David Grote <grote1@llnl.gov> * WARPX_USE_PSATD in WarpX.cpp: Fixes Fix more logic compile-time bugs that should be runtime. * Read algo.maxwell_solver early * Fix: Use-After-Free Co-authored-by: David Grote <grote1@llnl.gov>
2020-12-18Fix bug in interpolation function used with MR (#1584)Gravatar Edoardo Zoni 1-2/+2
* Fix bug in interpolation function * Reset benchmarks
2020-12-11PSATD Runtime Control (#1300)Gravatar Axel Huebl 2-60/+61
* Docs: PSATD Runtime Option * Tests: PSATD Runtime Option Add new runtime option to PSATD regression test matrix. * PICMI: PSATD runtime option * Source: PSATD Runtime Option
2020-12-02Improve spectral solver on staggered grids (#1468)Gravatar Edoardo Zoni 4-7/+264
* Implement Galilean PSATD equations on staggered grids * Implement high-order interpolation in 2D/3D * Include missing header file and small clean-up * Fix bug for FDTD build * Small clean-up * Modify current correction for staggered grids * Implement comoving PSATD scheme (formulation with rho) * Fix single-precision builds * Do not implement rho-free formulation for comoving PSATD yet * Invert sign of comoving velocity to match Galilean convention * Fix two bugs in comoving PSATD algorithm * Update benchmark of CI test momentum-conserving-gather * Update benchmark of CI test PlasmaAccelerationMR * Update documentation * Clean up comoving PSATD class * Clean up comoving PSATD class (more) * Clean up comoving PSATD class (more) * Implement changes requested in PR review * Add 2D regression test for staggered Galilean PSATD * Add 2D regression test for staggered comoving PSATD * Unify input files for new CI tests to avoid duplication * Fully rebase benchmarks changes on development * Update benchmark of Galilean hybrid test after #1536
2020-11-02[mini-PR ] Fix an "unused variable" warning (#1488)Gravatar Luca Fedeli 1-2/+1
* fixed unused variable * eliminate superfluous include
2020-10-26Filter diagnostic rho (#1295)Gravatar David Grote 1-6/+12
* First cut of adding filtering of rho diagnostic - not yet working * Fixed the handling of parallel transfers in RhoFunctor when filtering is turned on * Fixed end of line space * In RhoFunctor, only do kspace filter with RZ and PSATD * PICMI analytic applied fields (#1306) * PICMI Added applied fields * PICMI fixed typo in Constants * Change picmistandard version requirement to 0.0.9 * Increment required picmi version Co-authored-by: Remi Lehe <remi.lehe@normalesup.org> * Updated checksums with filter of diagnostic rho * Fix comment in rho diagnostic * Updated LaserIonAcc2d benchmark after filtering rho * Update galilean_rz_psatd benchmark with filtering of rho Co-authored-by: Remi Lehe <remi.lehe@normalesup.org>
2020-10-23Fix Warning message <pmap.resize> (#1461)Gravatar Lígia Diana Amorim 1-1/+2
2020-10-19Replace wherever possible '.reset(new' with '= make_unique' (#1429)Gravatar Luca Fedeli 2-45/+52
* replace wherever possible .reset(new with = make_unique * fixed bug * fixed bug * revert WarpXOpenPMD.cpp to the original version * removed another .reset(new
2020-10-16New reduced diag: number of macroparticles (#1414)Gravatar NeilZaim 1-2/+2
* New reduced diag: number of macroparticles * Add parentheses in header line * Apply suggestions from code review Co-authored-by: Luca Fedeli <luca.fedeli.88@gmail.com> * Add some consts * Fuse some lines with std::ofstream Co-authored-by: Luca Fedeli <luca.fedeli.88@gmail.com>
2020-09-30Use fewer guard cells for deposition (#1368)Gravatar Edoardo Zoni 2-1/+14
* Use fewer guard cells for deposition * Update some CI benchmarks * Revert "Update some CI benchmarks" This reverts commit 2f40062557d75e3f6b2d3ba284332b8a92a0d404. * Check if shapes and guard cells are compatible only for rho * Do not allocate one extra guard cell for J * Add more relaxed check on shapes and guard cells for J
2020-09-14Clean up profiling labels (#1320)Gravatar Edoardo Zoni 1-3/+3
2020-09-06CKC: Use Enum instead consequently (#1299)Gravatar Axel Huebl 1-1/+3
Use the enums `MaxwellSolverAlgo` consequently for value of `WarpX::maxwell_solver_id`.
2020-08-24Fix some warnings (#1239)Gravatar Luca Fedeli 3-5/+14
This PR fixes the source of several compilation warnings. Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
2020-08-06Use psatd.n<xz>_guard, not psatd.n<xy>_guard, in 2D and RZ (#1237)Gravatar Edoardo Zoni 1-1/+5
2020-07-27Rename Input to `algo.maxwell_solver` (#1188)Gravatar Axel Huebl 2-4/+4
* Rename Input to `algo.maxwell_solver` Rename the input option `algo.maxwell_fdtd_solver` to `algo.maxwell_solver` and throw a useful error message for users using the old option. This is in preparation to unify FDTD and PSATD input control to reduce mutually exclusive binary variants of WarpX. * MW Solver Name: Update Examples Update all example to use the new algo.maxwell_solver option. * Fix typo in comutation of dt Co-authored-by: Remi Lehe <remi.lehe@normalesup.org>
2020-07-17Fixing compiler warnings (#1178)Gravatar Olga Shapoval 1-3/+3
* Removed unused variables and made 'vy=v_galilean[1]' declaration conditional on 3D case (this fixes compiler warnings) * Avoid unnecessary conversion from 'double' to 'int' (this fixes compiler warnings)
2020-07-08Averaged Galilean PSATD (#869)Gravatar Olga Shapoval 1-0/+125
* Read Galilean velocity * Prepare structures for Galilean solver * Started implementing Galilean equations * Analytical limits for X1, X2, X3, X4 coefficients added * Slight changes added * Added Galilean position pusher * Scale galilean velocity * Remove unneeded Abort * Fix Galilean pusher * Allocate Theta2 array * Fix definition of coefficients * Increase guard cells for Galilean * Add guard cell in particle exchange * Added modifications for PICSAR galilean branch * Averaged coefficients added * Type corrected * v_gal added to warpx_current_deposition * v_gal added to WarpXParticleContainer.H * Bug fixed - update particle x-position over one time step * Fix issues with merge from dev * Averaged fileds allocated on fine patch (Ex) * Preparation for merging dev into galilean. * remove TABs by hand * Removed a tab. * Redeclared v_galilean as a vector & related changed * Added an automated test * Moved v_galilean inside WarpX constructor * Added analysis script for the automated test * Changed name of the automated test to galilean_psatd * Added InitializeSpectralCoefficients method * Removed temporary comments * Averaged fields added to FiledGather * Added infinite order k[i] * Setting kx_mod( kz_mod) = 0 for index = nx/2(-nz/2) * Adding galilean shift * Implemented galilean shift * Changed method's name from GalileanShift to ShiftGalileanBoundary * Added doxygen string for ShiftGalileanBoundary * Removed never used method LowerCornerWithCentering * Removed temporary comments * Removed dt as a variable from DepositCharge method and its dependencies * Changing type of v_galilean from amrex::Vector to amrex::Array * Changed back ng_fft to be equal nox_fft * Changed v_galilean's type from amrex::vector to amrex::array * Removed type * Removed temporary comments * Added flag 'do_time_averaging' to swith from unaveraged to averaged Galilean PSATD * Updated doxygen * Small fix on using 'do_time_averaging' flag * Remove some commented Print statements * [skip ci] Further cleanup * Fix compilation * Guard cells update of the averaged E,B fields * Corrected included header file accordingly * Removed EOL * Removed EOL * Corrected path of the included header file * Updated choice of the spectral solver. * Cleanup. * use amrex::exp instead of std::exp * no backward FFT for avg fields if avg is off * Need to shift avg fields in MoveWindow * Further cleaning * Added 2D automated test for averaged PSATD * Added automated 2D and 3D tests for averaged PSATD * Removed comments * Added specifications for averaged Galilean PSATD tests. * Bug fixed - do FillBoundary of the averaged fields only if averaged is activated * Do shiftMF of the averaged fields only if fft_do_time_averaging=true * Add checksum json benchmarks for averaged Galilean PSATD * Add missing indentation * Add missing indentation * Updated automated analysis script * Updated diags period * Fixed bug: no backtransform of the averaged fields when WARPX_DIM_RZ is set * Fixed an uninitialized variable * Clean-up * Changed permissions for analysis_avg_*d.py * Compactified variables initialization via conditional assignment * Initialized averaged E_avg, B_avg fields * Updated automated test for 2D averaged galilean PSATD * Updated benchmark for 2D averaged galilean PSATD * Updated automated 2D averaged PSATD test & benchmark * Updated automated 3D averaged PSATD test & benchmark * Fixed typo * Updated benchmark averaged_galilean_3d_psatd * Initialize the averaged fields only if the averaged Galilean PSATD algorithm is enabled * Encreased time step for automated 3D averaged Galilean PSATD (to be equal c*dt=dz while dz/dx=3) * Updated 3D automated test: specified standard deviations `.ux_th, .uy_th, .uz_th` along each direction * Typo from the previous commit: enabled the averaged algorithm * Trying out another input parameters for 3D averaged automated test with v_gal = v_plasma * Update benchmark for previously added 3D averaged automated test * Update 2D averaged automated test and corresponding benchmark * Revert "Updated diags period" This reverts commit 7334729b05e96589e020c981efdb430ca095991d. * Cleanup: removed unwanted comment. * Added descriptions of the 2D and 3D automated tests for the averaged Galilean PSATD. * Updated value calculated via standard Galilean PSATD. * Encreased relative error tolerance for 3D automated test for the averaged Galilean PSATD. * Removed to avoid duplication since it already specified in the corresponding 2D/3D input scripts. * Removed unwanted empty lines * Added spaces after function's names * Removed unwanted empty line * Removed * Fixed indentation * Cleanup: removed #include <math.h> * Cleanup: removed empty lines and fixed indentation * Added 'AvgGalileanAlgorithm.cpp' to 'CMakeLists.txt' Co-authored-by: Remi Lehe <remi.lehe@normalesup.org> Co-authored-by: MaxThevenet <mthevenet@lbl.gov>
2020-06-12Add CMake Scripts (#759)Gravatar Axel Huebl 1-0/+6
* [skip travis] CMake: Add CMake Scripts This add CMake scripts for the CMake build system generator. * CUDA Support Achieved Fastest compile via: cmake .. -DENABLE_CUDA=ON -DCUDA_ARCH=6.0 * CMake: openPMD & QED * CMake: Dimension Support WarpX_DIMS=2,3,RZ * CCache & Out-of-Source Enable CCache when found and prevent in-root builds (in-source okay-ish). * CMake: CUDA C++ Language Minimum * Add new Files * CI: Migrate macOS to CMake * Fortran: Enable Because Transitive ... for now :) * CI: Migrate Linux ICC to CMake * CI: Verbose Make * CMake: Add Missing PSATD Sources * CMake: PSATD, Compute, Precision, Clean
2020-05-12Implement intervals parser for override_sync (#1008)Gravatar NeilZaim 1-2/+2