Import external E/B fields from openPMD files (#3584)

* Squashed commit of the following:

commit 5452b0a818e952fc8ac989a306b0ea83738243f1
Merge: a21dda0f 90b72e80
Author: Yin-YinjianZhao <yinjianzhao@lbl.gov>
Date:   Fri Sep 23 13:22:03 2022 -0700

    Merge branch 'development' of https://github.com/ECP-WarpX/WarpX into add_external_fields

commit a21dda0f1bad8ee8601104c3322631d5f385ffdd
Author: Yin-YinjianZhao <yinjianzhao@lbl.gov>
Date:   Fri Sep 23 13:13:23 2022 -0700

    Update

commit 3394416cc0f7dc1e4742249b469957bedbfd5e84
Author: Yin-YinjianZhao <yinjianzhao@lbl.gov>
Date:   Wed Aug 24 13:37:30 2022 -0700

    Update Examples/Tests/LoadExternalField/*

commit f0650b02e2f2d8d9719163ed5d4fd932a9d62a5b
Author: Yin-YinjianZhao <yinjianzhao@lbl.gov>
Date:   Wed Aug 24 12:41:11 2022 -0700

    parameters.rst added

commit a96a3eb3a181d6e1a649a43bc9eb2e00e4e0ef57
Author: Yin-YinjianZhao <yinjianzhao@lbl.gov>
Date:   Thu Aug 18 15:15:43 2022 -0700

    Add 3D test

commit caf213cdff02213cf0ff7389b399898bc76951b5
Merge: e85a7105 2b60afe8
Author: Yin-YinjianZhao <yinjianzhao@lbl.gov>
Date:   Thu Aug 18 12:02:35 2022 -0700

    Merge branch 'add_external_fields' of https://github.com/Yin-YinjianZhao/WarpX into add_external_fields

commit e85a7105dceb80bc8769062a306248f57ac0939d
Merge: d00ce279 642f6c0f
Author: Yin-YinjianZhao <yinjianzhao@lbl.gov>
Date:   Thu Aug 18 12:01:59 2022 -0700

    Merge branch 'development' of https://github.com/ECP-WarpX/WarpX into add_external_fields

commit 2b60afe876b6e4400e9f2676824ec94a4728c468
Author: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Date:   Thu Aug 18 19:01:58 2022 +0000

    [pre-commit.ci] auto fixes from pre-commit.com hooks

    for more information, see https://pre-commit.ci

commit d00ce2790ae56148bd717baa7c8e074a2d12a6bf
Merge: fbaf5226 c21244d1
Author: Yin-YinjianZhao <yinjianzhao@lbl.gov>
Date:   Thu Aug 18 12:01:42 2022 -0700

    minor

commit fbaf5226ee92444b618029ad56722f8e54918448
Author: Yin-YinjianZhao <yinjianzhao@lbl.gov>
Date:   Thu Aug 18 11:57:46 2022 -0700

    The automated test of RZ passed, cleaning up needed.

commit c21244d103e3eb245ebb1d3b9d2c4b016afd8671
Author: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Date:   Wed Aug 17 20:51:59 2022 +0000

    [pre-commit.ci] auto fixes from pre-commit.com hooks

    for more information, see https://pre-commit.ci

commit 0a52f2976b58cdbb48673387decce61aabf61561
Author: Yin-YinjianZhao <yinjianzhao@lbl.gov>
Date:   Wed Aug 17 13:51:40 2022 -0700

    Add RZ test, but has a bug.

commit 2c6e1537e9fa1f630134e64800b2d509088481b8
Merge: d3b6b0cd b4686616
Author: Yin-YinjianZhao <yinjianzhao@lbl.gov>
Date:   Wed Aug 10 13:37:01 2022 -0700

    Merge branch 'development' of https://github.com/ECP-WarpX/WarpX into add_external_fields

commit d3b6b0cd8eb34a5832bc8b439e96744f7a01a2ce
Author: Yin-YinjianZhao <yinjianzhao@lbl.gov>
Date:   Wed Aug 10 13:36:36 2022 -0700

    Do not use WARPX_EXTERNAL_FIELD

commit a32207ab2c4476feab0cb4b1e9a65a6181c7bf81
Author: Yin-YinjianZhao <yinjianzhao@lbl.gov>
Date:   Wed Aug 10 10:50:18 2022 -0700

    In the middle of clean up.

commit dcfcb02b9e053391c59cef50c692e84cf379fb8a
Author: Yin-YinjianZhao <yinjianzhao@lbl.gov>
Date:   Tue Aug 9 11:08:42 2022 -0700

    RZ OpenPMD bug fixed, but code is messy, and chunk load is not done.

commit dec17c35d025ac4d40dcaca4b68853959e176782
Author: Yin-YinjianZhao <yinjianzhao@lbl.gov>
Date:   Tue Aug 9 10:43:58 2022 -0700

    Has bug using RZ OpenMPD

commit b32e6efdd766f5986d66b213479d7c84587fca7e
Author: Yin-YinjianZhao <yinjianzhao@lbl.gov>
Date:   Fri Aug 5 13:17:04 2022 -0700

    Add OpenPMD 3D.

commit 7b7f48e2ae751d0393703c912116f5cb4d4dffbc
Author: Yin-YinjianZhao <yinjianzhao@lbl.gov>
Date:   Thu Aug 4 15:50:00 2022 -0700

    Revert "minor"

    This reverts commit 01e022958bad3057737e57fda4ecfaed22778b51.

commit 01e022958bad3057737e57fda4ecfaed22778b51
Author: Yin-YinjianZhao <yinjianzhao@lbl.gov>
Date:   Tue Aug 2 15:03:24 2022 -0700

    minor

commit 774e591959fce76f94dcb96ae17f43c71ee7b931
Author: Yin-YinjianZhao <yinjianzhao@lbl.gov>
Date:   Fri Jul 29 11:13:14 2022 -0700

    Make 1d work

commit 5db4e862104459e5511dad9869764866dc88e62d
Author: Yinjian Zhao <yin@Yinjians-MacBook-Air.local>
Date:   Fri Jul 29 10:27:03 2022 -0700

    Make xz working

commit 03c3d0062e301c1dd525a50f7ea84c2ccad6d679
Author: Yinjian Zhao <yin@Yinjians-MacBook-Air.local>
Date:   Fri Jul 29 09:38:43 2022 -0700

    Make rz workiong

commit 8864a1780606bfa68e842d6902d28a34da659dc8
Merge: 0c1c7b7b a514e793
Author: Yinjian Zhao <yin@Yinjians-MacBook-Air.local>
Date:   Wed Jun 22 14:02:05 2022 -0700

    Merge branch 'add_external_fields' of https://github.com/Yin-YinjianZhao/WarpX into add_external_fields

commit 0c1c7b7bda98549559d44125bb48e78ac6af24aa
Author: Yinjian Zhao <yin@Yinjians-MacBook-Air.local>
Date:   Wed Jun 22 14:01:42 2022 -0700

    Fix a bug.

commit a514e793cb5e30591debc475082cc48f9172ee25
Author: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Date:   Wed Jun 15 19:47:43 2022 +0000

    [pre-commit.ci] auto fixes from pre-commit.com hooks

    for more information, see https://pre-commit.ci

commit dd3c3749e2793c8c489652a8152c166def995514
Author: Yinjian Zhao <yin@Yinjians-MacBook-Air.local>
Date:   Wed Jun 15 12:47:23 2022 -0700

    Use txt file.

commit 61fb96d9219598d10dab6a729eb4db7278583d2e
Author: Yinjian Zhao <yin@Yinjians-MacBook-Air.local>
Date:   Thu May 26 13:51:58 2022 -0700

    Now the loaded external field does not match the provided external field data.

commit 7354a1abcd3b36e97d2145eff0c4b1da0a3aca27
Author: Yinjian Zhao <yin@Yinjians-MacBook-Air.local>
Date:   Wed May 18 15:21:15 2022 -0700

    Try to add new arguments in the doGatherShapeN function, such that external fields can be added.

commit 3a0db5fcbaf4298c7cf59ba4443f3cdf7a21094b
Author: Yinjian Zhao <yin@Yinjians-MacBook-Air.local>
Date:   Tue May 10 14:35:53 2022 -0700

    Changed to mfi.growntilebox() to include the guard cells.

commit 33e3393eb6bf769a2d92043c1012796b8212e653
Merge: 53201644 32fe8aac
Author: Yinjian Zhao <yin@Yinjians-MacBook-Air.local.dhcp.lbl.gov>
Date:   Thu May 5 13:37:24 2022 -0700

    A bug needs to be fixed related to the growth of cells.

commit 53201644322ae2a65c121a6f0559cda874edd98d
Author: Yinjian Zhao <yin@Yinjians-MacBook-Air.local.dhcp.lbl.gov>
Date:   Thu May 5 13:35:39 2022 -0700

    A bug needs to be fixed related to the growth of cells.

commit 32fe8aacc16a749910956e28120e31f1807f3bc1
Author: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Date:   Tue May 3 22:54:42 2022 +0000

    [pre-commit.ci] auto fixes from pre-commit.com hooks

    for more information, see https://pre-commit.ci

commit f434bff0e12643ba03fc76b1693c4875ef799fc6
Author: Yinjian Zhao <yin@Yinjians-MacBook-Air.local>
Date:   Tue May 3 15:54:25 2022 -0700

    In the middle.

commit 698e8f3e69dac5945b332c67570272bf2f4ae168
Author: Yinjian Zhao <yin@Yinjians-MacBook-Air.local.dhcp.lbl.gov>
Date:   Tue May 3 09:32:54 2022 -0700

    In the middle.

commit 9d9f88d59820e00eed0e962c8c8905d3f1964083
Author: Yinjian Zhao <yin@Yinjians-MacBook-Air.local>
Date:   Wed Apr 27 14:07:31 2022 -0700

    In the middle of reading data.

commit 3640e134fa0970b6b4ec0bf7f1c7f779c730b9a1
Author: Yinjian Zhao <yin@Yinjians-MacBook-Air.local>
Date:   Thu Apr 21 09:43:06 2022 -0700

    Draft.

commit 85e16cafbf65cf502b82943d5bae98c4e1d750df
Author: Yinjian Zhao <yin@Yinjians-MacBook-Air.local>
Date:   Mon Apr 18 15:15:22 2022 -0700

    Minor: change tag.

commit 0b7d2723c225cc4eed09eb58ad7f049d6baa650a
Author: Yinjian Zhao <yin@Yinjians-MacBook-Air.local>
Date:   Mon Apr 18 15:04:19 2022 -0700

    Allocate MultiFabs for the external E and B fields.

* Conditionalize compilation on openPMD support

* Switch bit-flag integer configuration to separate booleans

* Clean up and modernize allocation and reading logic

* Roll back addition of external fields in all calls down to doGatherShapeN

* Do away with separate MultiFABs for file-read field values, and just read into the main ones

* Check that solved and externally-prescribed fields are mutually exclusive

* Clean up compilation issues

* Support GPUs

* Correct data sets URLs

* Fix compilation for 1D

* Fix some GPU issues

* Extract elements of various random little types that aren't GPU-friendly

* Run RZ test along with the others, not with the 2D tests

* Fix complaint about unused variable

* Fix complaints about unused variable from 1D

* Clean up 1D warning issues harder

* Clone the example dataset repository directly, like warpx-data

* Allow composition of external and internal fields

* Go back to reading external field into a separate MultiFAB

* Vaguely working for lab frame electrostatic case

* Fix copy-paste errors

* Replace magic constants by enumerators

* Get indexing order right for RZ

* Put external field contribution in the right place

* Fix GPU and unused variable issues

* Match renamed input variable

* Update input parameters

* reset benchmark for LoadExternalField3D because data actually get loaded now

* reset benchmark for LoadExternalFieldRZ because data actually get loaded now

* Drop unused includes

* Read data once for all boxes

* Guard against unimplemented mesh refinement support

* Rename function to slightly better reflect substance

* Use existing linear interpolation routines

* Specify type argument for interpolation to avoid mixture

* Account for Array4 indexing in Fortran order

* Address Edoardo's review comments

* Support load balancing

* Add assertions of assumptions about input file format

* Move assertion about max levels to earlier in initialization

* Error in XZ case since it's untested

* Don't pretend implementation covers XZ

* Name variables to correspond to dimensionality

* Reword comment per request

* Correct typos

* Add issue numbers for different unimplemented bits, and guard 1D case

* Fix comment about RZ ParallelFor

* Fix reStructuredText syntax

* Shift 2D case to the unimplemented path to avoid unused parameter/variable warnings

* Use 2020+ Curl Command

* Support older curl

---------

Co-authored-by: Tools <warpx@lbl.gov>
Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

1 files changed, 6 insertions, 0 deletions

diff --git a/Source/Parallelization/WarpXRegrid.cpp b/Source/Parallelization/WarpXRegrid.cpp
index 3c90a07cb..16ecd49ce 100644
--- a/Source/Parallelization/WarpXRegrid.cpp
+++ b/Source/Parallelization/WarpXRegrid.cpp
@@ -169,6 +169,12 @@ WarpX::RemakeLevel (int lev, Real /*time*/, const BoxArray& ba, const Distributi
         {
             RemakeMultiFab(Bfield_fp[lev][idim], dm, true);
             RemakeMultiFab(Efield_fp[lev][idim], dm, true);
+            if (add_external_B_field) {
+                RemakeMultiFab(Bfield_fp_external[lev][idim], dm, true);
+            }
+            if (add_external_E_field) {
+                RemakeMultiFab(Efield_fp_external[lev][idim], dm, true);
+            }
             RemakeMultiFab(current_fp[lev][idim], dm, false);
             RemakeMultiFab(current_store[lev][idim], dm, false);
             if (current_deposition_algo == CurrentDepositionAlgo::Vay) {