Tool: update AMReX dependency (#1710)

* Tool: update AMReX dependency

We currently pull the `development` branch of AMReX in CMake builds.

This is problematic in central workflows:
- checking out a release might not compile
- manual intervention is needed by users
- builds are not as reproducibile as they could be
- CI fails in a surprising wary in WarpX if a temporary bug is lands
  in AMReX' `development`

Instead, we can bump the AMReX requirement periodically in a PR.
This migh be the case when we:
- need new features or bug fixes
- do a release

Manual updates guarantee that we see problems with updates from
AMReX in the moment they are introduced - during a PR that changes
the dependency to AMReX.

* Tool: update PICSAR dependency

See AMReX' updater description.

* Docs: Dependencies & Releases

Update and add workflows.

* PICSAR & AMReX: Bump Version to HEAD

Bump the PICSAR and AMReX versions to the latest head.
Executed with:
```
./Tools/Release/updateAMReX.py
./Tools/Release/updatePICSAR.py
```

* Python Tools: Cleanup

* Fix typos

Co-authored-by: Luca Fedeli <luca.fedeli@for.unipi.it>

Co-authored-by: Luca Fedeli <luca.fedeli@for.unipi.it>

11 files changed, 339 insertions, 103 deletions

diff --git a/Docs/source/developers/developers.rst b/Docs/source/developers/developers.rst
index 94777c4e0..a1ffb7c65 100644
--- a/Docs/source/developers/developers.rst
+++ b/Docs/source/developers/developers.rst
@@ -25,5 +25,5 @@ Our Doxygen API documentation in classic formatting `is located here <../_static
    testing
    documentation
    performance_tests
-   workflow
+   release
    checksum
diff --git a/Docs/source/developers/release.rst b/Docs/source/developers/release.rst
new file mode 100644
index 000000000..336c19ca6
--- /dev/null
+++ b/Docs/source/developers/release.rst
@@ -0,0 +1,55 @@
+.. _developers-release:
+
+Dependencies & Releases
+=======================
+
+Update WarpX' Core Dependencies
+-------------------------------
+
+WarpX has direct dependencies on AMReX and PICSAR, which we periodically update.
+
+The following scripts automate this workflow, in case one needs a newer commit of AMReX or PICSAR between releases:
+
+.. code-block:: sh
+
+   ./Tools/Release/updateAMReX.py
+   ./Tools/Release/updatePICSAR.py
+
+
+Create a new WarpX release
+--------------------------
+
+WarpX has one release per month.
+The version number is set at the beginning of the month and follows the format ``YY.MM``.
+
+In order to create a GitHub release, you need to:
+
+ 1. Create a new branch from ``development`` and update the version number in all source files.
+    We usually wait for the AMReX release to be tagged first, then we also point to its tag.
+
+    There is a script for updating core dependencies of WarpX and the WarpX version:
+
+    .. code-block:: sh
+
+       ./Tools/Release/updateAMReX.py
+       ./Tools/Release/updatePICSAR.py
+
+       ./Tools/Release/newVersion.sh
+
+    For a WarpX release, ideally a *git tag* of AMReX & PICSAR shall be used instead of an unnamed commit.
+
+    Then open a PR, wait for tests to pass and then merge.
+
+ 2. **Local Commit** (Optional): at the moment, ``@ax3l`` is managing releases and signs tags (naming: ``YY.MM``) locally with his GPG key before uploading them to GitHub.
+
+    **Publish**: On the `GitHub Release page <https://github.com/ECP-WarpX/WarpX/releases>`__, create a new release via ``Draft a new release``.
+    Either select the locally created tag or create one online (naming: ``YY.MM``) on the merged commit of the PR from step 1.
+
+    In the *release description*, please specify the compatible versions of dependencies (see previous releases), and provide info on the content of the release.
+    In order to get a list of PRs merged since last release, you may run
+
+    .. code-block:: sh
+
+       git log --since=<last-release-tag> | grep -A 3 "Author: " | grep -B 1 "\-\-" | sed '/--/d' | sed -e 's/^    /- /'
+
+ 3. Optional/future: create a ``release-<version>`` branch, write a changelog, and backport bug-fixes for a few days.
diff --git a/Docs/source/developers/workflow.rst b/Docs/source/developers/workflow.rst
deleted file mode 100644
index 226cf77f3..000000000
--- a/Docs/source/developers/workflow.rst
+++ /dev/null
@@ -1,32 +0,0 @@
-.. _developers-workflow:
-
-Workflow
-========
-
-Create a new Github release
----------------------------
-
-WarpX has one release per month.
-In order to create a release, you need to:
-
- 1. Create a new branch from ``development`` and update the version number in all source files.
-    There is a script for that, so you can do:
-
-    .. code-block:: sh
-
-       cd Tools/DevUtils/
-       ./update_release.sh # This replaces the old version number with the new one.
-
-    Then open a PR, as usual. NOTE: do not merge this PR before step 2 is completed.
-
- 2. Click the ``Draft a new release`` button at https://github.com/ECP-WarpX/WarpX/releases and follow instructions.
-    Please specify the compatible versions of dependencies (see previous releases), and provide info on the content of the release.
-    In order to get a list of PRs merged since last release, you may run
-
-    .. code-block:: sh
-
-       git log --since=<date> | grep -A 3 "Author: " | grep -B 1 "\-\-" | sed '/--/d' | sed -e 's/^    /- /'
-
-    where ``<date>`` is the date of the last release, say ``2020-05-01`` if the last release was on May 1, 2020.
-
- 3. Optional: create a ``release-<version>`` branch, write a changelog, and backport bug-fixes for a few days.
diff --git a/Tools/DevUtils/update_release.sh b/Tools/DevUtils/update_release.sh
deleted file mode 100755
index 74e88cbf8..000000000
--- a/Tools/DevUtils/update_release.sh
+++ /dev/null
@@ -1,65 +0,0 @@
-#!/usr/bin/env bash
-
-# Copyright 2020 Maxence Thevenet
-#
-# This file is part of WarpX.
-#
-# License: BSD-3-Clause-LBNL
-
-
-# This script updates release version number in all source files.
-#
-# updates occurences like "version = '??.??'" where ??.?? is the version number
-# updates occurences like "WarpX v??.??" where ??.?? is the version number
-#
-# Requirements:
-# - gnu grep (default grep on MacOS does not have same -P option)
-# - gnu sed  (default grep on MacOS does not have same -i option)
-
-set -e
-
-# Get old tag number from git
-git_tag=`git describe --tags`
-old_release_number="${git_tag%%-*}"
-
-# Construct new tag number from current date
-d=`date +%Y.%m`
-new_release_number=${d:2}
-
-echo "Replace release number $old_release_number by $new_release_number"
-
-# Loop over files and sed patterns with version number
-pattern="\.c$|\.cpp$|\.F90$|\.h$|\.H$|\.ini$|\.md$|\.py$|"\
-"\.rst$|\.sh$|\.tex$|\.txt$|\.xml$|\.yml$|"\
-"CMakeLists\.txt|inputs"
-for i in $(find ../.. \
-                -not -path "../../.git/*"   \
-                -not -path "../../.idea/*"  \
-                -not -path "../../Docs/source/api/*" \
-                -not -path "../../Docs/build/*" \
-                -not -path "../../Docs/doxyxml/*" \
-                -not -path "*wp_parse*" \
-                -not -path "../../tmp_build_dir/*" \
-                -type f | \
-           grep -P "${pattern}")
-do
-    echo $i
-    # update occurences like "version = '??.??'" where ??.?? is the version number
-    # Note: sleep is required due to a bug in sed: without, the file
-    # permissions are modified
-    sed -i "s/version = "\'"$old_release_number"\'"/version = "\'"$new_release_number"\'"/g" $i && sleep .001
-    # update occurences like "WarpX v??.??" where ??.?? is the version number
-    sed -i "s/WarpX v$old_release_number/WarpX v$new_release_number/g" $i && sleep .001
-done
-
-echo ""
-echo "WARNING: Remaining occurences of $old_release_number are listed below."
-echo "         Is this expected? Or should these be updated too?"
-echo ""
-git grep "$old_release_number" . || echo ""
-echo "In order to get a list of PRs merged since date <date>, you can run:"
-echo "git log --since=<date> | grep -A 3 \"Author: \" | grep -B 1 \"\-\-\" | sed '/--/d' | sed -e 's/^    /- /'"
-echo "where <date> is replaced by the date since last relate, say 2020-02-02"
-# The actual command is below (commented), the one in echo above
-# has escape characters for printing purpose.
-# git log --since=<date> | grep -A 3 "Author: " | grep -B 1 "\-\-" | sed '/--/d' | sed -e 's/^    /- /'
diff --git a/Tools/Release/newVersion.sh b/Tools/Release/newVersion.sh
index e0563e0ff..0d23a36c7 100755
--- a/Tools/Release/newVersion.sh
+++ b/Tools/Release/newVersion.sh
@@ -2,7 +2,8 @@
 #
 # Copyright 2021 Axel Huebl
 #
-# This file is part of WarpX.#
+# This file is part of WarpX.
+#
 
 # This file is a maintainer tool to bump the versions inside WarpX'
 # source directory at all places where necessary.
diff --git a/Tools/Release/requirements.txt b/Tools/Release/requirements.txt
new file mode 100644
index 000000000..5ddc7784c
--- /dev/null
+++ b/Tools/Release/requirements.txt
@@ -0,0 +1 @@
+requests~=2.0
diff --git a/Tools/Release/updateAMReX.py b/Tools/Release/updateAMReX.py
new file mode 100755
index 000000000..e3557c347
--- /dev/null
+++ b/Tools/Release/updateAMReX.py
@@ -0,0 +1,138 @@
+#!/usr/bin/env python3
+#
+# Copyright 2021 Axel Huebl
+#
+# This file is part of WarpX.
+#
+
+# This file is a maintainer tool to bump the AMReX version that we pull in
+# when building WarpX.
+#
+import datetime
+from pathlib import Path
+import re
+import requests
+import sys
+
+
+# Maintainer Inputs ###########################################################
+
+print("""Hi there, this is a WarpX maintainer tool to update the source
+code of WarpX to a new commit/release of AMReX.
+For it to work, you need write access on the source directory and
+you should be working in a clean git branch without ongoing
+rebase/merge/conflict resolves and without unstaged changes.""")
+
+# check source dir
+REPO_DIR = Path(__file__).parent.parent.parent.absolute()
+print(f"\nYour current source directory is: {REPO_DIR}")
+
+REPLY = input("Are you sure you want to continue? [y/N] ")
+print()
+if not REPLY in ["Y", "y"]:
+    print("You did not confirm with 'y', aborting.")
+    sys.exit(1)
+
+
+# Current Versions ############################################################
+
+# AMReX development HEAD
+amrex_gh = requests.get('https://api.github.com/repos/AMReX-Codes/amrex/commits/development')
+amrex_HEAD = amrex_gh.json()["sha"]
+
+# WarpX references to AMReX: cmake/dependencies/AMReX.cmake
+amrex_cmake_path = str(REPO_DIR.joinpath("cmake/dependencies/AMReX.cmake"))
+#   branch/commit/tag (git fetcher) version
+#     set(WarpX_amrex_branch "development" ...
+amrex_branch = f"unknown (format issue in {amrex_cmake_path})"
+with open(amrex_cmake_path, encoding='utf-8') as f:
+    r_minimal = re.findall(r'.*set\(WarpX_amrex_branch\s+"(.+)"\s+.*',
+                           f.read(), re.MULTILINE)
+    if len(r_minimal) >= 1:
+        amrex_branch = r_minimal[0]
+
+#   minimal (external) version
+#     find_package(AMReX YY.MM CONFIG ...
+amrex_minimal = f"unknown (format issue in {amrex_cmake_path})"
+with open(amrex_cmake_path, encoding='utf-8') as f:
+    r_minimal = re.findall(r'.*find_package\(AMReX\s+(.+)\s+CONFIG\s+.*',
+                           f.read(), re.MULTILINE)
+    if len(r_minimal) >= 1:
+        amrex_minimal = r_minimal[0]
+
+
+# Ask for new #################################################################
+
+print("""We will now run a few sed commands on your source directory.
+Please answer the following questions about the version number
+you want to require from AMReX:\n""")
+
+print(f"Currently, WarpX builds against this AMReX commit/branch/sha: {amrex_branch}")
+print(f"AMReX HEAD commit (development branch): {amrex_HEAD}")
+amrex_new_branch = input(f"Update AMReX commit/branch/sha: ").strip()
+if not amrex_new_branch:
+    amrex_new_branch = amrex_branch
+    print(f"--> Nothing entered, will keep: {amrex_branch}")
+print()
+
+print(f"Currently, a pre-installed AMReX is required at least at version: {amrex_minimal}")
+today = datetime.date.today().strftime("%y.%m")
+amrex_new_minimal = input(f"New minimal AMReX version (e.g. {today})? ").strip()
+if not amrex_new_minimal:
+    amrex_new_minimal = amrex_minimal
+    print(f"--> Nothing entered, will keep: {amrex_minimal}")
+
+print()
+print(f"New AMReX commit/branch/sha: {amrex_new_branch}")
+print(f"New minimal AMReX version:   {amrex_new_minimal}\n")
+
+REPLY = input("Is this information correct? Will now start updating! [y/N] ")
+print()
+if not REPLY in ["Y", "y"]:
+    print("You did not confirm with 'y', aborting.")
+    sys.exit(1)
+
+
+# Updates #####################################################################
+
+# run_test.sh (used also for Azure Pipelines)
+run_test_path = str(REPO_DIR.joinpath("run_test.sh"))
+with open(run_test_path, encoding='utf-8') as f:
+    run_test_content = f.read()
+    #   branch/commit/tag (git fetcher) version
+    #     git clone --branch development https://github.com/AMReX-Codes/amrex.git
+    run_test_content = re.sub(
+        r'(.*git\s+clone\s+\-\-branch\s+)(.+)(\s+https://github\.com/AMReX\-Codes/amrex\.git)',
+        r'\g<1>{}\g<3>'.format(amrex_new_branch),
+        run_test_content, flags = re.MULTILINE)
+
+with open(run_test_path, "w", encoding='utf-8') as f:
+    f.write(run_test_content)
+
+# WarpX references to AMReX: cmake/dependencies/AMReX.cmake
+with open(amrex_cmake_path, encoding='utf-8') as f:
+    amrex_cmake_content = f.read()
+
+    #   branch/commit/tag (git fetcher) version
+    #     set(WarpX_amrex_branch "development" ...
+    amrex_cmake_content = re.sub(
+        r'(.*set\(WarpX_amrex_branch\s+")(.+)("\s+.*)',
+        r'\g<1>{}\g<3>'.format(amrex_new_branch),
+        amrex_cmake_content, flags = re.MULTILINE)
+
+    #   minimal (external) version
+    #     find_package(AMReX YY.MM CONFIG ...
+    amrex_cmake_content = re.sub(
+        r'(.*find_package\(AMReX\s+)(.+)(\s+CONFIG\s+.*)',
+        r'\g<1>{}\g<3>'.format(amrex_new_minimal),
+        amrex_cmake_content, flags = re.MULTILINE)
+
+with open(amrex_cmake_path, "w", encoding='utf-8') as f:
+    f.write(amrex_cmake_content)
+
+
+# Epilogue ####################################################################
+
+print("""Done. Please check your source, e.g. via
+  git diff
+now and commit the changes if no errors occurred.""")
diff --git a/Tools/Release/updatePICSAR.py b/Tools/Release/updatePICSAR.py
new file mode 100755
index 000000000..a3c3f79fc
--- /dev/null
+++ b/Tools/Release/updatePICSAR.py
@@ -0,0 +1,138 @@
+#!/usr/bin/env python3
+#
+# Copyright 2021 Axel Huebl
+#
+# This file is part of WarpX.
+#
+
+# This file is a maintainer tool to bump the PICSAR version that we pull in
+# when building WarpX.
+#
+import datetime
+from pathlib import Path
+import re
+import requests
+import sys
+
+
+# Maintainer Inputs ###########################################################
+
+print("""Hi there, this is a WarpX maintainer tool to update the source
+code of WarpX to a new commit/release of PICSAR.
+For it to work, you need write access on the source directory and
+you should be working in a clean git branch without ongoing
+rebase/merge/conflict resolves and without unstaged changes.""")
+
+# check source dir
+REPO_DIR = Path(__file__).parent.parent.parent.absolute()
+print(f"\nYour current source directory is: {REPO_DIR}")
+
+REPLY = input("Are you sure you want to continue? [y/N] ")
+print()
+if not REPLY in ["Y", "y"]:
+    print("You did not confirm with 'y', aborting.")
+    sys.exit(1)
+
+
+# Current Versions ############################################################
+
+# PICSAR development HEAD
+PICSAR_gh = requests.get('https://api.github.com/repos/ECP-WarpX/picsar/commits/development')
+PICSAR_HEAD = PICSAR_gh.json()["sha"]
+
+# WarpX references to PICSAR: cmake/dependencies/PICSAR.cmake
+PICSAR_cmake_path = str(REPO_DIR.joinpath("cmake/dependencies/PICSAR.cmake"))
+#   branch/commit/tag (git fetcher) version
+#     set(WarpX_picsar_branch "development" ...
+PICSAR_branch = f"unknown (format issue in {PICSAR_cmake_path})"
+with open(PICSAR_cmake_path, encoding='utf-8') as f:
+    r_minimal = re.findall(r'.*set\(WarpX_picsar_branch\s+"(.+)"\s+.*',
+                           f.read(), re.MULTILINE)
+    if len(r_minimal) >= 1:
+        PICSAR_branch = r_minimal[0]
+
+#   minimal (external) version
+#     find_package(PICSAR YY.MM CONFIG ...
+PICSAR_minimal = f"unknown (format issue in {PICSAR_cmake_path})"
+with open(PICSAR_cmake_path, encoding='utf-8') as f:
+    r_minimal = re.findall(r'.*find_package\(PICSAR\s+(.+)\s+CONFIG\s+.*',
+                           f.read(), re.MULTILINE)
+    if len(r_minimal) >= 1:
+        PICSAR_minimal = r_minimal[0]
+
+
+# Ask for new #################################################################
+
+print("""We will now run a few sed commands on your source directory.
+Please answer the following questions about the version number
+you want to require from PICSAR:\n""")
+
+print(f"Currently, WarpX builds against this PICSAR commit/branch/sha: {PICSAR_branch}")
+print(f"PICSAR HEAD commit (development branch): {PICSAR_HEAD}")
+PICSAR_new_branch = input(f"Update PICSAR commit/branch/sha: ").strip()
+if not PICSAR_new_branch:
+    PICSAR_new_branch = PICSAR_branch
+    print(f"--> Nothing entered, will keep: {PICSAR_branch}")
+print()
+
+print(f"Currently, a pre-installed PICSAR is required at least at version: {PICSAR_minimal}")
+today = datetime.date.today().strftime("%y.%m")
+PICSAR_new_minimal = input(f"New minimal PICSAR version (e.g. {today})? ").strip()
+if not PICSAR_new_minimal:
+    PICSAR_new_minimal = PICSAR_minimal
+    print(f"--> Nothing entered, will keep: {PICSAR_minimal}")
+
+print()
+print(f"New PICSAR commit/branch/sha: {PICSAR_new_branch}")
+print(f"New minimal PICSAR version:   {PICSAR_new_minimal}\n")
+
+REPLY = input("Is this information correct? Will now start updating! [y/N] ")
+print()
+if not REPLY in ["Y", "y"]:
+    print("You did not confirm with 'y', aborting.")
+    sys.exit(1)
+
+
+# Updates #####################################################################
+
+# run_test.sh (used also for Azure Pipelines)
+run_test_path = str(REPO_DIR.joinpath("run_test.sh"))
+with open(run_test_path, encoding='utf-8') as f:
+    run_test_content = f.read()
+    #   branch/commit/tag (git fetcher) version
+    #     git clone --branch development https://github.com/PICSAR-Codes/PICSAR.git
+    run_test_content = re.sub(
+        r'(.*git\s+clone\s+\-\-branch\s+)(.+)(\s+https://github\.com/ECP\-WarpX/picsar\.git)',
+        r'\g<1>{}\g<3>'.format(PICSAR_new_branch),
+        run_test_content, flags = re.MULTILINE)
+
+with open(run_test_path, "w", encoding='utf-8') as f:
+    f.write(run_test_content)
+
+# WarpX references to PICSAR: cmake/dependencies/PICSAR.cmake
+with open(PICSAR_cmake_path, encoding='utf-8') as f:
+    PICSAR_cmake_content = f.read()
+
+    #   branch/commit/tag (git fetcher) version
+    #     set(WarpX_picsar_branch "development" ...
+    PICSAR_cmake_content = re.sub(
+        r'(.*set\(WarpX_picsar_branch\s+")(.+)("\s+.*)',
+        r'\g<1>{}\g<3>'.format(PICSAR_new_branch),
+        PICSAR_cmake_content, flags = re.MULTILINE)
+
+    #   minimal (external) version
+    #     find_package(PICSAR YY.MM CONFIG ...
+    PICSAR_cmake_content = re.sub(
+        r'(.*find_package\(PICSAR\s+)(.+)(\s+CONFIG\s+.*)',
+        r'\g<1>{}\g<3>'.format(PICSAR_new_minimal),
+        PICSAR_cmake_content, flags = re.MULTILINE)
+
+with open(PICSAR_cmake_path, "w", encoding='utf-8') as f:
+    f.write(PICSAR_cmake_content)
+
+
+# Epilogue ####################################################################
+
+print("""Done. Please check your source, e.g. via
+  git diff
+now and commit the changes if no errors occurred.""")
diff --git a/cmake/dependencies/AMReX.cmake b/cmake/dependencies/AMReX.cmake
index adc61cad5..ab8c23d89 100644
--- a/cmake/dependencies/AMReX.cmake
+++ b/cmake/dependencies/AMReX.cmake
@@ -185,7 +185,7 @@ set(WarpX_amrex_src ""
 set(WarpX_amrex_repo "https://github.com/AMReX-Codes/amrex.git"
     CACHE STRING
     "Repository URI to pull and build AMReX from if(WarpX_amrex_internal)")
-set(WarpX_amrex_branch "development"
+set(WarpX_amrex_branch "e611121acca7696320f3757982de857cbab74926"
     CACHE STRING
     "Repository branch for WarpX_amrex_repo if(WarpX_amrex_internal)")
 
diff --git a/cmake/dependencies/PICSAR.cmake b/cmake/dependencies/PICSAR.cmake
index 96c852b2a..7e496c0e8 100644
--- a/cmake/dependencies/PICSAR.cmake
+++ b/cmake/dependencies/PICSAR.cmake
@@ -98,7 +98,7 @@ if(WarpX_QED)
     set(WarpX_picsar_repo "https://github.com/ECP-WarpX/picsar.git"
         CACHE STRING
         "Repository URI to pull and build PICSAR from if(WarpX_picsar_internal)")
-    set(WarpX_picsar_branch "development"
+    set(WarpX_picsar_branch "b35f07243c51ac35d47857fe36f0aafb6b517955"
         CACHE STRING
         "Repository branch for WarpX_picsar_repo if(WarpX_picsar_internal)")
 
diff --git a/run_test.sh b/run_test.sh
index 5b998b751..28e5082f7 100755
--- a/run_test.sh
+++ b/run_test.sh
@@ -48,9 +48,9 @@ cd test_dir
 echo "cd $PWD"
 
 # Clone PICSAR and AMReX
-git clone --branch development https://github.com/AMReX-Codes/amrex.git
+git clone --branch e611121acca7696320f3757982de857cbab74926 https://github.com/AMReX-Codes/amrex.git
 # Use QED brach for QED tests
-git clone --branch development https://github.com/ECP-WarpX/picsar.git
+git clone --branch b35f07243c51ac35d47857fe36f0aafb6b517955 https://github.com/ECP-WarpX/picsar.git
 
 # Clone the AMReX regression test utility
 git clone https://github.com/ECP-WarpX/regression_testing.git