Frontier/Crusher: Less Invasive libFabric Work-Around (#3396)

From Steve Abbott (HPE):
> There's a known problem with the default libfabrics memory
> registration cache monitor that impacts codes that allocate and
> free MPI buffers frequently. What you're doing now,
> FI_MR_CACHE_MAX_COUNT=0 is a big hammer that disables the memory
> registration cache all together. That can have a negative
> performance impact, because memory registration is a heavy
> operation, but it doesn't seem to be hitting WarpX very hard. If
> you're mostly following an allocate-communicate-free pattern, the
> memory registration cache won't help you anyway.
>
> An alternative is to set FI_MR_CACHE_MONITOR=memhooks , which uses
> an alternative cache monitor that doesn't have the same problem. I
> tested on an 8 node WarpX case we have in a bug and only saw a 2%
> speedup over FI_MR_CACHE_MAX_COUNT=0, and that speedup was in
> FillBoundary which I'm guessing is the only place you might have
> some MPI buffer reuse. If you start to scale out you may want to
> try it.
>
> We're working on a new default cache monitor that won't have this
> problem but I'm not sure the timeline for it. We'll make sure that
> when it gets pushed out we'll let you know, but for now you have to
> keep using either of these two environment variables.

0 files changed, 0 insertions, 0 deletions