ADIOS2: Optionally write attributes only from given ranks

[franzpoeschel]

Performance optimization especially for large scale
Ref.
ornladios/ADIOS2#3846 (comment)

TODO:

• Documentation
• Is this relevant for HDF5 too?

[ax3l]

@eschnett thinking about portable code that also supports HDF5, would this interrupt you if we enabled this by default (in a follow-up PR)? 😅

[eschnett]

@ax3l I don't understand the ramifications. On the surface this all seems benign and beneficial.

[franzpoeschel]

@eschnett The possible pitfall is that this would by default ignore any attribute that is not written on rank 0.
Background:

• attribute writes in HDF5 are collective
• attribute writes in ADIOS2 are non-collective, and defining them only on one rank is more performant than doing it collectively

In consequence: Portable code using the openPMD-api should treat attribute writes as something collective. If attributes are written collectively, there's no harm in ADIOS2 to ignore any attribute write except for those on rank 0. This is why we're thinking about enabling this by default.

If for some reason you have routines that create some data only on a few select ranks, this could lead to problems.

[franzpoeschel]

@ax3l If we decide to activate this by default, we should maybe add some easy way to activate attribute writing from all ranks. Currently, attribute_writing_ranks is a list of ranks that opt into writing attributes. Something like attribute_writing_ranks = "all" might then be needed to restore the current behavior.

[eschnett]

Thanks for the explanation.

I was not aware that openPMD treated attribute writes as global. This will be good to know.