alternative format for .yaml database;

[pbosler]

Summary

The current format of pcd.yaml may not be as much as we'd like for e3sm. It might be worth merging its format and entries with a complementary project from a few years ago.

Background

Consistency with physical constants has come up several times over recent years in e3sm, e.g., E3SM-Project/EKAT#48. Before @rljacob pointed me to the PhysicalConstantsDictionary, @jeff-cohere and I had decided on the same approach: hold the constants in a .yaml database that could be parsed into auto-generated source files in either c++ or fortran, to use consistently throughout e3sm. As a fundamental guiding principle, we felt that all constants should be accompanied by metadata that gives a reputable reference source for their value. Since we're a U.S. government project, we should use NIST or other agencies' publications (e.g., NASA) as source documents whenever possible.

The database that we came up with, now here in the pbosler/database2 branch file constants.yaml reflects several decisions that could be worth adopting or revisiting. To make the database, I grepped through E3SM's main components and gathered as many constants as I could find, then searched the SNL library, NIST, etc., to find their source values and reference material, and made this file. I've likely missed several constants, but it's a good start.

Only after I had done that and presented this work to @mt5555, did I talk to @rljacob and learn about PhysicalConstantsDictionary. PCD's database has some fields that mine doesn't; mine has some fields that they don't. PCD has constants that I have not found in E3SM, whereas constants.yaml is targeted at E3SM specifically. It's likely we'll want aspects of both if this effort moves forward.

The basic format for each entry in constants.yaml is:

Constant_Name:  # human readable standard name
  software_name: code_constant # what you'd see in code
  units: "unit string" # e.g., "W/m2/K4"
  source:
     name: "NIST CODATA 2018"
     url: https://physics.nist.gov/cuu/Constants/index.html
     citation: "E. Tiesinga, P. J. Mohr, D. B. Newell, and B. N. Taylor, 2021, CODATA recommended values of the fundamental physical constants: 2018, Rev. Modern Phys. 93." 

For derived constants, the source doesn't have to be a citation, it could instead be a definition. For example, the length of a sidereal day is given as a constant (source: NASA) in constants.yaml, so the angular velocity of the Earth can be derived from it as 2*pi / sidereal_day. In these cases, the numerical value is computed in double precision and appears as a given value like base constants, but the source metadata field contains the definition string, e.g., "2*pi/sidereal_day" instead of a citation.

Some possible points for discussion with @bartgol or @xylar (or others) have come up:

• After finding some numerical ill-conditioning even in algebraic formulas in some components of E3SM, for constants.yaml we decided that all constants should use double precision, with as many significant digits as possible. pcd.yaml seems to allow other precision values.
• Some constants have units that are hard to express using just SI terms; for example, seawater isotope ratios like "#17o/#16o" might not be the best way to proceed, but "1" or "non-dimensional" is less expressive and unsatisfying, too.
• Our first draft tried to group constants by type, e.g., "math," "physics," "chemistry," etc., but these aren't mutually exclusive and became unwieldy. We ended up deciding on a flat database structure.

[rljacob]

The syntax for pcd.yaml is explained in https://github.com/ESCOMP/PhysicalConstantsDictionary/wiki/Dictionary-Syntax

[mt5555]

If we are going to change constants, I hope we can fix this at the same time!
E3SM-Project/E3SM#4743

[bartgol]

The syntax for pcd.yaml is explained in Wiki: Dictionary Syntax (ESCOMP/PhysicalConstantsDictionary)

We could think of adding entries to each constant though, no? I like the idea of showing a what the source is, including a formula for the derived constants, so that one knows immediately it's not something whose number is established by a committee (like c, by the International System of Units), measured (so it may be found in a paper) or derived from other constants (the formula can still be found in a paper, but showing it helps). I would even go further and have a "type" entry, which specifies how the var comes to be, like "measured" (like the mass of a proton), "fundamental" (like boltzmann constant), or "derived" (like Rgas=k_b*N_A).

[rljacob]

"type" is already in pcd.yaml.

[xylar]

type is, however, not explained in the documentation there, and is currently always strict.

[rljacob]

A documentation oversight. The developer told me what it was for.

[rljacob]

Our first draft tried to group constants by type, e.g., "math," "physics," "chemistry," etc., but these aren't mutually exclusive and became unwieldy. We ended up deciding on a flat database structure.

Those groups are a little to broad. I think it will be less ambiguous to group them as in the current pcd.yaml with "universal" "earth" "water" (dropping "physical"). We could add "air" and "standard_atmosphere" and "molecular_weights".

[rljacob]

The constants yaml file could get large so it would help to have some higher level grouping.

[xylar]

... "seawater", "sea_ice", "land_ice"?

[rljacob]

Definitely "seawater". The dictionary should have constants needed by multiple subroutines and multiple models. It should not have every single physical constant in it.

[xylar]

Agree, but I believe there are sea-ice and land-ice constants involved in coupling that we need in the same sense as seawater constants.

[rljacob]

We have

Constant_Name:  # human readable standard name
  software_name: code_constant # what you'd see in code
  value:  the value
  units: "unit string" # e.g., "W/m2/K4"
  source:
     name: name of source
     url:  url of source
     citation:  reference for source

vs.

- group
  description:   of the group
  citation:  reference for constants in group
  entries:
    - name:  code_constant.  
      value: the value
      units: "unit string"
      prec: double   (used for the the fortran kind or C++ float/double)
      type: strict   (was meant to separate basic constants and ones derived from formulae)
      uncertainty: (exact for math, a number for measured constants)
      description: "
         Long description of constant"

These aren't really very different in the core content.
The big difference between the two is the status of "reference". PCD is trying to group constants by their source reference. I'd rather define references as a property of the constant but use an alias so the details don't have be written multiple times.

[rljacob]

After finding some numerical ill-conditioning even in algebraic formulas in some components of E3SM, for constants.yaml we decided that all constants should use double precision, with as many significant digits as possible. pcd.yaml seems to allow other precision values.

I agree they should always be double and yes PCD lets you specify if it should be single or double in the output code with "prec".. But the code generator could handle that ("all double", "all single", "use pcd")

[xylar]

Replying to E3SM-Project/Omega#179 (comment)

The point of PhysicalConstantsDictionary is meant for multiple modeling groups to share constant values. Truly model-specific constants would continue to be in things like shr_const_mod.

I think we'd better figure out how we want to organize things, then. It seems like we need the pcd.yaml file and another that is specific to E3SM, as well as a code that parses both into C++ and YAML files. Perhaps pcd.yaml can be taken straight from https://github.com/ESCOMP/PhysicalConstantsDictionary but so far that's not clear.

It no longer seems intuitive to me if the E3SM-specific YAML file exists in a fork of PhysicalConstantsDictionary, nor does it make sense for a code designed to parse both PhysicalConstantsDictionary and E3SM-specific constants should try to be general enough to contribute back to PhysicalConstantsDictionary.

Also, as we already discussed, there are significant advantages in having both YAML files and the code to parse them living outside of E3SM itself.

Does this point to making yet another repo?

[bartgol]

I would not be in favor of 2+ repos. It seems to me that, being our PCD repo a fork inside our org, we can easily make a single and comprehensive database of the E3SM project (and siblings) needs. I don't see the advantage of keeping a "pristine" PCD file and a "e3sm-specific" one.

Some other thoughts:

• I am in favor of doing away with the "prec" entry, and always save constants in double precision. The customer code can always get them in a different precision. This is, for instance, how I think the generated constants should be used in eamxx:

<header guard>
#include <e3sm_physics_constants.hpp>

#include "share/scream_config.hpp"
namespace scream
{
namespace constants
{
constexpr Real g    = e3sm_constants::gravity_acceleration;
constexpr Real Rgas = e3sm_constants::ideal_gas_constant;
...
}  // namespace constants
} // namespace scream

If eamxx is built with Real=float, then g, Rgas, etc. will be in single precision. I think that the pcd dictionary should provide the most accurate value of the constants. Also, the generated files will likely be COPIED into the customer repo (like e3sm), and committed. Since the option of building in single vs double precision is a build time option (not that we have it now, but thinking programmatically here), the generated file must be as precise as possible, leaving the rest to the customer app.

• My preference would be to make the citation entry specific to each constant. If we group constants by "category" (rather than source), then it is probably impossible to ensure that a single reference provides the numerical value for all the constants in the group

• As @pbosler suggested, we could consider making the "citation" field more generic, and turn it into a "source" sublist, which can contain a reference (url, or doi, or both) and/or a formula. E.g., since Rgas=k_b*N_A, there may not be a proper source for the correct numerical value (i.e., a value that agrees with the formula AND with the values of k_b and N_A mentioned before). Adding a formula field (maybe with a reference for the formula) can help the user figure out how the value was calculated.

• One more obs: I noticed these two:

          - name: radius_of_sphere_of_same_surface
             ...
          - name: radius_of_sphere_of_same_volume
            ...

Since our goal is to make the whole model agree on constants, I would avoid storing two values for the earth radius (or any constant, for that matter), in case a component uses the first and another uses the second...

[pbosler]

I would avoid storing two values for the earth radius, in case a component uses the first and another uses the second...

A good question to ask might be: are we trying to provide a dictionary for the broader community as a complement to something like the CF Convention? Or are we simply trying to produce a single point of reference for E3SM to ensure consistency inside E3SM?

If we're aiming for a broadly used dictionary, we'll end up with multiple sources that potentially define things differently, but we'll have a comprehensive reference better capable of supporting new research. In contrast, if we limit scope to the "for E3SM by E3SM" philosophy, we can simply define the only value that we use and presume that, especially since we include references, models that use other approximations and require other constants can figure this out for themselves.

[rljacob]

Note to everyone: this discussion format works better if you reply to a comment you want to remark on instead of making a new comment.

[rljacob]

We are trying to produce a single point of reference for E3SM but the broader community (at least Navy and NOAA) already came up with something after months of discussion so lets just use it. We can certainly engage with the original developers to change the yaml spec and add constants. I'm on the same ICAMs committee with them.

For a given physical quantity, there will be one agreed upon value. If there are authoritative sources that disagree, we will pick ONE. That is one of the goals of PCD: to get people to stop using their custom values because that becomes a (hidden) issue when you start coupling things for the first time. PCD goal's were bigger then ours but they include ours.

[rljacob]

I've been experimenting with yaml and have a format that puts the citation with the constant but don't require re-entering the info for each constant.

  set:
    - mathematical:
        description: "
          Common mathematical constants.
        "
        entries:
          - name: pi
            value: 3.141592653589793238462643
            units: none
            prec: double
            type: strict
            uncertainty: exact
            description: "
              Ratio of a circle's circumference to its diameter.
            "
            source: &ASHandbook1964
              citation: "
                Abramowitz, A., and Stegun, I.A., Handbook of Mathematical Functions
                With Formulas, Graphs, and Mathematical Tables, DOC/National Bureau of Standards,
                Applied Mathematics Series, 55, June 1964.
              "
          - name: e
            value: 2.718281828459045235360287
            units: none
            prec: double
            type: strict
            uncertainty: exact
            description: "
              Base of natural logarithms.
            "
            source: *ASHandbook1964

[rljacob]

The first constant from a reference provides the full info and an "anchor" &ASHandbook1964 then other constants with the same ref use the alias for it. *ASHandbook1964

[bartgol]

Nice. Maybe we could put all the citations anchors in a dictionary at the top, and just use the citation alias in the constants dictionary. Like

citations:
  - &ASHandbook1964: |
       Abramowitz, A., and Stegun, I.A., Handbook of Mathematical Functions
       With Formulas, Graphs, and Mathematical Tables, DOC/National Bureau of Standards,
       Applied Mathematics Series, 55, June 1964.
  - &SomeOtherWork: |
      SomeOtherFolks, blahblah.

and then in the constants dictionary do

  - name: pi
    source:
       type: citaton
       citation: *ASHandbook1964

Formula constants could instead have

  - name: ideal_gas_constant
    source:
      type: formula
      formula: boltzmann_constant*avogardo_number