Time decoding for String data

[lesserwhirls]

As a simplified example, let say I have a netCDF4 file that contains a time variable where time is encoded as ISO 8601 strings:

dimensions:
	lon = 576 ;
	lat = 361 ;
	time = 3 ;
variables:
	double lon(lon) ;
		string lon:units = "degrees_east" ;
	double lat(lat) ;
		string lat:units = "degrees_north" ;
	float data_variable(time, lat, lon) ;
	string time(time) ;
data:

 time = "2024-07-01T00:30:00Z", "2024-08-01T00:30:00Z", "2024-09-01T00:30:00Z" ;
}

While not CF compliant, with a little work these time values can be decoded for better use in xarray using something like:

ds = xarray.open_dataset('string_time.nc4')
times = ds.time.values
dt64 = np.array([datetime.fromisoformat(t).replace(tzinfo=None) for t in times], dtype='datetime64[ns]')
ds2 = ds.assign_coords(time=dt64)

Side note: when reading this through OPeNDAP, the times array has dtype='|S64', so the values need to be decoded to utf-8 inside the datetime.fromisoformat call.

My question is if there is already way to decode these time values using xarray.open_dataset directly without the extra steps, and if not, would there be interest in a contribution to enable that extra bit of flexibility?

[dcherian]

Is there a CF attribute that tells you these values are timestamps in strings? Without that, I don't see how you can do this automatically.

[DocOtak]

CF coordinate variables (xarray Dimension coordinate) must be numeric and monotonic, strings variables are explicitly prohibited to be coordinate variables .

Might having the "preprocess" hook in open_mfdataset be available on the normal open/read_dataset functions be a potential solution?

[lesserwhirls]

Yes, the time coordinate variable here is not CF compliant, but such a time coordinate variable representation is one of the options outlined in the NetCDF Users Guide (as well as presented at various netCDF users workshops throughout the years).

[DocOtak]

I'm not sure I've ever actually encountered a data file using strings for time "in the wild". For my own stuff, which is encoded using the "days since.. " method, I'll look at the CDL using ncdump with the -i or -t flags to get more human time strings.

[lesserwhirls]

In this simplified example there isn't. The workflow driving this case, however, would have some attributes that could be used, for example:

_CoordinateAxisType: Time
long_name: time coordinate
standard_name: time

The other piece of information that could help identify this case automatically is that it is a string valued coordinate variable (one-dimensional variable with the same name as its dimension - CF, which refers to the netCDF User's Guide). This is perhaps the bigger thing to key off of to check if the values are ISO timestamps. @dopplershift had some ideas where this might fit within the codebase, so maybe he has some thoughts on how it could be best identified.

[kmuehlbauer]

_CoordinateAxisType

There is no such thing in CF Conventions, or do I miss something?

[dcherian]

yes not CF : https://docs.unidata.ucar.edu/netcdf-java/current/userguide/common_data_model_overview.html#coordinate-system-object-model

[lesserwhirls]

Indeed, these are not strictly CF compliant datasets, and that attribute is part of the _Coordinate Attribute Convention (which preceded CF efforts and is mentioned in the netCDF User's Guide).

[kmuehlbauer]

I'm not convinced that encoding of times as strings is a good idea wrt memory and diskspace overhead, but that's only my personal view. CF has it's downsides, too. But I'm also hesitant to integrate this decoding into the CF decoding. The current decoding is already very complex and adding more complexity just doesn't make sense, imho.

There is movement to combine the current CF decoding options into a single kwarg (see #4490 [fixed issue number!]). We could think of having additional CDMDecodingOptions beside the proposed CFDecodingOptions.

Then something like the following would be possible:

cf_ds = xr.open_dataset(cf_file, decode=xr.CFDecodingOptions(decode_times=True))
cdm_ds = xr.open_dataset(cdm_file, decode=xr.CDMDecodingOptions(decode_times=True))

This will need at least one brave volunteer to do the implementation work and quite some time.

A more quick solution would be to create you own Backend. Another solution using preprocess was already suggested by @DocOtak.

Also cf-xarray can handle at least _CoordinateAxisType, but not sure if automatic decoding is implemented.

[lesserwhirls]

I'm not convinced that encoding of times as strings is a good idea wrt memory and diskspace overhead, but that's only my personal view.'

100% agree, and for sure it's is not (and I am not suggesting anyone do this). That said, there are archived datasets that do this (pre-CF), and certain kinds of granule level aggregations performed by the THREDDS Data Server do this (underlying monthly summary granules that only contain time in their filenames, for example).

[dopplershift]

I want to clarify a few things:

The behavior of encoding date/times in a String variable is one of two specific recommendations (the other being CF-style) made by the netCDF User's Guide:

There are two strategies for storing a date/time into a netCDF variable. One is to encode it as a numeric value and a unit that includes the reference time, e.g. "seconds since 2001-1-1 0:0:0" or"days since 2001-1-1 0:0:0". The other is to store it as a String using a standard encoding and Calendar. The former is more compact if you have more than one date, and makes it easier to compute intervals between two dates.

This is very much "in the wild", since this discussion is being prompted by a support request for the THREDDS team; the user is aggregating netCDF files using NCML and complaining that xarray gives an error when parsing the data returned by the THREDDS server in this case.

I'll also note that the current documentation for xarray.open_dataset() states for the decode_times argument:

If True, decode times encoded in the standard NetCDF datetime format into datetime objects.

One could easily interpret "standard netCDF datetime format" as any of the options recommended by the netCDF User's Guide, though I understand if the resolution is "fix the docs" for that.

The work-around isn't horrific:

ds = xarray.open_dataset(opendap_url)
times = ds.time.values
dt64 = np.array([datetime.fromisoformat(t.decode('utf-8')).replace(tzinfo=None) for t in times], dtype='datetime64[ns]')
ds2 = ds.assign_coords(time=dt64)

(suggestions for improvement welcome). The question is if there was an appetite for supporting another "standard netCDF datetime format" directly within xarray, and where we might possibly look to put that if we were putting together a PR.

[kmuehlbauer]

Thanks @dopplershift for the clarification.

One could easily interpret "standard netCDF datetime format" as any of the options recommended by the netCDF User's Guide, though I understand if the resolution is "fix the docs" for that.

Indeed, but the same document also states:

Use an existing netCDF Convention for your coordinate variables, especially to identify spatio-temporal coordinates.

The list of registered conventions has the CF Conventions listed as recommended standard as first entry. I'm unsure to what extent xarray follows those other mentioned conventions.

The question is if there was an appetite for supporting another "standard netCDF datetime format" directly within xarray, and where we might possibly look to put that if we were putting together a PR.

While "fix the docs" seems a like a nice approach to close the case it somehow would deny xarray's roots. But as @dcherian already mentioned, without some attribute indicating a variable as time coordinate it's hard to imagine an automated way of decoding. This is also a bit different from the _Coordinate_Attribute_Convention @lesserwhirls mentioned. Solving #4490 will enable decoders for different conventions.