Request for advice on writing a large Dataset

[rpgoldman]

I'm having trouble with a large data set that I am able to manipulate in memory, but that causes out of memory errors when writing to disk with to_netcdf(). The docs provide a wealth of information about different options, but there's so much that I find myself lost. Would it be possible to provide a general guideline for what to try when this kind of problem happens? For example, there is discussion of chunking, but I don't know if that's likely to help in this problem or not.

I considered the possibility of using Dask to break up the saving computation, but the best I could come up with was this:

d_cont = ds.to_netcdf('positive_controls.nc', compute=False)

and I get an out of memory error before I get to trying to force this computation. (I was going to try d_cont.compute(scheduler='processes') next.)

The backtrace seems to indicate that dask might not even be used, despite compute=False:

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-20-61d913a51d15> in <module>
----> 1 d_cont = ds.to_netcdf('positive_controls.nc', compute=False)

~_packages/envs/dask/lib/python3.8/site-packages/xarray/core/dataset.py in to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute, invalid_netcdf)
   1642         from ..backends.api import to_netcdf
   1643 
-> 1644         return to_netcdf(
   1645             self,
   1646             path,

~_packages/envs/dask/lib/python3.8/site-packages/xarray/backends/api.py in to_netcdf(dataset, path_or_file, mode, format, group, engine, encoding, unlimited_dims, compute, multifile, invalid_netcdf)
   1109         # TODO: allow this work (setting up the file for writing array data)
   1110         # to be parallelized with dask
-> 1111         dump_to_store(
   1112             dataset, store, writer, encoding=encoding, unlimited_dims=unlimited_dims
   1113         )

~_packages/envs/dask/lib/python3.8/site-packages/xarray/backends/api.py in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims)
   1156         variables, attrs = encoder(variables, attrs)
   1157 
-> 1158     store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
   1159 
   1160 

~_packages/envs/dask/lib/python3.8/site-packages/xarray/backends/common.py in store(self, variables, attributes, check_encoding_set, writer, unlimited_dims)
    248             writer = ArrayWriter()
    249 
--> 250         variables, attributes = self.encode(variables, attributes)
    251 
    252         self.set_attributes(attributes)

~_packages/envs/dask/lib/python3.8/site-packages/xarray/backends/common.py in encode(self, variables, attributes)
    338         # to write times, for example, would fail.
    339         variables, attributes = cf_encoder(variables, attributes)
--> 340         variables = {k: self.encode_variable(v) for k, v in variables.items()}
    341         attributes = {k: self.encode_attribute(v) for k, v in attributes.items()}
    342         return variables, attributes

~_packages/envs/dask/lib/python3.8/site-packages/xarray/backends/common.py in <dictcomp>(.0)
    338         # to write times, for example, would fail.
    339         variables, attributes = cf_encoder(variables, attributes)
--> 340         variables = {k: self.encode_variable(v) for k, v in variables.items()}
    341         attributes = {k: self.encode_attribute(v) for k, v in attributes.items()}
    342         return variables, attributes

~_packages/envs/dask/lib/python3.8/site-packages/xarray/backends/scipy_.py in encode_variable(self, variable)
    186 
    187     def encode_variable(self, variable):
--> 188         variable = encode_nc3_variable(variable)
    189         return variable
    190 

~_packages/envs/dask/lib/python3.8/site-packages/xarray/backends/netcdf3.py in encode_nc3_variable(var)
     92         coding.strings.CharacterArrayCoder(),
     93     ]:
---> 94         var = coder.encode(var)
     95     data = coerce_nc3_dtype(var.data)
     96     attrs = encode_nc3_attrs(var.attrs)

~_packages/envs/dask/lib/python3.8/site-packages/xarray/coding/strings.py in encode(self, variable, name)
     63             safe_setitem(attrs, "_Encoding", string_encoding, name=name)
     64             # TODO: figure out how to handle this in a lazy way with dask
---> 65             data = encode_string_array(data, string_encoding)
     66 
     67         return Variable(dims, data, attrs, encoding)

~_packages/envs/dask/lib/python3.8/site-packages/xarray/coding/strings.py in encode_string_array(string_array, encoding)
     88     string_array = np.asarray(string_array)
     89     encoded = [x.encode(encoding) for x in string_array.ravel()]
---> 90     return np.array(encoded, dtype=bytes).reshape(string_array.shape)
     91 
     92 

MemoryError: Unable to allocate 4.02 GiB for an array with shape (159840000,) and data type |S27

Note that this is a dataset that I can read and write with Pandas and CSV. Presumably the more complex Xarray indexing is causing my problem here?

Follow up: Thing I did not expect

The pandas CSV for this data set (before I converted it from Dataframe to Dataset) is 786 meg. The nc file is 15.4 gig and it's not finished writing. So clearly I do not know what I am doing here.

[fmaussion]

Hi @rpgoldman , thanks for the report. There are a couple of things we need to know better in order to help you.

Can you post the output of ds.info()? After your additional info (786 meg vs 15.4 gig), it is very likely that the conversion pandas -> xarray did not go as you expected.

[rpgoldman]

@fmaussion Here's what I can gather:

1. The "raw" data frame has this description, as read from CSV:

<class 'pandas.core.frame.DataFrame'>
Int64Index: 1801709 entries, 0 to 1801708
Data columns (total 37 columns):
#   Column            Dtype  
---  ------            -----  
0   inc_temp          object 
1   inc_time_1        object 
2   inc_time_2        object 
3   plan              object 
4   lab               object 
5   growth_media_1    object 
6   growth_media_2    object 
7   od_cutoff         float64
8   source_container  object 
9   lab_id            object 
10  id                object 
11  strain_name       object 
12  gate              float64
13  input             float64
14  od                float64
15  filename          object 
16  replicate         int64  
17  output            float64
18  media             object 
19  plate_id          int64  
20  Time              float64
21  FSC_A             float64
22  SSC_A             float64
23  BL1_A             float64
24  RL1_A             float64
25  FSC_H             float64
26  SSC_H             float64
27  BL1_H             float64
28  RL1_H             float64
29  FSC_W             float64
30  SSC_W             float64
31  BL1_W             float64
32  RL1_W             float64
33  index             int64  
34  live_no_stain     float64
35  live              float64
36  well              object 
dtypes: float64(20), int64(3), object(14)
memory usage: 522.3+ MB

2. I reindex the data frame so that I can access data based on values of independent variables.

<class 'pandas.core.frame.DataFrame'>
MultiIndex: 1801709 entries, ('NOR-00-Control', 37, 18, 0.0003, 2, 39, 0) to ('NOR-00-Control', 30, 15, 0.0003, 1, 52, 29999)
Data columns (total 34 columns):
#   Column            Dtype   
---  ------            -----   
0   inc_temp          object  
1   inc_time_1        object  
2   inc_time_2        object  
3   plan              object  
4   lab               object  
5   growth_media_1    object  
6   growth_media_2    object  
7   od_cutoff         float64 
8   source_container  object  
9   lab_id            object  
10  id                object  
11  gate              float64 
12  input             float64 
13  filename          object  
14  output            float64 
15  media             category
16  Time              float64 
17  FSC_A             float64 
18  SSC_A             float64 
19  BL1_A             float64 
20  RL1_A             float64 
21  FSC_H             float64 
22  SSC_H             float64 
23  BL1_H             float64 
24  RL1_H             float64 
25  FSC_W             float64 
26  SSC_W             float64 
27  BL1_W             float64 
28  RL1_W             float64 
29  index             int64   
30  live_no_stain     float64 
31  live              float64 
32  well              object  
33  inc_time_1_hrs    int64   
dtypes: category(1), float64(19), int64(2), object(12)
memory usage: 470.3+ MB

3. Now I translate to a dataset

xarray.Dataset {
dimensions:
  event = 30000 ;
  inc_temp_degrees = 2 ;
  inc_time_2_hrs = 6 ;
  od = 1 ;
  plate_id = 6 ;
  replicate = 74 ;
  strain_name = 1 ;

variables:
  object strain_name(strain_name) ;
  int64 inc_temp_degrees(inc_temp_degrees) ;
  int64 inc_time_2_hrs(inc_time_2_hrs) ;
  float64 od(od) ;
  int64 plate_id(plate_id) ;
  int64 replicate(replicate) ;
  int64 event(event) ;
  object inc_temp(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object inc_time_1(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object inc_time_2(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object plan(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object lab(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object growth_media_1(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object growth_media_2(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 od_cutoff(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object source_container(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object lab_id(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object id(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 gate(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 input(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object filename(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 output(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object media(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 Time(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 FSC_A(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 SSC_A(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 BL1_A(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 RL1_A(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 FSC_H(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 SSC_H(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 BL1_H(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 RL1_H(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 FSC_W(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 SSC_W(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 BL1_W(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 RL1_W(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 index(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 live_no_stain(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 live(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object well(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 inc_time_1_hrs(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;

// global attributes:
}

4. When I try to write this to file, I get the following error:

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-26-79698b8ab498> in <module>
----> 1 ds.to_netcdf('positive_controls.nc')

~_packages/envs/dask/lib/python3.8/site-packages/xarray/core/dataset.py in to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute, invalid_netcdf)
 1642         from ..backends.api import to_netcdf
 1643 
-> 1644         return to_netcdf(
 1645             self,
 1646             path,

~_packages/envs/dask/lib/python3.8/site-packages/xarray/backends/api.py in to_netcdf(dataset, path_or_file, mode, format, group, engine, encoding, unlimited_dims, compute, multifile, invalid_netcdf)
 1109         # TODO: allow this work (setting up the file for writing array data)
 1110         # to be parallelized with dask
-> 1111         dump_to_store(
 1112             dataset, store, writer, encoding=encoding, unlimited_dims=unlimited_dims
 1113         )

~_packages/envs/dask/lib/python3.8/site-packages/xarray/backends/api.py in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims)
 1156         variables, attrs = encoder(variables, attrs)
 1157 
-> 1158     store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
 1159 
 1160 

~_packages/envs/dask/lib/python3.8/site-packages/xarray/backends/common.py in store(self, variables, attributes, check_encoding_set, writer, unlimited_dims)
  248             writer = ArrayWriter()
  249 
--> 250         variables, attributes = self.encode(variables, attributes)
  251 
  252         self.set_attributes(attributes)

~_packages/envs/dask/lib/python3.8/site-packages/xarray/backends/common.py in encode(self, variables, attributes)
  337         # All NetCDF files get CF encoded by default, without this attempting
  338         # to write times, for example, would fail.
--> 339         variables, attributes = cf_encoder(variables, attributes)
  340         variables = {k: self.encode_variable(v) for k, v in variables.items()}
  341         attributes = {k: self.encode_attribute(v) for k, v in attributes.items()}

~_packages/envs/dask/lib/python3.8/site-packages/xarray/conventions.py in cf_encoder(variables, attributes)
  771     _update_bounds_encoding(variables)
  772 
--> 773     new_vars = {k: encode_cf_variable(v, name=k) for k, v in variables.items()}
  774 
  775     # Remove attrs from bounds variables (issue #2921)

~_packages/envs/dask/lib/python3.8/site-packages/xarray/conventions.py in <dictcomp>(.0)
  771     _update_bounds_encoding(variables)
  772 
--> 773     new_vars = {k: encode_cf_variable(v, name=k) for k, v in variables.items()}
  774 
  775     # Remove attrs from bounds variables (issue #2921)

~_packages/envs/dask/lib/python3.8/site-packages/xarray/conventions.py in encode_cf_variable(var, needs_copy, name)
  256     var = maybe_default_fill_value(var)
  257     var = maybe_encode_bools(var)
--> 258     var = ensure_dtype_not_object(var, name=name)
  259     return var
  260 

~_packages/envs/dask/lib/python3.8/site-packages/xarray/conventions.py in ensure_dtype_not_object(var, name)
  211                 fill_value = inferred_dtype.type(np.nan)
  212 
--> 213             data = _copy_with_dtype(data, dtype=inferred_dtype)
  214             data[missing] = fill_value
  215         else:

~_packages/envs/dask/lib/python3.8/site-packages/xarray/conventions.py in _copy_with_dtype(data, dtype)
  171     up on the resulting array.
  172     """
--> 173     result = np.empty(data.shape, dtype)
  174     result[...] = data
  175     return result

MemoryError: Unable to allocate 1.19 GiB for an array with shape (1, 2, 6, 1, 6, 74, 30000) and data type object

5. I don't know if chunking this data would help. I tried chunking automatically and got the following:

xarray.Dataset {
dimensions:
  event = 30000 ;
  inc_temp_degrees = 2 ;
  inc_time_2_hrs = 6 ;
  od = 1 ;
  plate_id = 6 ;
  replicate = 74 ;
  strain_name = 1 ;

variables:
  object strain_name(strain_name) ;
  int64 inc_temp_degrees(inc_temp_degrees) ;
  int64 inc_time_2_hrs(inc_time_2_hrs) ;
  float64 od(od) ;
  int64 plate_id(plate_id) ;
  int64 replicate(replicate) ;
  int64 event(event) ;
  object inc_temp(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object inc_time_1(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object inc_time_2(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object plan(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object lab(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object growth_media_1(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object growth_media_2(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 od_cutoff(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object source_container(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object lab_id(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object id(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 gate(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 input(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object filename(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 output(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object media(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 Time(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 FSC_A(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 SSC_A(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 BL1_A(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 RL1_A(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 FSC_H(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 SSC_H(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 BL1_H(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 RL1_H(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 FSC_W(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 SSC_W(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 BL1_W(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 RL1_W(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 index(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 live_no_stain(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 live(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  object well(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;
  float64 inc_time_1_hrs(strain_name, inc_temp_degrees, inc_time_2_hrs, od, plate_id, replicate, event) ;

// global attributes:
}

I'm not sure that did anything. When I look at the chunks, I see:

Frozen(SortedKeysDict({'strain_name': (1,), 'inc_temp_degrees': (2,), 'inc_time_2_hrs': (6,), 'od': (1,), 'plate_id': (6,), 'replicate': (74,), 'event': (30000,)}))

6. When I try to write the chunked dataset, I get the same MemoryError, but also these warnings:

/work/05204/rpg/jupyter_packages/envs/dask/lib/python3.8/site-packages/xarray/conventions.py:184: SerializationWarning: variable inc_temp has data in the form of a dask array with dtype=object, which means it is being loaded into memory to determine a data type that can be safely stored on disk. To avoid this, coerce this variable to a fixed-size dtype with astype() before saving it.
warnings.warn(
/work/05204/rpg/jupyter_packages/envs/dask/lib/python3.8/site-packages/xarray/conventions.py:184: SerializationWarning: variable inc_time_1 has data in the form of a dask array with dtype=object, which means it is being loaded into memory to determine a data type that can be safely stored on disk. To avoid this, coerce this variable to a fixed-size dtype with astype() before saving it.
warnings.warn(
/work/05204/rpg/jupyter_packages/envs/dask/lib/python3.8/site-packages/xarray/conventions.py:184: SerializationWarning: variable inc_time_2 has data in the form of a dask array with dtype=object, which means it is being loaded into memory to determine a data type that can be safely stored on disk. To avoid this, coerce this variable to a fixed-size dtype with astype() before saving it.
warnings.warn(
/work/05204/rpg/jupyter_packages/envs/dask/lib/python3.8/site-packages/xarray/conventions.py:184: SerializationWarning: variable plan has data in the form of a dask array with dtype=object, which means it is being loaded into memory to determine a data type that can be safely stored on disk. To avoid this, coerce this variable to a fixed-size dtype with astype() before saving it.
warnings.warn(
/work/05204/rpg/jupyter_packages/envs/dask/lib/python3.8/site-packages/xarray/conventions.py:184: SerializationWarning: variable lab has data in the form of a dask array with dtype=object, which means it is being loaded into memory to determine a data type that can be safely stored on disk. To avoid this, coerce this variable to a fixed-size dtype with astype() before saving it.
warnings.warn(
/work/05204/rpg/jupyter_packages/envs/dask/lib/python3.8/site-packages/xarray/conventions.py:184: SerializationWarning: variable growth_media_1 has data in the form of a dask array with dtype=object, which means it is being loaded into memory to determine a data type that can be safely stored on disk. To avoid this, coerce this variable to a fixed-size dtype with astype() before saving it.
warnings.warn(
/work/05204/rpg/jupyter_packages/envs/dask/lib/python3.8/site-packages/xarray/conventions.py:184: SerializationWarning: variable growth_media_2 has data in the form of a dask array with dtype=object, which means it is being loaded into memory to determine a data type that can be safely stored on disk. To avoid this, coerce this variable to a fixed-size dtype with astype() before saving it.
warnings.warn(

[rpgoldman]

Follow-up 2:

I made a MultiIndex in my pandas data frame, dropping many of the columns for values of independent variables. When I did this, and tried to translate to a Dataset, I get an out of memory error.

I did a little further work, and realized I needed to install sparse and then pass sparse=True to the converter. This time the conversion to Xarray Dataset worked fine.

But ... then to_netcdf errored out with the following:

AttributeError                            Traceback (most recent call last)
<ipython-input-31-79698b8ab498> in <module>
----> 1 ds.to_netcdf('positive_controls.nc')

~_packages/envs/dask/lib/python3.8/site-packages/xarray/core/dataset.py in to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute, invalid_netcdf)
   1642         from ..backends.api import to_netcdf
   1643 
-> 1644         return to_netcdf(
   1645             self,
   1646             path,

~_packages/envs/dask/lib/python3.8/site-packages/xarray/backends/api.py in to_netcdf(dataset, path_or_file, mode, format, group, engine, encoding, unlimited_dims, compute, multifile, invalid_netcdf)
   1109         # TODO: allow this work (setting up the file for writing array data)
   1110         # to be parallelized with dask
-> 1111         dump_to_store(
   1112             dataset, store, writer, encoding=encoding, unlimited_dims=unlimited_dims
   1113         )

~_packages/envs/dask/lib/python3.8/site-packages/xarray/backends/api.py in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims)
   1156         variables, attrs = encoder(variables, attrs)
   1157 
-> 1158     store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
   1159 
   1160 

~_packages/envs/dask/lib/python3.8/site-packages/xarray/backends/common.py in store(self, variables, attributes, check_encoding_set, writer, unlimited_dims)
    248             writer = ArrayWriter()
    249 
--> 250         variables, attributes = self.encode(variables, attributes)
    251 
    252         self.set_attributes(attributes)

~_packages/envs/dask/lib/python3.8/site-packages/xarray/backends/common.py in encode(self, variables, attributes)
    337         # All NetCDF files get CF encoded by default, without this attempting
    338         # to write times, for example, would fail.
--> 339         variables, attributes = cf_encoder(variables, attributes)
    340         variables = {k: self.encode_variable(v) for k, v in variables.items()}
    341         attributes = {k: self.encode_attribute(v) for k, v in attributes.items()}

~_packages/envs/dask/lib/python3.8/site-packages/xarray/conventions.py in cf_encoder(variables, attributes)
    769 
    770     # add encoding for time bounds variables if present.
--> 771     _update_bounds_encoding(variables)
    772 
    773     new_vars = {k: encode_cf_variable(v, name=k) for k, v in variables.items()}

~_packages/envs/dask/lib/python3.8/site-packages/xarray/conventions.py in _update_bounds_encoding(variables)
    411         is_datetime_type = np.issubdtype(
    412             v.dtype, np.datetime64
--> 413         ) or contains_cftime_datetimes(v)
    414 
    415         if (

~_packages/envs/dask/lib/python3.8/site-packages/xarray/core/common.py in contains_cftime_datetimes(var)
   1758 def contains_cftime_datetimes(var) -> bool:
   1759     """Check if an xarray.Variable contains cftime.datetime objects"""
-> 1760     return _contains_cftime_datetimes(var.data)
   1761 
   1762 

~_packages/envs/dask/lib/python3.8/site-packages/xarray/core/common.py in _contains_cftime_datetimes(array)
   1746     else:
   1747         if array.dtype == np.dtype("O") and array.size > 0:
-> 1748             sample = array.ravel()[0]
   1749             if is_duck_dask_array(sample):
   1750                 sample = sample.compute()

AttributeError: 'COO' object has no attribute 'ravel'

There aren't any date time structures in this dataset, so I'm not sure why that failed.

[keewis]

cftime support currently works by putting cftime objects into arrays of dtype object. This might change once NEP-40, 41, 42, and 43 are implemented and included in a numpy release (and cftime supports it).

For now, you might be able to skip the check by uninstalling cftime (if you don't need it), but the real issue is that we don't support saving sparse arrays to netcdf files (see #4156). You can find some related discussion for zarr here: zarr-developers/zarr-python#152

[rpgoldman]

@dcherian I'm not sure what to do about this -- yes, there are a lot of string data types in the data set. But I can't do without them. I suppose I could turn them into Categorical, but then if I pull something out of this Dataset, and try to combine it with another Dataset from the same experimental campaign, that will involve trying to merge non-identical Categorical dtypes. So I would prefer to leave these as strings. It would be possible for me to identify an upper bound on string lengths for many of these columns, but I don't know that that helps at all.

[rpgoldman]

@keewis Would it be possible to modify to_netcdf() to add a check for sparse arrays and raise a more informative error message like "Writing sparse Xarray data structures to NetCDF is not supported."? That would be hugely helpful in cases like this.

I'm not sure where this leaves me -- I can't build this Dataset from a pandas Dataframe without using sparse arrays, because I get memory errors, and I can't save the Dataset if I do use sparse arrays.

Would it fix this for me to reset the index of the Pandas Dataframe to lose the MultiIndex, and then construct the Dataset from that? But that would mean writing code to construct the Xarray dimensions and coordinates from just a big hunk of data and at that point, shouldn't I just give up and use Pandas and my MultiIndex instead?

[keewis]

I agree, we should improve the error message / raise before the dtype check. Could you raise an issue for that (or open a PR, if you're up for it)?

But that would mean writing code to construct the Xarray dimensions and coordinates from just a big hunk of data and at that point, shouldn't I just give up and use Pandas and my MultiIndex instead?

I guess so, but this depends on what you want to do with the data. If you need the nD-arrays for your analysis, I would suggest writing some functions to do the conversion from / to a Dataset containing only 1D variables (not sure if that's the best way, though):

def as_dense(arr):
    if not isinstance(arr.data, sparse.COO):
        return arr
    return arr.copy(data=arr.data.todense())

def prepare_for_computation(ds, dim, level_names):
    # I can't seem to find a way to get `unstack(dim=..., sparse=True)` to actually create a sparse
    # array for my test dataset. so instead this uses `from_dataframe(_.to_dataframe(), sparse=True)`
    return xr.Dataset.from_dataframe(ds.set_index({dim: level_names}).to_dataframe(), sparse=True)

# for loading from netcdf
xr.open_dataset("x.nc").pipe(lambda _: prepare_for_computation(_, dim, level_names))

def prepare_for_storage(ds, dim, level_names):
    return ds.stack({dim: level_names}).map(as_dense).reset_index(dim)

# for saving to netcdf
prepare_for_storage(ds, dim, level_names).to_netcdf(...)

---

If you're free to choose the file formats, it might be possible to somehow save a sparse array to zarr (no official support, though, see zarr-developers/zarr-python#152)

[dcherian]

You could modify the code here to save a sparse array to netCDF too: #1077 (comment) . There are multiple CF-compliant ways of writing sparse info to netCDF (#1077 (comment))