Herbie crashing

[jp2nyy]

Tonight/this morning, Herbie kept on crashing on Google Colab where it says my session crashed for an unknown reason. I noticed that it took a bit longer on the herbie.toolbox import I had in my code and it stopped right after calling for a certain variable.

[fleegs79]

I've been running into the same thing with Google Colab starting on Nov 25th (worked fine on the 23rd and for the last year plus - Thank you for all of your work Brian!).

I added some logging to the Core.py file and it's bombing during the xarray function, specifically with cfgrib.open_datasets on line 1137. I did try manually adding that same cfgrib.open_datasets to my Colab script and was able to get it to run fine, so I'm not sure what changed in Google Colab on the 25th to cause the issue. I did open a ticket with Google Colab, but I haven't received a response back yet.

[jp2nyy]

cfgrib gives me fits when I tried xarray originally in reading grib2 files.

[fleegs79]

I think I have it tracked down to the ecCodes package. The version 2.39.0 (released on 11/24/2024) doesn't work with Herbie (at least in Google Colab), but if I roll it back to 2.38.3 everything runs fine.

For those using Google Colab, add this before installing herbie-data via pip:
!pip install eccodes==2.38.3

Hope that helps get your scripts working again.

fleegs79

[jp2nyy]

By George, the eccodes 2.38.3 worked so far.

[blaylockbk]

Thanks for tracking this down. I don't use colab, so haven't ever tested there.

Should an issue be opened with cfgrib?

[blaylockbk]

yeah, this crash must just be in colab?? I can't replicate with a fresh install on linux (WSL). (I actually haven't ever used colab.)

Create new env

mamba env create -n herbie_fresh herbie-data
mamba activate herbie_fresh

Some versions

# Name                    Version                   Build    Channel
python                    3.13.1       ha99a958_102_cp313    conda-forge
herbie-data               2024.8.0           pyhd8ed1ab_0    conda-forge
eccodes                   2.39.0               hf413ef6_1    conda-forge
python-eccodes            2.37.0          py313ha014f3b_0    conda-forge
cfgrib                    0.9.14.1           pyhd8ed1ab_0    conda-forge
xarray                    2024.11.0          pyhd8ed1ab_0    conda-forge
pandas                    2.2.3           py313ha87cce1_1    conda-forge
numpy                     2.1.3           py313h4bf6692_0    conda-forge

Read single GRIB message in xarray

from herbie import Herbie
ds = Herbie('2024-01-01').xarray('TMP:2 m')
#^ Had no issues

[fleegs79]

Thanks for testing that out Brian! Maybe it's something with the older python/numpy that Google Colab uses? Here's the same list for the scripts that crash (only eccodes gets rolled back to make it work).

# Name                    Version
python                    3.10.12
herbie-data               2024.8.0 
eccodes                   2.39.0
python-eccodes            Not installed
cfgrib                    0.9.14.1
xarray                    2024.10.0 
pandas                    2.2.2
numpy                     1.26.4 

They are planning on updating python to 3.11 in January.

[dgketchum]

I had the same issue on my local Ubuntu machine on fresh conda env (Python 3.12) with herbie installed today, sigkill 139 on running H.xarray with an inclusive search param, FWIW. Resolved by running 'pip install eccodes==2.38.3', as mentioned by @fleegs79.

Awesome project!