-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
No module named 'pygribjump' #133
Comments
Hi @jmb64, You are quite right, we had not made gribjump public yet, but now we have :) It is very experimental and we are actively developing this whole stack. To use gribjump you'll need to build it, along with FDB (https://github.com/ecmwf/fdb) and it's dependencies. pygribjump needs to be pip installed directly from the git repository too, as its not on pypi yet. To be honest the polytope-gribjump-fdb stack is not really in a state where we'd expect people to be able to deploy it succesfully, but you're welcome to try! |
Hi James,
It seems that access to the python api pygribjump is rectricted though. python3 -m pip install --upgrade git+ssh://git@github.com/ecmwf/gribjump.git@master Collecting git+ssh://@github.com/ecmwf/gribjump.git@master Please make sure you have the correct access rights and the repository exists. |
Hi @jmb64, try pip installing it using https instead of ssh: pip install --upgrade git+https://github.com/ecmwf/gribjump |
Right on. Thanks! |
Would you happen to have an example showing the content for this file ? GRIBJUMP_CONFIG_FILE = "/path/to/gribjump/config.yaml" |
Hi @jmb64 ,
|
Hi,
I will follow your development and docs . Thanks. |
Hi Chris,
readthedocs.io (fdb and earthkit-data). To answer my own question about an FDB example for testing ... i was able to create |
Maybe one last question ... Running this test it seems I cannot get "step" to make it as an FDB datacube axes. This is the bit of code which I believe should create the datacube instance: self.options = {
|
Logging from polytope for cat lis_test_fdb_datacube.log INFO:root:Created an FDB datacube with options: {'config': [{'axis_name': 'step', 'transformations': [{'name': 'type_change', 'type': 'int'}]}, {'axis_name': 'number', 'transformations': [{'name': 'type_change', 'type': 'int'}]}, {'axis_name': 'date', 'transformations': [{'name': 'merge', 'other_axis': 'time', 'linkers': ['T', '00']}]}, {'axis_name': 'values', 'transformations': [{'name': 'mapper', 'type': 'octahedral', 'resolution': 1280, 'axes': ['latitude', 'longitude']}]}, {'axis_name': 'latitude', 'transformations': [{'name': 'reverse', 'is_reverse': True}]}, {'axis_name': 'longitude', 'transformations': [{'name': 'cyclic', 'range': [0, 360]}]}]} DEBUG:root:Skipping /etc/polytope/config.json, file not found. INFO:root:Axes returned from GribJump are: {} INFO:root:Polytope created axes for: dict_keys(['latitude', 'longitude']) |
Hi @jmb64, It seems like maybe there isn't the right data for this particular example written on the FDB that Polytope is connecting to? Unfortunately, I don't think the data that goes with these particular tests can be viewed publicly. However, I think it should be possible to adapt the Polytope options and datacube configuration to work with the data written on your local FDB. The datacube axes created in the Polytope API aren't really specified by the options, but are built by "reading" the axes from the data on the FDB. The options only really help create an easy user interface to make requesting data more intuitive for users, such as being able to request data at longitude=420 (instead of having to request data at longitude=60) for example. The options here refer to particular "transformations" on the data, such as cyclic axes or special grids, like octahedral grids for example (when the data grib file doesn't store the data along latitude/longitude, but rather along a combined "index" axis). There is also the type change transformation, which we use when the request type (step in integers for example) doesn't match how the data is stored on the FDB (in this case as strings), as well as the reverse transformation, when the data stored in the FDB on an axis is in decreasing order (for example latitude indexes are stored from 90 to -90 instead of -90 to 90). I also think that currently, the Polytope version on develop doesn't yet work with the GribJump master branch. Instead, I believe the feature/gribjump_axis_branching branch of Polytope should work with the latest GribJump master branch. I hope this helps, but I would be happy to answer more questions! |
Hi Mathilde,
major role in the datacube configuration :-) ... just by changing the FDB I noticed I was getting past the configuration (i.e. Regards, Jean-Marc PS: You probably know the answer to this one ... In Tiago's presentation (Bytes from Petabytes, slide 20) there's a graph showing linear scaling for the extraction process for 1000 to 4500 values on Lustre ... we're talking about double precision values right ? Hence the 36kB(4500 values)/730ms ---> 4-6 ms/value ... Are those results from /examples or /performance on github ? |
Hello again,
#!/usr/bin/env python c = cdsapi.Client() c.retrieve("reanalysis-era5-complete", {
|
Hello @jmb64, I think maybe this fails because of the grid option given to Polytope for the data. I think the grid keyword in the MARS requests transform the data automatically to a regular lat/lon grid and so the regular grid "transformation" would be needed in this case. I believe here the options should thus be
I hope this helps! |
Hi Mathilde,
End of listing with your suggestion: JMB fdb.py assign_fdb_output_to_nodes: sorted_range_lengths= [2, 2, 2, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 2, 2, 2, 2, 1, 1, 1, 2, 2, 2, 2, 2, 1, 2, 1, 2, 2, 2, 2, 2, 1, 2, 1, 2, 2, 2, 2, 2, 1, 1, 1, 1, 2, 2, 3] During handling of the above exception, another exception occurred: Traceback (most recent call last): Thread Thread 22435342517824 stopping (queue closed)22435554211392 stopping (queue closed)It seems we got output values from GribJump now and they sure look like surface specific humidities around 18-19 g/Kg ... which is nice. There's a glitch remaining ...which appears minor ? What is the meaning of "resolution" :360 then ? I thought we had to specify the data resolution (0.25 deg in this case). I guess this information is retrieved from FDB (GRIB) then. I was also wondering about request about NEMO output. The software is not able to deal with such a grid yet (ORCA025), am I right ? Anyway. Your help is much appreciated! Jean-Marc |
Hi @jmb64 , This is great! It looks like it's almost working :) I believe maybe this print line in the code might be causing issues:
One of the problems of Polytope which happens here is that the output of Polytope isn't necessarily "box"-shaped so most of the numpy functions like shape() do not work on Polytope's output! We've also just updated the develop branch of Polytope yesterday which might fix some of the last bugs! For the resolution, it's currently maybe a bit counterintuitive. It was initially built around the octahedral grid, where the resolution was the number x in Ox, so somewhat related to the number of data points on a latitude line. We've tried to keep it somewhat "uniform" for the different grids and so the resolution for the regular lat/lon grid is the number of points on a latitude line (in this case it's 90/0.25=360). I hope this explains a bit more! Unfortunately, I do not believe we support ORCA grids yet. I hope this helps! |
What happened?
Hi,
python 3D_shipping_route_wave_model.py
Traceback (most recent call last):
File ".../3D_shipping_route_wave_model.py", line 8, in
from polytope.datacube.backends.fdb import FDBDatacube
File .../python3.10/site-packages/polytope/datacube/backends/fdb.py", line 4, in
import pygribjump as pygj
ModuleNotFoundError: No module named 'pygribjump'
I searched for pygribjump ... it was nowhere to be found.
I realize that the software is a work in progress. I just want to check with you that the module (GribJump, pygribjump)
is indeed not publicly available and the outcome is as expected ?
I look forward to see more performance results with earthkit/polytope. Your work is interesting and appears to be really tigthly
integrated to the software infrastructure at ecmwf. This is new stuff in comparison to what is done in the context of PANGEO
for example.
Anyway, thanks for your time,
Oh, and I did succeed in running some other tests though ... :-)
What are the steps to reproduce the bug?
none
Version
polytope 1.03
Platform (OS and architecture)
cat /etc/os-release --> Ubuntu 22.04.4 LTS uname -a ---> Linux 4.18.0-240.el8.x86_64
Relevant log output
Accompanying data
none
Organisation
Environment and Climate Change Canada
The text was updated successfully, but these errors were encountered: