Skip to content

Does xarray automatically use Opportunistic Caching from Dask? #4580

Answered by dcherian
ahuang11 asked this question in Q&A
Discussion options

You must be logged in to vote

Does it automatically do something like
dd.compute(df.amount.max(), df.amount.min()) so that Dask will share intermediates intelligently, reading through the dataset only once?

Yes. The graph for that dataset is computed all at once so intermediaries are shared, and data is only read once.

This is not caching though. For that you need to turn on the cache: https://docs.dask.org/en/latest/caching.html#automatic-opportunistic-caching and then you may see speedups when repeatedly accessing the same data.

Replies: 1 comment

Comment options

You must be logged in to vote
0 replies
Answer selected by shoyer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
2 participants