Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[query] Remove BlockMatrix persist from Backend interface #14690

Open
wants to merge 1 commit into
base: ehigham/ctx-references
Choose a base branch
from

Conversation

ehigham
Copy link
Member

@ehigham ehigham commented Sep 17, 2024

This change is split out from a larger refactoring effort on the various Backend
implementations. The goals of this effort are to provide query-level
configuration to the backend that's currently tied to the lifetime of a backend,
reduce code duplication and reduce state duplication.

In this change, I'm removing blockmatrix persist/unpersist from the Backend
interface by adding BlockMatrixCache: mutable.Map[String, BlockMatrix] to
ExecuteContext. The various reader/writer implementations simply fetch the
block matrix from this cache. For the spark backend, this is backed by a cache
whose lifetime is tied to the spark backend. Since block matrices are not
supported in the local and service backends, the cache is an empty map.

Note that block matrix persist is broken in python (#14689)

@ehigham ehigham marked this pull request as ready for review September 17, 2024 21:46
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 92da6f0 to 4576acf Compare September 19, 2024 20:52
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch from b0be25e to 6038f9b Compare September 19, 2024 20:52
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 4576acf to 8a42e93 Compare October 1, 2024 19:44
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch 2 times, most recently from 15f2275 to 71fcb35 Compare October 1, 2024 20:03
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 341e6a7 to 1682964 Compare October 8, 2024 19:18
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch from 71fcb35 to 360af7c Compare October 8, 2024 19:18
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 1682964 to b1c0d04 Compare October 8, 2024 20:30
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch from 360af7c to d16f9c6 Compare October 8, 2024 20:30
@ehigham ehigham force-pushed the ehigham/ctx-references branch from b1c0d04 to 84ddcf3 Compare October 16, 2024 20:02
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch from d16f9c6 to 783df28 Compare October 16, 2024 20:02
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 84ddcf3 to a2ff477 Compare October 16, 2024 21:30
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch from 783df28 to 7e661e4 Compare October 16, 2024 21:30
@ehigham ehigham force-pushed the ehigham/ctx-references branch from a2ff477 to df5c723 Compare October 17, 2024 14:59
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch from 7e661e4 to 952b01d Compare October 17, 2024 15:00
@ehigham ehigham force-pushed the ehigham/ctx-references branch from df5c723 to adc0602 Compare October 21, 2024 15:24
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch from 952b01d to a265389 Compare October 21, 2024 15:24
@ehigham ehigham force-pushed the ehigham/ctx-references branch from adc0602 to 24b20f3 Compare October 21, 2024 18:50
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 92087ca to d7df668 Compare December 12, 2024 22:02
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch from 6c61a48 to e115c2c Compare December 12, 2024 22:02
@ehigham ehigham force-pushed the ehigham/ctx-references branch from d7df668 to a33bb7f Compare December 16, 2024 18:49
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch from e115c2c to c834eaf Compare December 16, 2024 18:49
@ehigham ehigham force-pushed the ehigham/ctx-references branch from a33bb7f to b19073d Compare December 17, 2024 16:47
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch from c834eaf to 0d9d8f3 Compare December 17, 2024 16:47
@ehigham ehigham force-pushed the ehigham/ctx-references branch from b19073d to 75f4fd5 Compare December 17, 2024 20:00
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch 2 times, most recently from 531890c to a7f6c50 Compare December 17, 2024 20:28
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 75f4fd5 to 5c0308f Compare January 13, 2025 16:36
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch from a7f6c50 to 96e508a Compare January 13, 2025 16:36
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 5c0308f to 90a9649 Compare January 17, 2025 21:15
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch from 96e508a to f23b4f2 Compare January 17, 2025 21:15
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 90a9649 to ff175f0 Compare January 21, 2025 16:48
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch from f23b4f2 to 7949c41 Compare January 21, 2025 16:48
@ehigham ehigham force-pushed the ehigham/ctx-references branch from ff175f0 to f1a3197 Compare January 21, 2025 20:14
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch from 7949c41 to 85f3e21 Compare January 21, 2025 20:15
@ehigham ehigham force-pushed the ehigham/ctx-references branch from f1a3197 to 482d085 Compare January 21, 2025 21:02
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch 2 times, most recently from 8fe9826 to 35728c2 Compare January 22, 2025 15:50
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 482d085 to 901fafe Compare January 22, 2025 16:45
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch 2 times, most recently from 8ddcb28 to c5b8c2d Compare January 22, 2025 17:30
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 901fafe to 7c3af40 Compare January 24, 2025 20:54
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch from c5b8c2d to 574082e Compare January 24, 2025 20:54
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 7c3af40 to dd052e7 Compare January 27, 2025 23:15
@ehigham ehigham force-pushed the ehigham/ctx-bm-cache branch from 574082e to 2749402 Compare January 27, 2025 23:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant