[query] Extract Backend Methods called from Python into Py4JBackendExtensions
#14767
Conversation
ehigham
force-pushed
the
ehigham/py4j-backend-extensions
branch
2 times, most recently
from
2d4edc1 to
79ecea8
Compare
ehigham
force-pushed
the
ehigham/py4j-backend-extensions
branch
from
79ecea8 to
1d36659
Compare
| // From https://github.com/hail-is/hail/issues/14580 : | ||
| // IR can get quite big, especially as it can contain an arbitrary | ||
| // amount of encoded literals from the user's python session. This | ||
| // was a (controversial) restriction imposed by Jackson and should be lifted. | ||
| // | ||
| // We remove this restriction at the earliest point possible for each backend/ | ||
| // This can't be unified since each backend has its own entry-point from python | ||
| // and its own specific initialisation code. | ||
| StreamReadConstraints.overrideDefaultStreamReadConstraints( | ||
| StreamReadConstraints.builder().maxStringLength(Integer.MAX_VALUE).build() | ||
| ) |
There was a problem hiding this comment.
What's the reasoning for moving this from the Backend constructor?
There was a problem hiding this comment.
This was tricky to debug. We need to set the defaults before com.fasterxml.jackson.core.JsonFactory is initialised (which is initialised by spark).
That factory holds onto a copy of these constants and so subsequent calls to overrideDefaultStreamReadConstraints will have no effect after the factory is initialised. It's that instance that's used to determine max string length.
Moving it to each backend entry point makes it more likely we'll initialise the defaults first but there are no guarantees. Say someone provides their own spark context before the spark backend is initialised. They're likely to encounter this problem.
hail/src/main/scala/is/hail/backend/py4j/Py4JBackendExtensions.scala
Outdated
Show resolved
Hide resolved
ehigham
force-pushed
the
ehigham/py4j-backend-extensions
branch
from
1d36659 to
0f02bdc
Compare
Refactors backend code to improve separation of concerns by extracting Python-specific functionality into a new
Py4JBackendExtensionstrait. This change consolidates Python interface methods that were previously scattered across different backend implementations into a single shared trait.The change also improves reference genome handling by making the references map mutable and passing it explicitly through constructors rather than modifying it after creation.
Security Assessment
This change has no security impact
Impact Description
Low-level refactoring of internal backend code that moves functionality between classes without changing behavior. No changes to external interfaces or security boundaries.