You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Currently, every write or read is performed with an expensive phase including open, seek, write/read, and close. This could be fixed by storing opened files and keeping them opened on the executors.
Describe the solution you'd like
Explore the cost and benefit of keeping files open across I/O operations, and possible different policies for improving this
Describe alternatives you've considered
It need not be the case that every file is opened on every executor. I would like to explore the possibility of having executors responsible for different files, forcing scheduling to push DataTasks to the executors that have different files opened or incorporating that information into a novel scheduling policy.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
Currently, every write or read is performed with an expensive phase including open, seek, write/read, and close. This could be fixed by storing opened files and keeping them opened on the executors.
Describe the solution you'd like
Explore the cost and benefit of keeping files open across I/O operations, and possible different policies for improving this
Describe alternatives you've considered
It need not be the case that every file is opened on every executor. I would like to explore the possibility of having executors responsible for different files, forcing scheduling to push DataTasks to the executors that have different files opened or incorporating that information into a novel scheduling policy.
The text was updated successfully, but these errors were encountered: