Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ExeUnit transfer component separation #2913

Merged
merged 41 commits into from
May 6, 2024

Conversation

nieznanysprawiciel
Copy link
Contributor

@nieznanysprawiciel nieznanysprawiciel commented Nov 27, 2023

Main goal of this PR is separating transfer library as distinct crate to allow ya-runtime-ai to deploy models.
A few additional changes and fixes were made:

  • Transfer examples were rewritten to SystemTests (feature framework-tests)
  • Transferring compressed files isn't working (issue: Fix downloading archives in ExeUnit #2953), so it was moved to separate test and #[ignore]
  • Problem with VM image corrupted was fixed (Partially downloaded files from previous ExeUnit executions were incorrectly hashed)
    • If file already existed on disk, HashStream loads currently existing data first (happens in case of retries)
    • If image download was not completed in previous execution, it is removed to avoid problems
  • File transfers to container:// can be retried and previous progress won't be lost

@nieznanysprawiciel nieznanysprawiciel force-pushed the extract-exeunit-transfer-component branch from c1ab24d to 892cde3 Compare November 27, 2023 12:54
@nieznanysprawiciel nieznanysprawiciel force-pushed the extract-exeunit-transfer-component branch from 41be131 to 5a6070b Compare December 20, 2023 15:35
@nieznanysprawiciel nieznanysprawiciel force-pushed the extract-exeunit-transfer-component branch from 95484b1 to 87da50f Compare December 21, 2023 11:22
@nieznanysprawiciel nieznanysprawiciel force-pushed the extract-exeunit-transfer-component branch 2 times, most recently from 4fa6845 to c5e6e6b Compare December 21, 2023 16:05
@nieznanysprawiciel nieznanysprawiciel force-pushed the extract-exeunit-transfer-component branch from c5e6e6b to b78b1c7 Compare December 21, 2023 16:34
@nieznanysprawiciel nieznanysprawiciel marked this pull request as ready for review December 22, 2023 16:14
@nieznanysprawiciel nieznanysprawiciel requested a review from a team as a code owner December 22, 2023 16:14
@prekucki prekucki force-pushed the extract-exeunit-transfer-component branch from 55171ad to b962998 Compare April 24, 2024 16:14
@nieznanysprawiciel nieznanysprawiciel merged commit 53b8c79 into master May 6, 2024
34 checks passed
@nieznanysprawiciel nieznanysprawiciel deleted the extract-exeunit-transfer-component branch May 6, 2024 12:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants