Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

consider setting repo to use linux binaries from PPM #217

Open
slopp opened this issue Oct 7, 2020 · 2 comments
Open

consider setting repo to use linux binaries from PPM #217

slopp opened this issue Oct 7, 2020 · 2 comments

Comments

@slopp
Copy link
Contributor

slopp commented Oct 7, 2020

Note that the very first time you submit a job to CloudML the various packages required to run your script will be compiled from source. This will make the execution time of the job considerably longer that you might expect. It’s only the first job that incurs this overhead though (since the package installations are cached), and subsequent jobs will run more quickly.

We could significantly reduce the first job time and compilation errors by using the public package manager to provide binary packages, potentially as an opt-out option

@javierluraschi
Copy link
Contributor

This sounds pretty great, honestly! Our hesitation here is that we need to reconsider how one trains torch jobs in the cloud, if the answer is cloudml, which I think might be, then we should totally do this work.

@javierluraschi
Copy link
Contributor

I'd add that currently, cloudml does not have a dependency to Python/reticulate, so this could be a great way to train models. However, is also worth considering if we could come up with a multi-cloud approach that supports more than just Google Cloud, maybe even RStudio Connect or the Job Launcher?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants