-
Notifications
You must be signed in to change notification settings - Fork 400
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Internal] Restart Cluster before Library Installation #4384
Conversation
If integration tests don't run automatically, an authorized user can run them manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
@parthban-db we have #4036 that mentions that we can install libraries in UI without starting the cluster - the question is if it works via API the same way or not... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In general is good, see my comment about existing issue about installing on terminated clusters via UI
I will ask the team about this. |
@alexott I talked with the team, and according to them, |
@parthban-db, We can make it controllable by the user - we start the cluster and install the library just to check if there are no errors. But if we can allow users to specify that they don't want to wait for it, then it could be a solution. @mgyucht wdyt? |
LGTM: we revert this back to the original & documented behavior of starting the cluster if it isn't already running. Longer term, we can revisit how Terraform users will think about cluster and library lifecycles. I can see this as very useful in some use-cases (e.g. DABs), but there may also be users who don't want a cluster to be started for the sake of creating/updating a library. |
PreConfig: func() { | ||
// Delete the created cluster | ||
w := databricks.Must(databricks.NewWorkspaceClient()) | ||
getter, err := w.Clusters.Delete(context.Background(), compute.DeleteCluster{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What a confusing name for this method... @renaudhartert-db maybe we want to revisit this in the next version of the SDK. It should really be "terminate cluster" to align with basically all other terminology.
### New Features and Improvements * Add `fallback` to `databricks_external_location` ([#4372](#4372)). ### Bug Fixes * Send only what is required in Update of `databricks_credential` ([#4349](#4349)). ### Documentation * Add `MANAGE` privilege to the relevant UC resources ([#4371](#4371)). * Add a note about schema evolution and `databricks_sql_table` ([#4352](#4352)). * Add description of `clean_rooms_notebook_task` in `databricks_job` ([#4368](#4368)). * Add docs for cluster attributes for clusters assigned to groups ([#4359](#4359)). * Add note about use of `azurerm_databricks` for enhanced security settings ([#4393](#4393)). * Clarify views sharing in `databricks_share` ([#4378](#4378)). * Document job edit_mode ([#4354](#4354)). ### Internal Changes * Auto generated customizable schemas ([#4356](#4356)). * Disable `terraform fmt` in exporter to workaround build issues ([#4394](#4394)). * Fixed redundant optional owners in share resource and bumped Go SDK ([#4396](#4396)). * Load auto-generated resources and data sources ([#4367](#4367)). * Migrate workflows that need write access to use hosted runners ([#4377](#4377)). * Remove TF annotations on TFSDK models ([#4383](#4383)). * Restart Cluster before Library Installation ([#4384](#4384)). ### Dependency Updates * Bump github.com/zclconf/go-cty from 1.15.1 to 1.16.0 ([#4374](#4374)). * Bump github.com/zclconf/go-cty from 1.16.0 to 1.16.1 ([#4397](#4397)). ### Exporter * Add exporting of `for_each_task` in jobs ([#4347](#4347)). * Add references for missing permissions/grants types ([#4390](#4390)). * Support for group assigned clusters ([#4361](#4361)). * Use Go SDK to list clusters ([#4385](#4385)). * Use `Workspace.Export` from Go SDK for notebooks/files ([#4391](#4391)).
### New Features and Improvements * Add `fallback` to `databricks_external_location` ([#4372](#4372)). ### Bug Fixes * Send only what is required in Update of `databricks_credential` ([#4349](#4349)). ### Documentation * Add `MANAGE` privilege to the relevant UC resources ([#4371](#4371)). * Add a note about schema evolution and `databricks_sql_table` ([#4352](#4352)). * Add description of `clean_rooms_notebook_task` in `databricks_job` ([#4368](#4368)). * Add docs for cluster attributes for clusters assigned to groups ([#4359](#4359)). * Add note about use of `azurerm_databricks` for enhanced security settings ([#4393](#4393)). * Clarify views sharing in `databricks_share` ([#4378](#4378)). * Document job edit_mode ([#4354](#4354)). ### Internal Changes * Auto generated customizable schemas ([#4356](#4356)). * Disable `terraform fmt` in exporter to workaround build issues ([#4394](#4394)). * Fixed redundant optional owners in share resource and bumped Go SDK ([#4396](#4396)). * Load auto-generated resources and data sources ([#4367](#4367)). * Migrate workflows that need write access to use hosted runners ([#4377](#4377)). * Remove TF annotations on TFSDK models ([#4383](#4383)). * Restart Cluster before Library Installation ([#4384](#4384)). ### Dependency Updates * Bump github.com/zclconf/go-cty from 1.15.1 to 1.16.0 ([#4374](#4374)). * Bump github.com/zclconf/go-cty from 1.16.0 to 1.16.1 ([#4397](#4397)). ### Exporter * Add exporting of `for_each_task` in jobs ([#4347](#4347)). * Add references for missing permissions/grants types ([#4390](#4390)). * Support for group assigned clusters ([#4361](#4361)). * Use Go SDK to list clusters ([#4385](#4385)). * Use `Workspace.Export` from Go SDK for notebooks/files ([#4391](#4391)).
Changes
Library resource with SDK_V2 implementation waits for the cluster to get into the running state before starting installation. This PR mimics this behavior in the plugin framework implementation of the library resource.
Fixes #4353
Tests
Added a integration test.
make test
run locallydocs/
folderinternal/acceptance