Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for installing the CLI on DBR #85

Merged
merged 11 commits into from
Feb 22, 2024
Merged

Add support for installing the CLI on DBR #85

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
09e8c6a
Set installation target to ~/bin when install.sh is run from DBR
shreyas-goenka Feb 6, 2024
981f8f9
add mkdir
shreyas-goenka Feb 9, 2024
0d15a8e
add tests
shreyas-goenka Feb 22, 2024
1f39d31
Merge remote-tracking branch 'origin' into install-dbr
shreyas-goenka Feb 22, 2024
2657454
-
shreyas-goenka Feb 22, 2024
a747642
-
shreyas-goenka Feb 22, 2024
5377eb4
+x to path.sh
shreyas-goenka Feb 22, 2024
dbbcefb
add assertions
shreyas-goenka Feb 22, 2024
5138a84
added not installed check
shreyas-goenka Feb 22, 2024
7beabd1
chmod +x not-installed.sh
shreyas-goenka Feb 22, 2024
6a1445f
-
shreyas-goenka Feb 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 99 additions & 4 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ jobs:
- run: databricks version
shell: bash

- run: ./setup-cli/test.sh $(cat ./setup-cli/VERSION)
- run: ./setup-cli/assert/version.sh $(cat ./setup-cli/VERSION)
shell: bash

action_with_version:
Expand All @@ -57,7 +57,7 @@ jobs:
- run: databricks version
shell: bash

- run: ./setup-cli/test.sh 0.200.0
- run: ./setup-cli/assert/version.sh 0.200.0
shell: bash

install:
Expand All @@ -79,13 +79,62 @@ jobs:
- run: env
shell: bash

- name: Assert databricks CLI is not already installed
run: ./setup-cli/assert/not-installed.sh
shell: bash

- run: ./setup-cli/install.sh
shell: bash

- run: databricks version
shell: bash

- run: ./setup-cli/test.sh $(cat ./setup-cli/VERSION)
- name: Assert the version of the CLI installed
run: ./setup-cli/assert/version.sh $(cat ./setup-cli/VERSION)
shell: bash

- name: Assert installation path is /usr/local/bin for non-windows platforms
run: ./setup-cli/assert/path.sh /usr/local/bin/databricks
if: matrix.os != 'windows-latest'
shell: bash

- name: Assert installation path is /c/Windows for windows platforms
if: matrix.os == 'windows-latest'
run: ./setup-cli/assert/path.sh /c/Windows/databricks


install-dbr:
# All DBR images are built on top of Ubuntu
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3
with:
path: ./setup-cli

# Append ~/bin to the PATH. This helps with the assertions in the next steps.
- run: echo "PATH=$PATH:$HOME/bin" >> $GITHUB_ENV

- name: Assert databricks CLI is not already installed
run: ./setup-cli/assert/not-installed.sh
shell: bash

- name: Install the CLI
run: ./setup-cli/install.sh
shell: bash
env:
DATABRICKS_RUNTIME_VERSION: value-does-not-matter
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this env variable always set on DBR?

Copy link
Contributor Author

@shreyas-goenka shreyas-goenka Feb 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While it's not documented to be guaranteed, it's been present on all clusters I have tried out and is very well-established as a rubric to detect that code is being run from DBR.

For example, Mlflow also uses the same check here
https://github.com/mlflow/mlflow/blob/30462203357010cba51d743a96c0cae5397e7e1b/mlflow/utils/databricks_utils.py#L180

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or the python SDK:
https://github.com/databricks/databricks-sdk-py/blob/3a2798f608da46ec44eebabae21b398d22ccb8f8/databricks/sdk/credentials_provider.py#L88


- name: Run the CLI
run: databricks version
shell: bash

- name: Assert the version of the CLI installed
run: ./setup-cli/assert/version.sh $(cat ./setup-cli/VERSION)
shell: bash

- name: Assert installation path is ~/bin
run: ./setup-cli/assert/path.sh ~/bin/databricks
shell: bash

curl:
Expand All @@ -104,11 +153,57 @@ jobs:
with:
path: ./setup-cli

- name: Assert databricks CLI is not already installed
run: ./setup-cli/assert/not-installed.sh
shell: bash

- run: curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/${{ github.sha }}/install.sh | sh
shell: bash

- run: databricks version
shell: bash

- run: ./setup-cli/test.sh $(cat ./setup-cli/VERSION)
shreyas-goenka marked this conversation as resolved.
Show resolved Hide resolved
- name: Assert the version of the CLI installed
run: ./setup-cli/assert/version.sh $(cat ./setup-cli/VERSION)
shell: bash

- name: Assert installation path is /usr/local/bin for non-windows platforms
run: ./setup-cli/assert/path.sh /usr/local/bin/databricks
if: matrix.os != 'windows-latest'
shell: bash

- name: Assert installation path is /c/Windows for windows platforms
if: matrix.os == 'windows-latest'
run: ./setup-cli/assert/path.sh /c/Windows/databricks

curl-dbr:
# All DBR images are built on top of Ubuntu
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3
with:
path: ./setup-cli

# Append ~/bin to the PATH. This helps with the assertions in the next steps.
shreyas-goenka marked this conversation as resolved.
Show resolved Hide resolved
- run: echo "PATH=$PATH:$HOME/bin" >> $GITHUB_ENV
shreyas-goenka marked this conversation as resolved.
Show resolved Hide resolved

- name: Assert databricks CLI is not already installed
run: ./setup-cli/assert/not-installed.sh
shell: bash

- run: curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/${{ github.sha }}/install.sh | sh
shell: bash
env:
DATABRICKS_RUNTIME_VERSION: value-does-not-matter

- run: databricks version
shell: bash

- name: Assert the version of the CLI installed
run: ./setup-cli/assert/version.sh $(cat ./setup-cli/VERSION)
shell: bash

- name: Assert installation path is ~/bin
run: ./setup-cli/assert/path.sh ~/bin/databricks
shell: bash
10 changes: 10 additions & 0 deletions assert/not-installed.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#!/usr/bin/env bash

set -euo pipefail

# Assert databricks CLI is not already installed
if which databricks >/dev/null 2>&1; then
exit 1
else
exit 0
fi
shreyas-goenka marked this conversation as resolved.
Show resolved Hide resolved
14 changes: 14 additions & 0 deletions assert/path.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/usr/bin/env bash

set -euo pipefail

EXPECTED="${1:-invalid}"
ACTUAL=$(which databricks)

if [[ "$EXPECTED" != "$ACTUAL" ]]; then
echo "Path \"$ACTUAL\" does not match expectation \"$EXPECTED\"."
exit 1
else
echo "Path \"$ACTUAL\" matches expectation."
exit 0
fi
0 test.sh → assert/version.sh
View file
Open in desktop
File renamed without changes.
9 changes: 9 additions & 0 deletions install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,15 @@ MINGW64_NT)
;;
esac

# Set target to ~/bin if DATABRICKS_RUNTIME_VERSION environment variable is set.
if [ -n "$DATABRICKS_RUNTIME_VERSION" ]; then
# Set the installation target to ~/bin when run on DBR
TARGET="$HOME/bin"
shreyas-goenka marked this conversation as resolved.
Show resolved Hide resolved

# Create the target directory if it does not exist
mkdir -p "$TARGET"
fi

# Include architecture in file name.
ARCH="$(uname -m)"
case "$ARCH" in
Expand Down
Loading