Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Jupyuterhub, workbench and model serving related tests #971

Merged
merged 32 commits into from
Oct 27, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
6b89749
fix minimal-cuda-test
FedeAlonso Oct 7, 2023
13029f6
Merge branch 'master' into fix/refactor_jupyter_nb_accelerators
FedeAlonso Oct 9, 2023
37e240d
fix workbenches
FedeAlonso Oct 17, 2023
3bad746
Merge branch 'fix/refactor_jupyter_nb_accelerators' of github.com:red…
FedeAlonso Oct 17, 2023
7acebd5
Merge branch 'master' of github.com:red-hat-data-services/ods-ci into…
FedeAlonso Oct 17, 2023
b29b2cb
fix model serving
FedeAlonso Oct 17, 2023
62544c7
Merge branch 'master' into fix/refactor_jupyter_nb_accelerators
FedeAlonso Oct 17, 2023
7339697
Linter fixes
FedeAlonso Oct 17, 2023
e2e74fb
Merge branch 'fix/refactor_jupyter_nb_accelerators' of github.com:red…
FedeAlonso Oct 17, 2023
cbf6eec
PR Fixes
FedeAlonso Oct 18, 2023
58f7086
PR fixes
FedeAlonso Oct 18, 2023
7e13d55
fix PR comment: Generic accelerator setter
FedeAlonso Oct 19, 2023
e77a26f
Merge branch 'master' into fix/refactor_jupyter_nb_accelerators
FedeAlonso Oct 20, 2023
734a405
Add a rerun migration for accelerators in gpu deploy script
FedeAlonso Oct 20, 2023
c3ad521
PR fixes
FedeAlonso Oct 20, 2023
a355851
Update ods_ci/tasks/Resources/Provisioning/GPU/gpu_deploy.sh
FedeAlonso Oct 23, 2023
a01fe71
Update ods_ci/tasks/Resources/Provisioning/GPU/gpu_deploy.sh
FedeAlonso Oct 23, 2023
043a198
Update ods_ci/tasks/Resources/Provisioning/GPU/gpu_deploy.sh
FedeAlonso Oct 23, 2023
097807a
PR fixes
FedeAlonso Oct 23, 2023
8d37c3b
Merge branch 'master' into fix/refactor_jupyter_nb_accelerators
FedeAlonso Oct 24, 2023
1c7c280
delete one pod instead the complete dashboard replica set
FedeAlonso Oct 25, 2023
4cc8da4
Pod deletion more clean and smart
FedeAlonso Oct 25, 2023
b1fe64c
modify delete dashboard error message
FedeAlonso Oct 25, 2023
1a9484f
Merge branch 'master' into fix/refactor_jupyter_nb_accelerators
FedeAlonso Oct 25, 2023
9c8746a
Rollback restart instead of pod deletion
FedeAlonso Oct 25, 2023
bec218e
Merge branch 'fix/refactor_jupyter_nb_accelerators' of github.com:red…
FedeAlonso Oct 25, 2023
dbda6a4
Merge branch 'master' into fix/refactor_jupyter_nb_accelerators
FedeAlonso Oct 25, 2023
23c43ff
fix typo in variable in workbenches
FedeAlonso Oct 26, 2023
a2bfb4a
Merge branch 'fix/refactor_jupyter_nb_accelerators' of github.com:red…
FedeAlonso Oct 26, 2023
f9577ca
Delete unused variables in gpu_deploy script
FedeAlonso Oct 26, 2023
a0ff660
Merge branch 'master' of github.com:red-hat-data-services/ods-ci into…
FedeAlonso Oct 26, 2023
206acd5
Merge branch 'master' into fix/refactor_jupyter_nb_accelerators
FedeAlonso Oct 27, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 37 additions & 16 deletions ods_ci/tests/Resources/Page/ODH/JupyterHub/JupyterHubSpawner.robot
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,13 @@ ${KFNBC_SPAWNER_HEADER_XPATH} = //h1[.="Start a notebook server"]
${JUPYTERHUB_DROPDOWN_XPATH} = //button[@aria-label="Options menu"]
${KFNBC_CONTAINER_SIZE_TITLE} = //div[.="Deployment size"]/..//span[.="Container Size"]
${KFNBC_CONTAINER_SIZE_DROPDOWN_XPATH} = //label[@for="modal-notebook-container-size"]/../..//button[@aria-label="Options menu"]
${KFNBC_GPU_DROPDOWN_XPATH} = //button[contains(@aria-labelledby, "gpu-numbers")]
${KFNBC_ACCELERATOR_HEADER_XPATH} = //span[text()='Accelerator']
${KFNBC_ACCELERATOR_DROPDOWN_XPATH} = //label[@for='modal-notebook-accelerator']/ancestor::div[@class='pf-c-form__group']/descendant::button
${KFNBC_ACCELERATOR_DROPDOWN_NVIDIA_XPATH} = //div[@class and text()='Nvidia GPU']
${KFNBC_ACCELERATOR_INPUT_XPATH} = //input[@aria-label='Number of accelerators']
${KFNBC_ACCELERATOR_LESS_BUTTON_XPATH} = ${KFNBC_ACCELERATOR_INPUT_XPATH}/preceding-sibling::button
${KFNBC_ACCELERATOR_PLUS_BUTTON_XPATH} = ${KFNBC_ACCELERATOR_INPUT_XPATH}/following-sibling::button
${KFNBC_MAX_ACCELERATOR_WARNING_XPATH} = //div[@aria-label='Warning Alert']//h4[contains(text(), 'accelerator detected')]

Check warning

Code scanning / Robocop

Line is too long ({{ line_length }}/{{ allowed_length }}) Warning test

Line is too long (124/120)
${KFNBC_MODAL_HEADER_XPATH} = //div[@aria-label="Starting server modal"]
${KFNBC_MODAL_CANCEL_XPATH} = ${KFNBC_MODAL_HEADER_XPATH}//button[.="Cancel"]
${KFNBC_MODAL_CLOSE_XPATH} = ${KFNBC_MODAL_HEADER_XPATH}//button[.="Close"]
Expand Down Expand Up @@ -92,26 +98,40 @@ Select Container Size
Click Element xpath:${JUPYTERHUB_DROPDOWN_XPATH}\[1]
Click Element xpath://span[.="${container_size}"]/../..

Wait Until GPU Dropdown Exists
[Documentation] Verifies that the dropdown to select the no. of GPUs exists
Wait Until Page Contains Number of GPUs
Wait until Accelerator Dropdown Exists
[Documentation] Verifies that the dropdown to select the Accelerator exists
Page Should Not Contain All GPUs are currently in use, try again later.
Wait Until Page Contains Element xpath:${KFNBC_GPU_DROPDOWN_XPATH}
... error=GPU selector is not present in JupyterHub Spawner
Wait Until Page Contains Element xpath:${KFNBC_ACCELERATOR_DROPDOWN_XPATH}
... error=Accelerator selector is not present in JupyterHub Spawner

Set Number Of Required GPUs
[Documentation] Sets the gpu count based on the ${gpus} argument
Set NVidia GPU Accelerator
[Documentation] Set NVidia GPU Accelerator
Click Element xpath:${KFNBC_ACCELERATOR_DROPDOWN_XPATH}
Click Element xpath:${KFNBC_ACCELERATOR_DROPDOWN_NVIDIA_XPATH}

Set Number Of Required Accelerators
[Documentation] Sets the Accelerators count based on the ${gpus} argument
[Arguments] ${gpus}
FedeAlonso marked this conversation as resolved.
Show resolved Hide resolved
Click Element xpath:${KFNBC_GPU_DROPDOWN_XPATH}
Click Element xpath:${KFNBC_GPU_DROPDOWN_XPATH}/../..//button[.="${gpus}"]
${acc_num}= Get Value xpath:${KFNBC_ACCELERATOR_INPUT_XPATH}
Log Actual num of Accelerators: ${acc_num}
IF ${acc_num} != ${gpus}
Input Text ${KFNBC_ACCELERATOR_INPUT_XPATH} ${gpus}
END


Fetch Max Number Of GPUs In Spawner Page
[Documentation] Returns the maximum number of GPUs a user can request from the spawner
${gpu_visible} = Run Keyword And Return Status Wait Until GPU Dropdown Exists
${gpu_visible} = Run Keyword And Return Status Wait until Accelerator Dropdown Exists
IF ${gpu_visible}==True
Click Element xpath:${KFNBC_GPU_DROPDOWN_XPATH}
${maxGPUs} = Get Text xpath://li[@class="pf-c-select__menu-wrapper"][last()]/button
${maxGPUs} = Convert To Integer ${maxGPUs}
Set NVidia GPU Accelerator
${max_operator_detected}= Run Keyword And Return Status Page Should Contain Element xpath=${KFNBC_MAX_ACCELERATOR_WARNING_XPATH}
WHILE ${max_operator_detected} == ${FALSE}
Click Element xpath:${KFNBC_ACCELERATOR_PLUS_BUTTON_XPATH}
${max_operator_detected}= Run Keyword And Return Status Page Should Contain Element xpath=${KFNBC_MAX_ACCELERATOR_WARNING_XPATH}
${maxGPUs} = Get Value xpath:${KFNBC_ACCELERATOR_INPUT_XPATH}
${maxGPUs} = Convert To Integer ${maxGPUs}
${maxGPUs} = Set Variable ${maxGPUs-1}
END
ELSE
${maxGPUs} = Set Variable ${0}
END
Expand Down Expand Up @@ -262,9 +282,10 @@ Spawn Notebook With Arguments # robocop: disable
IF ${spawner_ready}==True
Select Notebook Image ${image} ${version}
Select Container Size ${size}
${gpu_visible} = Run Keyword And Return Status Wait Until GPU Dropdown Exists
${gpu_visible} = Run Keyword And Return Status Wait until Accelerator Dropdown Exists
IF ${gpu_visible}==True and ${gpus}>0
Set Number Of Required GPUs ${gpus}
Set NVidia GPU Accelerator
Set Number Of Required Accelerators ${gpus}
ELSE IF ${gpu_visible}==False and ${gpus}>0
IF ${index} < ${retries}
Sleep 30s reason=Wait for GPU to free up
Expand Down
3 changes: 2 additions & 1 deletion ods_ci/tests/Tests/500__jupyterhub/autoscaling-gpus.robot
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,8 @@ Spawn Notebook And Trigger Autoscale
... of the GPU node.
Select Notebook Image ${NOTEBOOK_IMAGE}
Select Container Size Small
Set Number Of Required GPUs 1
Set NVidia GPU Accelerator
Set Number Of Required Accelerators 1
Spawn Notebook spawner_timeout=20 minutes expect_autoscaling=${True}
Run Keyword And Warn On Failure Wait Until Page Contains Log in with OpenShift timeout=15s
${oauth_prompt_visible} = Is OpenShift OAuth Login Prompt Visible
Expand Down