Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Test and Kw for authorino #1396

Merged
merged 6 commits into from
May 7, 2024
Merged

Add Test and Kw for authorino #1396

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9d815d5
Add Test and Kw for authorino
Apr 19, 2024
381658c
Add Test and Kw for authorino
Apr 20, 2024
be2ddc9
Add Test and Kw for authorino
May 6, 2024
6094403
Merge branch 'master' into authzy
mwaykole May 6, 2024
2006dc9
Add Test and Kw for authorino
May 6, 2024
3990774
Add Test and Kw for authorino
May 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions ods_ci/tests/Resources/CLI/ModelServing/llm.resource
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,10 @@
${INFERENCESERVICE_FILLED_FILEPATH}= ${LLM_RESOURCES_DIRPATH}/serving_runtimes/isvc_filled.yaml
${DEFAULT_BUCKET_SECRET_NAME}= models-bucket-secret
${DEFAULT_BUCKET_SA_NAME}= models-bucket-sa
${DEFAULT_BUCKET_PREFIX}= models-bucket
${BUCKET_SECRET_FILEPATH}= ${LLM_RESOURCES_DIRPATH}/bucket_secret.yaml
${BUCKET_SA_FILEPATH}= ${LLM_RESOURCES_DIRPATH}/bucket_sa.yaml
${ROLE_BINDING_FILEPATH}= ${LLM_RESOURCES_DIRPATH}/rolebinding_view.yaml
${USE_BUCKET_HTTPS}= "1"
${MODELS_BUCKET}= ${S3.BUCKET_3}
${SERVICEMESH_CR_NS}= istio-system
Expand Down Expand Up @@ -315,6 +317,7 @@
... ${port}=443
... ${body_params}=&{EMPTY}
... ${cert}=${False}
... ${token}=${None}
... &{args}
IF "${inference_type}" == "streaming"
${streamed_response}= Set Variable ${TRUE}
Expand All @@ -339,6 +342,9 @@
${body} ${header} ${extra_args}= llm.Prepare Payload runtime=${runtime} protocol=${protocol}
... inference_type=${inference_type} model_name=${model_name} body_params=${body_params}
... query_text=${EXP_RESPONSES}[queries][${query_idx}][query_text]
IF "${token}" != "${None}"
${header}= Set Variable "Authorization: Bearer ${token}" -H ${header}
END
${runtime_details}= Set Variable ${RUNTIME_FORMATS}[${runtime}][${inference_type}][${protocol}]
${endpoint}= Set Variable ${runtime_details}[endpoint]
Set To Dictionary ${args} &{extra_args}
Expand Down Expand Up @@ -774,3 +780,14 @@
... alias=${process_alias} stderr=STDOUT shell=yes
Process Should Be Running ${process}
sleep 7s

Create Role Binding For Authorino

Check warning

Code scanning / Robocop

Missing documentation in '{{ name }}' keyword Warning test

Missing documentation in 'Create Role Binding For Authorino' keyword
[Arguments] ${name} ${namespace}
Set Test Variable ${name}

Check notice

Code scanning / Robocop

{{ set_variable_keyword }} can be replaced with VAR Note test

Set Test Variable can be replaced with VAR

Check warning

Code scanning / Robocop

Test, suite and global variables should be uppercase Warning test

Test, suite and global variables should be uppercase
Set Test Variable ${namespace}

Check notice

Code scanning / Robocop

{{ set_variable_keyword }} can be replaced with VAR Note test

Set Test Variable can be replaced with VAR

Check warning

Code scanning / Robocop

Test, suite and global variables should be uppercase Warning test

Test, suite and global variables should be uppercase
Set Test Variable ${nameview} ${name}-view

Check notice

Code scanning / Robocop

{{ set_variable_keyword }} can be replaced with VAR Note test

Set Test Variable can be replaced with VAR

Check warning

Code scanning / Robocop

Test, suite and global variables should be uppercase Warning test

Test, suite and global variables should be uppercase
Set Test Variable ${namesa} ${name}-sa

Check notice

Code scanning / Robocop

{{ set_variable_keyword }} can be replaced with VAR Note test

Set Test Variable can be replaced with VAR

Check warning

Code scanning / Robocop

Test, suite and global variables should be uppercase Warning test

Test, suite and global variables should be uppercase
Create File From Template ${ROLE_BINDING_FILEPATH} ${LLM_RESOURCES_DIRPATH}/rb.filled.yaml
${rc} ${out}= Run And Return Rc And Output

Check notice

Code scanning / Robocop

Variable '{{ name }}' is assigned but not used Note test

Variable '${out}' is assigned but not used
... oc apply -f ${LLM_RESOURCES_DIRPATH}/rb.filled.yaml
Should Be Equal As Integers ${rc} ${0}

Check warning

Code scanning / Robocop

File has too many lines ({{ lines_count }}/{{max_allowed_count }}) Warning test

File has too many lines (584/400)
12 changes: 12 additions & 0 deletions ods_ci/tests/Resources/Files/llm/rolebinding_view.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: ${NAMEVIEW}
namespace: ${NAMESPACE}
subjects:
- kind: ServiceAccount
name: ${NAMESA}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: view
10 changes: 10 additions & 0 deletions ods_ci/tests/Resources/Files/llm/serving_runtimes/overlay/authorino/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
name: authorino
resources:
- ../../base/

commonAnnotations:
security.opendatahub.io/enable-auth: "true"

6 changes: 6 additions & 0 deletions ods_ci/tests/Resources/RHOSi.resource
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,3 +97,9 @@
Set Suite Variable ${OPERATOR_DEPLOYMENT_NAME} opendatahub-operator-controller-manager
Set Suite Variable ${OPERATOR_LABEL_SELECTOR} control-plane=controller-manager
END

Create Inference Access Token

Check warning

Code scanning / Robocop

Missing documentation in '{{ name }}' keyword Warning test

Missing documentation in 'Create Inference Access Token' keyword
mwaykole marked this conversation as resolved.
Show resolved Hide resolved
[Arguments] ${test_namespace} ${bucket_sa_name}
${rc} ${out}= Run And Return Rc And Output oc create token -n ${test_namespace} ${bucket_sa_name}
Should Be Equal As Strings ${rc} 0
[Return] ${out}

Check warning

Code scanning / Robocop

Missing trailing blank line at the end of file Warning test

Missing trailing blank line at the end of file

Check warning

Code scanning / Robocop

'{{ statement_name }}' is deprecated since Robot Framework version {{ version }}, use '{{ alternative }}' instead Warning test

'[Return]' is deprecated since Robot Framework version 5.*, use 'RETURN' instead
30 changes: 30 additions & 0 deletions ods_ci/tests/Tests/400__ods_dashboard/420__model_serving/LLMs/422__model_serving_llm.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -550,6 +550,36 @@
[Teardown] Clean Up Test Project test_ns=${test_namespace}
... isvc_names=${models_names} wait_prj_deletion=${FALSE}

Verify User Can Serve And Query A Model With Token

Check warning

Code scanning / Robocop

Test case '{{ test_name }}' is too long ({{ test_length }}/{{ allowed_length }}) Warning test

Test case 'Verify User Can Serve And Query A Model With Token' is too long (29/20)

Check warning

Code scanning / Robocop

Test case '{{ test_name }}' has too many keywords inside ({{ keyword_count }}/{{ max_allowed_count }}) Warning test

Test case 'Verify User Can Serve And Query A Model With Token' has too many keywords inside (11/10)
[Documentation] Basic tests for preparing, deploying and querying a LLM model
... With Token using Kserve and Caikit+TGIS runtime
[Tags] RHOAIENG-6333
... Tier1
[Setup] Set Project And Runtime namespace=${TEST_NS}-cli
${test_namespace}= Set Variable ${TEST_NS}-cli
${flan_model_name}= Set Variable flan-t5-small-caikit
${models_names}= Create List ${flan_model_name}

Check notice

Code scanning / Robocop

{{ create_keyword }} can be replaced with VAR Note test

Create List can be replaced with VAR
${overlays}= Create List authz

Check notice

Code scanning / Robocop

{{ create_keyword }} can be replaced with VAR Note test

Create List can be replaced with VAR
Compile Inference Service YAML isvc_name=${flan_model_name}
... sa_name=${DEFAULT_BUCKET_SA_NAME}
... model_storage_uri=${FLAN_STORAGE_URI}
... overlays=${overlays}

Deploy Model Via CLI isvc_filepath=${INFERENCESERVICE_FILLED_FILEPATH}
... namespace=${test_namespace}
Wait For Pods To Be Ready label_selector=serving.kserve.io/inferenceservice=${flan_model_name}
... namespace=${test_namespace}
Create Role Binding For Authorino name=${DEFAULT_BUCKET_PREFIX} namespace=${test_namespace}

Check warning

Code scanning / Robocop

{{ bad_indent_msg }} Warning test

Line is under-indented
${inf_token} Create Inference Access Token ${test_namespace} ${DEFAULT_BUCKET_SA_NAME}

Check warning

Code scanning / Robocop

{{ bad_indent_msg }} Warning test

Line is under-indented

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected '=' but got '' instead
Query Model Multiple Times model_name=${flan_model_name}
... inference_type=all-tokens n_times=1
... namespace=${test_namespace} token=${inf_token}
Query Model Multiple Times model_name=${flan_model_name}
... inference_type=streaming n_times=1
... namespace=${test_namespace} token=${inf_token}

[Teardown] Clean Up Test Project test_ns=${test_namespace}
... isvc_names=${models_names} wait_prj_deletion=${FALSE}

*** Keywords ***
Install Model Serving Stack Dependencies
Expand Down
48 changes: 48 additions & 0 deletions .../tests/Tests/400__ods_dashboard/420__model_serving/LLMs/422__model_serving_llm_tgis.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -625,6 +625,54 @@
[Teardown] Clean Up Test Project test_ns=${test_namespace}
... isvc_names=${models_names} wait_prj_deletion=${FALSE}

Verify User Can Serve And Query A Model With Token

Check warning

Code scanning / Robocop

Test case '{{ test_name }}' is too long ({{ test_length }}/{{ allowed_length }}) Warning test

Test case 'Verify User Can Serve And Query A Model With Token' is too long (45/20)

Check warning

Code scanning / Robocop

Test case '{{ test_name }}' has too many keywords inside ({{ keyword_count }}/{{ max_allowed_count }}) Warning test

Test case 'Verify User Can Serve And Query A Model With Token' has too many keywords inside (15/10)
[Documentation] Basic tests for preparing, deploying and querying a LLM model

Check warning

Code scanning / Robocop

{{ bad_indent_msg }} Warning test

Line is over-indented
mwaykole marked this conversation as resolved.
Show resolved Hide resolved
... with token using Kserve and TGIS runtime
... With Token using Kserve and Caikit+TGIS runtime
[Tags] RHOAIENG-6306

Check warning

Code scanning / Robocop

{{ bad_indent_msg }} Warning test

Line is over-indented
... Tier1
[Setup] Set Project And Runtime runtime=${TGIS_RUNTIME_NAME} namespace=${TEST_NS}-cli

Check warning

Code scanning / Robocop

{{ bad_indent_msg }} Warning test

Line is over-indented
${test_namespace}= Set Variable ${TEST_NS}-cli

Check warning

Code scanning / Robocop

{{ bad_indent_msg }} Warning test

Line is over-indented
${flan_model_name}= Set Variable flan-t5-small-caikit

Check warning

Code scanning / Robocop

{{ bad_indent_msg }} Warning test

Line is over-indented
${models_names}= Create List ${flan_model_name}

Check warning

Code scanning / Robocop

{{ bad_indent_msg }} Warning test

Line is over-indented

Check notice

Code scanning / Robocop

{{ create_keyword }} can be replaced with VAR Note test

Create List can be replaced with VAR
${overlays}= Create List authz

Check warning

Code scanning / Robocop

{{ bad_indent_msg }} Warning test

Line is over-indented

Check notice

Code scanning / Robocop

{{ create_keyword }} can be replaced with VAR Note test

Create List can be replaced with VAR

Compile Inference Service YAML isvc_name=${flan_model_name}
... sa_name=${DEFAULT_BUCKET_SA_NAME}
... model_storage_uri=${FLAN_STORAGE_URI}
... model_format=pytorch serving_runtime=${TGIS_RUNTIME_NAME}
... limits_dict=${GPU_LIMITS}
... overlays=${overlays}
Deploy Model Via CLI isvc_filepath=${INFERENCESERVICE_FILLED_FILEPATH}
... namespace=${test_namespace}
Wait For Pods To Be Ready label_selector=serving.kserve.io/inferenceservice=${flan_model_name}
... namespace=${test_namespace}
Create Role Binding For Authorino name=${DEFAULT_BUCKET_PREFIX} namespace=tgis-standalone-cli
${inf_token} Create Inference Access Token ${test_namespace} ${DEFAULT_BUCKET_SA_NAME}

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected '=' but got '' instead
${pod_name}= Get Pod Name namespace=${test_namespace} label_selector=serving.kserve.io/inferenceservice=${flan_model_name}

Check warning

Code scanning / Robocop

Line is too long ({{ line_length }}/{{ allowed_length }}) Warning test

Line is too long (132/120)
IF ${IS_KSERVE_RAW} Start Port-forwarding namespace=${test_namespace} pod_name=${pod_name}
Query Model Multiple Times model_name=${flan_model_name} runtime=${TGIS_RUNTIME_NAME}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you add a Query to the model without passing the token and check it fails ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

discussed offline and agreed on adding the test in the next PR

... inference_type=all-tokens n_times=1
... namespace=${test_namespace} port_forwarding=${IS_KSERVE_RAW} token=${inf_token}
Query Model Multiple Times model_name=${flan_model_name} runtime=${TGIS_RUNTIME_NAME}
... inference_type=tokenize n_times=1 port_forwarding=${IS_KSERVE_RAW}

Check warning

Code scanning / Robocop

Continuation marker should be aligned with starting row Warning test

Continuation marker should be aligned with starting row
... namespace=${test_namespace} validate_response=${TRUE} string_check_only=${TRUE}

Check warning

Code scanning / Robocop

Continuation marker should be aligned with starting row Warning test

Continuation marker should be aligned with starting row
... token=${inf_token}

Check warning

Code scanning / Robocop

Continuation marker should be aligned with starting row Warning test

Continuation marker should be aligned with starting row
Query Model Multiple Times model_name=${flan_model_name} runtime=${TGIS_RUNTIME_NAME}
... inference_type=model-info n_times=1 port_forwarding=${IS_KSERVE_RAW}

Check warning

Code scanning / Robocop

Continuation marker should be aligned with starting row Warning test

Continuation marker should be aligned with starting row
... namespace=${test_namespace} validate_response=${TRUE} string_check_only=${TRUE}

Check warning

Code scanning / Robocop

Continuation marker should be aligned with starting row Warning test

Continuation marker should be aligned with starting row
... token=${inf_token}

Check warning

Code scanning / Robocop

Continuation marker should be aligned with starting row Warning test

Continuation marker should be aligned with starting row
Query Model Multiple Times model_name=${flan_model_name} runtime=${TGIS_RUNTIME_NAME}
... inference_type=streaming n_times=1 port_forwarding=${IS_KSERVE_RAW}

Check warning

Code scanning / Robocop

Continuation marker should be aligned with starting row Warning test

Continuation marker should be aligned with starting row
... namespace=${test_namespace} validate_response=${FALSE}

Check warning

Code scanning / Robocop

Continuation marker should be aligned with starting row Warning test

Continuation marker should be aligned with starting row
... token=${inf_token}

Check warning

Code scanning / Robocop

Continuation marker should be aligned with starting row Warning test

Continuation marker should be aligned with starting row

[Teardown] Run Keywords

Check warning

Code scanning / Robocop

{{ bad_indent_msg }} Warning test

Line is under-indented
... Clean Up Test Project test_ns=${test_namespace}
... isvc_names=${models_names} wait_prj_deletion=${FALSE}
... AND
... Run Keyword If ${IS_KSERVE_RAW} Terminate Process llm-query-process kill=true


*** Keywords ***
Suite Setup
Expand Down
Loading