Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add base test for vLLM and its metrics #1438

Merged
merged 15 commits into from
May 20, 2024
Merged

Add base test for vLLM and its metrics #1438

merged 15 commits into from
May 20, 2024

Conversation

lugi0
Copy link
Contributor

@lugi0 lugi0 commented May 10, 2024

No description provided.

@lugi0
Add base test for vLLM and its metrics
fff7959 
Signed-off-by: Luca Giorgi <lgiorgi@redhat.com>
@lugi0 lugi0 added new test New test(s) added (PR will be listed in release-notes) do not merge Do not merge this yet please labels May 10, 2024
@lugi0 lugi0 requested review from mwaykole, bdattoma and tarukumar May 10, 2024 15:45
@lugi0 lugi0 self-assigned this May 10, 2024
github-advanced-security[bot]
github-advanced-security bot found potential problems May 10, 2024
View reviewed changes
ods_ci/tests/Resources/Files/llm/vllm/download_model.yaml
namespace: vllm-gpt2
labels:
gpt-download-pod: 'true'
spec:

Check warning

Code scanning / SonarCloud

Service account tokens should not be mounted in pods

<!--SONAR_ISSUE_KEY:AY9jLyHKqk86X9Pay0bC-->Set automountServiceAccountToken to false for this specification of kind Pod. <p>See more on <a href="https://sonarcloud.io/project/issues?id=red-hat-data-services_ods-ci&issues=AY9jLyHKqk86X9Pay0bC&open=AY9jLyHKqk86X9Pay0bC&pullRequest=1438">SonarCloud</a></p>
ods_ci/tests/Resources/Files/llm/vllm/download_model.yaml
- mountPath: "/mnt/models/"
name: model-volume
containers:
- name: download-model

Check warning

Code scanning / SonarCloud

Storage limits should be enforced

<!--SONAR_ISSUE_KEY:AY9jLyHKqk86X9Pay0bD-->Specify a storage limit for this container. <p>See more on <a href="https://sonarcloud.io/project/issues?id=red-hat-data-services_ods-ci&issues=AY9jLyHKqk86X9Pay0bD&open=AY9jLyHKqk86X9Pay0bD&pullRequest=1438">SonarCloud</a></p>
github-advanced-security[bot]
github-advanced-security bot found potential problems May 10, 2024
View reviewed changes
ods_ci/tests/Resources/CLI/ModelServing/llm.resource Dismissed Show dismissed Hide dismissed
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot Fixed Show fixed Hide fixed
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot Dismissed Show dismissed Hide dismissed
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot Fixed Show fixed Hide fixed
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot Fixed Show fixed Hide fixed
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot Dismissed Show dismissed Hide dismissed
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot Fixed Show fixed Hide fixed
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot Fixed Show fixed Hide fixed
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 10, 2024 •
edited
Loading

Robot Results

✅ Passed ❌ Failed ⏭️ Skipped Total Pass %
461 0 0 461 100

lugi0 and others added 2 commits May 13, 2024 14:42
@lugi0 @VedantMahabaleshwarkar
Update ods_ci/tests/Resources/Files/llm/vllm/vllm_servingruntime.yaml
ddaea06 
Co-authored-by: Vedant Mahabaleshwarkar <vmahabal@redhat.com>
@lugi0
reimplement using common keywords
787048e 
Signed-off-by: Luca Giorgi <lgiorgi@redhat.com>
github-advanced-security[bot]
github-advanced-security bot found potential problems May 14, 2024
View reviewed changes
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot Fixed Show fixed Hide fixed
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot Fixed Show fixed Hide fixed
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot Fixed Show fixed Hide fixed
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot Fixed Show fixed Hide fixed
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot
Verify Vllm Metrics Are Present
[Documentation] Confirm vLLM metrics are exposed in OpenShift metrics
[Tags] Tier1 Sanity Resources-GPU ODS-XXX
${host} = llm.Get KServe Inference Host Via CLI isvc_name=vllm-gpt2-openai namespace=${TEST_NS}

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected '=' but got ' =' instead
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot
IF ${is_self_managed}
Configure User Workload Monitoring
Enable User Workload Monitoring
#TODO: Find reliable signal for UWM being ready

Check warning

Code scanning / Robocop

Missing blank space after comment character Warning test

Missing blank space after comment character
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot
Configure User Workload Monitoring
Enable User Workload Monitoring
#TODO: Find reliable signal for UWM being ready
#Sleep 10m

Check warning

Code scanning / Robocop

Missing blank space after comment character Warning test

Missing blank space after comment character
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot
END
Load Expected Responses

Suite Teardown

Check warning

Code scanning / Robocop

Missing documentation in '{{ name }}' keyword Warning test

Missing documentation in 'Suite Teardown' keyword
mwaykole
mwaykole previously approved these changes May 16, 2024
View reviewed changes
@lugi0
small cleanup
6c0c3d9 
Signed-off-by: Luca Giorgi <lgiorgi@redhat.com>
bdattoma
bdattoma previously approved these changes May 16, 2024
View reviewed changes
Copy link
Contributor

@bdattoma bdattoma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving, but we could re-use some keywords and file we already have (e.g., download model and inference serving runtime). I'd suggest raising a new PR to improve it besides considering adding support for Kserve raw deployment

tarukumar
tarukumar previously approved these changes May 16, 2024
View reviewed changes
@lugi0
Move keyword, some cleanup, comments
95093e4 
Signed-off-by: Luca Giorgi <lgiorgi@redhat.com>
@lugi0 lugi0 dismissed stale reviews from tarukumar and bdattoma via 95093e4 May 17, 2024 09:58
github-advanced-security[bot]
github-advanced-security bot found potential problems May 17, 2024
View reviewed changes
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot
${rc} ${out}= Run And Return Rc And Output oc apply -f ${DL_POD_FILEPATH}
Should Be Equal As Integers ${rc} ${0}
Wait For Pods To Succeed label_selector=gpt-download-pod=true namespace=${TEST_NS}
${rc} ${out}= Run And Return Rc And Output oc apply -f ${SR_FILEPATH}

Check notice

Code scanning / Robocop

Variable '{{ name }}' is assigned but not used Note test

Variable '${out}' is assigned but not used
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot
Wait For Pods To Succeed label_selector=gpt-download-pod=true namespace=${TEST_NS}
${rc} ${out}= Run And Return Rc And Output oc apply -f ${SR_FILEPATH}
Should Be Equal As Integers ${rc} ${0}
#TODO: Switch to common keyword for model DL and SR deploy

Check warning

Code scanning / Robocop

Missing blank space after comment character Warning test

Missing blank space after comment character
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot
${rc} ${out}= Run And Return Rc And Output oc apply -f ${SR_FILEPATH}
Should Be Equal As Integers ${rc} ${0}
#TODO: Switch to common keyword for model DL and SR deploy
#Set Project And Runtime runtime=vllm namespace=${TEST_NS}

Check warning

Code scanning / Robocop

Missing blank space after comment character Warning test

Missing blank space after comment character
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot
Should Be Equal As Integers ${rc} ${0}
#TODO: Switch to common keyword for model DL and SR deploy
#Set Project And Runtime runtime=vllm namespace=${TEST_NS}
#... download_in_pvc=${DOWNLOAD_IN_PVC} model_name=gpt2

Check warning

Code scanning / Robocop

Missing blank space after comment character Warning test

Missing blank space after comment character
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot
#TODO: Switch to common keyword for model DL and SR deploy
#Set Project And Runtime runtime=vllm namespace=${TEST_NS}
#... download_in_pvc=${DOWNLOAD_IN_PVC} model_name=gpt2
#... storage_size=10Gi

Check warning

Code scanning / Robocop

Missing blank space after comment character Warning test

Missing blank space after comment character
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot
Verify Vllm Metrics Are Present
[Documentation] Confirm vLLM metrics are exposed in OpenShift metrics
[Tags] Tier1 Sanity Resources-GPU RHOAIENG-6264
... Depends On Test Verify User Can Deploy A Model With Vllm Via CLI

Check warning

Code scanning / Robocop

Tag '{{ tag }}' should not contain spaces Warning test

Tag 'Depends On Test' should not contain spaces
.../Tests/400__ods_dashboard/420__model_serving/LLMs/vllm/426__model_serving_vllm_metrics.robot
Verify Vllm Metrics Are Present
[Documentation] Confirm vLLM metrics are exposed in OpenShift metrics
[Tags] Tier1 Sanity Resources-GPU RHOAIENG-6264
... Depends On Test Verify User Can Deploy A Model With Vllm Via CLI

Check warning

Code scanning / Robocop

Tag '{{ tag }}' should not contain spaces Warning test

Tag 'Verify User Can Deploy A Model With Vllm Via CLI' should not contain spaces
@lugi0 lugi0 added enhancements Bugfixes, enhancements, refactoring, ... in tests or libraries (PR will be listed in release-notes) and removed do not merge Do not merge this yet please labels May 17, 2024
@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Passed Quality Gate passed

Issues
0 New issues
2 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

@lugi0 lugi0 merged commit 7b6969a into red-hat-data-services:master May 20, 2024
8 checks passed
adolfo-ab pushed a commit to adolfo-ab/ods-ci that referenced this pull request May 21, 2024
@lugi0 @VedantMahabaleshwarkar @adolfo-ab
Add base test for vLLM and its metrics (red-hat-data-services#1438)
c0d502b 
* Add base test for vLLM and its metrics

Signed-off-by: Luca Giorgi <lgiorgi@redhat.com>

* Update ods_ci/tests/Resources/Files/llm/vllm/vllm_servingruntime.yaml

Co-authored-by: Vedant Mahabaleshwarkar <vmahabal@redhat.com>

* reimplement using common keywords

Signed-off-by: Luca Giorgi <lgiorgi@redhat.com>

* Better handle missing metrics from UWM, change expected response format

Signed-off-by: Luca Giorgi <lgiorgi@redhat.com>

* small cleanup

Signed-off-by: Luca Giorgi <lgiorgi@redhat.com>

* Move keyword, some cleanup, comments

Signed-off-by: Luca Giorgi <lgiorgi@redhat.com>

---------

Signed-off-by: Luca Giorgi <lgiorgi@redhat.com>
Co-authored-by: Vedant Mahabaleshwarkar <vmahabal@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@VedantMahabaleshwarkar VedantMahabaleshwarkar VedantMahabaleshwarkar left review comments

@Jooho Jooho Jooho left review comments

@tarukumar tarukumar tarukumar approved these changes

@mwaykole mwaykole mwaykole approved these changes

@bdattoma bdattoma bdattoma left review comments

Assignees

@lugi0 lugi0

Labels
enhancements Bugfixes, enhancements, refactoring, ... in tests or libraries (PR will be listed in release-notes) new test New test(s) added (PR will be listed in release-notes)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@lugi0 @Jooho @VedantMahabaleshwarkar @mwaykole @bdattoma @tarukumar