Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhancement: add monitoring test to check targets are up and running #2165

Merged
merged 2 commits into from
Jan 10, 2025
Merged

enhancement: add monitoring test to check targets are up and running #2165

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a3ee87f
enhancement: add monitoring test to check targets are up and running
CFSNM Jan 9, 2025
9386e30
Merge branch 'master' into ods-179-prometheus-targets-up
CFSNM Jan 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions ods_ci/tests/Resources/Page/ODH/Prometheus/Prometheus.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,22 @@
Wait Until Keyword Succeeds ${timeout} 30s
... Alert Should Not Be Firing ${pm_url} ${pm_token} ${rule_group} ${alert} ${alert-duration}

Get Target Pools
[Documentation] Returns list of Targets scrape pools
[Arguments] ${pm_url} ${pm_token} ${username} ${password}
${links}= Run curl --silent -X GET -H "Authorization:Bearer ${pm_token}" -u ${username}:${password} -k ${pm_url}/api/v1/targets | jq '.data.activeTargets[] | .scrapePool' #robocop:disable
${links}= Replace String ${links} " ${EMPTY}
@{links}= Split String ${links} \n

Check notice

Code scanning / Robocop

Variable '{{ variable_name }}' may overwrite similar variable inside '{{ block_name }}' {{ block_type }}. Note that variables are case-insensitive, and also spaces and underscores are ignored. Note test

Variable '@{links}' may overwrite similar variable inside 'Get Target Pools' Keyword. Note that variables are case-insensitive, and also spaces and underscores are ignored.
RETURN ${links}

Get Target Pools Which Have State Up
[Documentation] Returns list of Targets scrape pools which have state as "UP"
[Arguments] ${pm_url} ${pm_token} ${username} ${password}
${links}= Run curl --silent -X GET -H "Authorization:Bearer ${pm_token}" -u ${username}:${password} -k ${pm_url}/api/v1/targets | jq '.data.activeTargets[] | select(.health == "up") | .scrapePool' #robocop:disable
${links}= Replace String ${links} " ${EMPTY}
@{links}= Split String ${links} \n

Check notice

Code scanning / Robocop

Variable '{{ variable_name }}' may overwrite similar variable inside '{{ block_name }}' {{ block_type }}. Note that variables are case-insensitive, and also spaces and underscores are ignored. Note test

Variable '@{links}' may overwrite similar variable inside 'Get Target Pools Which Have State Up' Keyword. Note that variables are case-insensitive, and also spaces and underscores are ignored.
RETURN ${links}

Get Target Endpoints
[Documentation] Returns list of Endpoint URLs
[Arguments] ${target_name} ${pm_url} ${pm_token} ${username} ${password}
Expand Down
30 changes: 30 additions & 0 deletions ods_ci/tests/Tests/0100__platform/0102__monitor_and_manage/0101__metrics/0101__metrics.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,36 @@
Should Contain ${list_values} ${resp.json()["data"]["result"][0]["value"][-1]}


Test Targets Are Available And Up In RHOAI Prometheus

Check warning

Code scanning / Robocop

Test case '{{ test_name }}' is too long ({{ test_length }}/{{ allowed_length }}) Warning test

Test case 'Test Targets Are Available And Up In RHOAI Prometheus' is too long (27/20)

Check warning

Code scanning / Robocop

Test case '{{ test_name }}' has too many keywords inside ({{ keyword_count }}/{{ max_allowed_count }}) Warning test

Test case 'Test Targets Are Available And Up In RHOAI Prometheus' has too many keywords inside (17/10)
[Documentation] Verifies the expected targets in Prometheus are available and up running
[Tags] Sanity
... Tier1
... ODS-179
... RHOAIENG-13066
... Monitoring
Skip If RHODS Is Self-Managed
@{targets} = Prometheus.Get Target Pools Which Have State Up
... pm_url=${RHODS_PROMETHEUS_URL}
... pm_token=${RHODS_PROMETHEUS_TOKEN}
... username=${OCP_ADMIN_USER.USERNAME}
... password=${OCP_ADMIN_USER.PASSWORD}
List Should Contain Value ${targets} CodeFlare Operator
List Should Contain Value ${targets} Data Science Pipelines Operator
List Should Contain Value ${targets} Federate Prometheus
List Should Contain Value ${targets} Kserve Controller Manager
List Should Contain Value ${targets} KubeRay Operator
List Should Contain Value ${targets} Kubeflow Notebook Controller Service Metrics
List Should Contain Value ${targets} Kueue Operator
List Should Contain Value ${targets} Modelmesh Controller
List Should Contain Value ${targets} ODH Model Controller
List Should Contain Value ${targets} ODH Notebook Controller Service Metrics
List Should Contain Value ${targets} TrustyAI Controller Manager
List Should Contain Value ${targets} user_facing_endpoints_status_codeflare
List Should Contain Value ${targets} user_facing_endpoints_status_dsp
List Should Contain Value ${targets} user_facing_endpoints_status_rhods_dashboard
List Should Contain Value ${targets} user_facing_endpoints_status_workbenches


*** Keywords ***
Begin Metrics Web Test
[Documentation] Test Setup
Expand Down
Loading