Skip to content

Commit

Permalink
module updates
Browse files Browse the repository at this point in the history
  • Loading branch information
mrobson committed Jan 10, 2025
1 parent 418cc10 commit 8c99567
Show file tree
Hide file tree
Showing 5 changed files with 26 additions and 165 deletions.
23 changes: 12 additions & 11 deletions content/modules/ROOT/pages/index.adoc
Original file line number Diff line number Diff line change
@@ -1,21 +1,22 @@
= {lab_name}

Welcome to Red Hat One 2025 and this exciting, hand-on lab experience. Throughout this lab, we will explore a wide variety of tools available to analyze, search and debug OpenShift cluster data to find patterns, issues and other useful information that will help you engage with your customers.
Welcome to Red Hat One 2025 and this exciting, hand-on, lab experience. Throughout this lab, we will explore a wide variety of tools available to analyze, search and debug OpenShift cluster data to find patterns, issues and other useful information that will help you engage with your customers.

image::openshift-logo.svg[openshift-logo,100%,100%]

== Lab structure

This lab is broken down into a set of 9 unique modules that explore a variety of tools. There is no specific ordering or pre-requisite to move between each module, but we do recommend you review `Module 1` and `Module 2`.
This lab is broken down into a set of 10 unique modules that explore a variety of tools. There is no specific ordering or pre-requisite to move between each module, but we do recommend you review `Module 1` and `Module 2`.

Once you have reviewed `Module 1` and `Module 2`, feel free to continue to `Module 3` or pick one that interests you the most.

. Module 1 introduces all of the tools we will be using
. Module 2 does a deeper dive on the functionality in the `omc` tool
. Module 3 uses `omc` to explore an issue with a vSphere IPI cluster
. Module 4 uses `kubectl-dev_tool` to explore a cluster with an overloaded OpenShift API
. Module 5 uses `ocp_insights.sh` to proactively look for performance issues with OpenShift Insights data
. Module 6 uses `etcd-ocp-diag.sh` to review etcd logs for performance related issues
. Module 7 uses `omc` to review Cluster Upgrades
. Module 8 uses `omc` to review install OLM Operators
. Module 9 uses `omc` and _manual checks_ on the cluster must-gather in order to explore an OCP networking issue
. `Module 1` introduces all of the tools we will be using
. `Module 2` does a deeper dive on the functionality in the `omc` tool
. `Module 3` uses `omc` to explore an issue with a vSphere IPI cluster
. `Module 4` uses `kubectl-dev_tool` to explore a cluster with an overloaded OpenShift API
. `Module 5` uses `ocp_insights.sh` to proactively look for performance issues with OpenShift Insights data
. `Module 6` uses `etcd-ocp-diag.sh` to review etcd logs for performance related issues
. `Module 7` uses `omc` to review Cluster Upgrades
. `Module 8` uses `omc` to review install OLM Operators
. `Module 9` uses `omc` and _manual checks_ on the cluster must-gather in order to explore an OCP networking issue
. `Module 10` uses `koff` to get and inspect data from an etcd database snapshot
2 changes: 1 addition & 1 deletion content/modules/ROOT/pages/module-01.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Inspect your `audit logs` with ease
[#koffintro]
=== `koff`

Read objects from the `etcd` snapshot
Read objects from an `etcd` snapshot

[#yqjqintro]
=== `yq` and `jq`
Expand Down
70 changes: 7 additions & 63 deletions content/modules/ROOT/pages/module-05.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,12 @@ In this module we will be reviewing how to look at a script that parses an `Open

.ocp_insights
====
[source,bash]
----
cd ~/Module5/
----
[source,bash]
----
$ ocp_insights.sh --help
Expand Down Expand Up @@ -61,11 +67,6 @@ Options:
.Output
====
[source,bash]
----
cd ~/Module5/
----
[source,bash]
----
$ ocp_insights.sh --file module5-insights-data
Expand All @@ -86,64 +87,7 @@ ingressIP: 10.0.0.1
etcd Encryption: None
Audit Profile: Default
NAME READY ROLE CREATED ON VERSION OS CPU MEMORY
ocp4-2nvq7-8qn4p True worker 2024-08-01T21:07:31Z v1.27.13+048520e Red Hat Enterprise Linux CoreOS 414.92.202405162017-0 (Plow) 16 63G
ocp4-2nvq7-b2gvz True worker 2024-08-01T21:18:13Z v1.27.13+048520e Red Hat Enterprise Linux CoreOS 414.92.202405162017-0 (Plow) 16 63G
ocp4-2nvq7-master-0 True master 2022-11-17T16:30:11Z v1.27.13+048520e Red Hat Enterprise Linux CoreOS 414.92.202405162017-0 (Plow) 8 63G
ocp4-2nvq7-master-1 True master 2022-11-17T16:29:32Z v1.27.13+048520e Red Hat Enterprise Linux CoreOS 414.92.202405162017-0 (Plow) 8 63G
ocp4-2nvq7-master-2 True master 2022-11-17T16:29:53Z v1.27.13+048520e Red Hat Enterprise Linux CoreOS 414.92.202405162017-0 (Plow) 8 63G
NAME VERSION AVAILABLE PROGRESSING DEGRADED
authentication 4.14.27 True False False
baremetal 4.14.27 True False False
cloud-controller-manager 4.14.27 True False False
...
service-ca 4.14.27 True False False
storage 4.14.27 True False False
Installed Operators:
grafana-operator.v5.12.0
openshift-gitops-operator.v1.12.5
quay-operator.v3.10.6
Installed OLM Operators:
DISPLAY NAME VERSION NAME
Ansible Automation Platform v2.4.0-0.1692675723 ansible-automation-platform-operator.aap
Grafana Operator v5.12.0 grafana-operator.openshift-operators
Red Hat OpenShift GitOps v1.12.5 openshift-gitops-operator.openshift-operators
Portworx Enterprise v24.1.1 portworx-certified.openshift-operators
Red Hat Quay v3.9.8 quay-operator.openshift-operators
MachineConfigPools:
NAME CONFIG PAUSED UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT
master rendered-master-8831ba6d556d1c6a582116beaa537dbb False True False False 3 3 3 0
worker rendered-worker-b33efe42325e084f9dcef59f47b93fc9 False True False False 5 5 5 0
MachineSets:
NAME DESIRED CURRENT READY AVAILABLE
ocp4-2nvq7-infra 2 2 2 2
ocp4-2nvq7-worker 3 3 3 3
Cluster Namespace Memory Usage.
NAMESPACE MEMORY
kube-system 632.0000K
openshift-apiserver 2.1674G
...
openshift-user-workload-monitoring 998.7500M
openshift-vsphere-infra 2.4600G
Total Cluster Namespace Memory Usage: 65.0040G
ALERT NAME STATE START TIME
ArgoCDSyncAlert ACTIVE 2024-08-12T18:52:43.454Z
To see all Alerts run: jq -r . insights-2024-08-14-144858/config/alerts.json
etc...
----
====
Expand Down
86 changes: 1 addition & 85 deletions content/modules/ROOT/pages/module-06.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -76,55 +76,7 @@ Stats about etcd "apply request took too long" messages: etcd-ocp4-2nvq7-master-
Count: 3614
Expected: 200ms
Stats about etcd "apply request took too long" messages: etcd-ocp4-2nvq7-master-1
First Occurrence: 2024-07-28T05:16:07
Last Occurrence: 2024-08-14T15:18:23
Maximum: 2303.5215ms
Minimum: 200.0820ms
Median: 593.4356ms
Average: 631.4544ms
Count: 4315
Expected: 200ms
Stats about etcd "apply request took too long" messages: etcd-ocp4-2nvq7-master-2
First Occurrence: 2024-07-28T03:56:33
Last Occurrence: 2024-08-14T15:18:23
Maximum: 3432.5113ms
Minimum: 200.0972ms
Median: 597.9008ms
Average: 643.1600ms
Count: 4559
Expected: 200ms
Stats about etcd "slow fdatasync" messages: etcd-ocp4-2nvq7-master-0
First Occurrence: 2024-07-28T05:54:22
Last Occurrence: 2024-08-14T15:12:30
Maximum: 1151.2715ms
Minimum: 1000.0142ms
Median: 1058.2370ms
Average: 1045.1226ms
Count: 60
Expected: 1s
Stats about etcd "slow fdatasync" messages: etcd-ocp4-2nvq7-master-1
First Occurrence: 2024-07-30T00:20:55
Last Occurrence: 2024-08-14T15:12:30
Maximum: 1151.6210ms
Minimum: 1001.7959ms
Median: 1055.1022ms
Average: 1035.2390ms
Count: 34
Expected: 1s
Stats about etcd "slow fdatasync" messages: etcd-ocp4-2nvq7-master-2
First Occurrence: 2024-07-28T04:02:07
Last Occurrence: 2024-08-14T15:12:30
Maximum: 1151.4257ms
Minimum: 1000.1440ms
Median: 1053.3669ms
Average: 1042.6280ms
Count: 59
Expected: 1s
etc..
----
====

Expand All @@ -147,20 +99,6 @@ etcd-ocp4-2nvq7-master-0 slow fdatasync
etcd-ocp4-2nvq7-master-0 apply request took too long 3614
etcd-ocp4-2nvq7-master-0 leader is overloaded likely from slow disk 500
etcd-ocp4-2nvq7-master-0 elected leader 9
etcd-ocp4-2nvq7-master-0 lost leader 8
etcd-ocp4-2nvq7-master-1 waiting for ReadIndex response took too long, retrying 320
etcd-ocp4-2nvq7-master-1 slow fdatasync 34
etcd-ocp4-2nvq7-master-1 apply request took too long 4315
etcd-ocp4-2nvq7-master-1 leader is overloaded likely from slow disk 28
etcd-ocp4-2nvq7-master-1 elected leader 7
etcd-ocp4-2nvq7-master-1 lost leader 6
etcd-ocp4-2nvq7-master-2 waiting for ReadIndex response took too long, retrying 385
etcd-ocp4-2nvq7-master-2 slow fdatasync 59
etcd-ocp4-2nvq7-master-2 apply request took too long 4559
etcd-ocp4-2nvq7-master-2 leader is overloaded likely from slow disk 98
etcd-ocp4-2nvq7-master-2 elected leader 9
etcd-ocp4-2nvq7-master-2 lost leader 8
etcd-ocp4-2nvq7-master-2 sending buffer is full 1922
----
====

Expand Down Expand Up @@ -189,28 +127,18 @@ $ etcd-ocp-diag.py --path module6-must-gather.6521552859184261155/ --ttl
POD DATE COUNT
etcd-ocp4-2nvq7-master-0 2024-07-28 121
etcd-ocp4-2nvq7-master-0 2024-07-29 112
etcd-ocp4-2nvq7-master-0 2024-07-30 133
etcd-ocp4-2nvq7-master-0 2024-07-31 202
etcd-ocp4-2nvq7-master-0 2024-08-01 102
...
etcd-ocp4-2nvq7-master-0 2024-08-13 550
etcd-ocp4-2nvq7-master-0 2024-08-14 702
etcd-ocp4-2nvq7-master-1 2024-07-28 60
etcd-ocp4-2nvq7-master-1 2024-07-29 83
etcd-ocp4-2nvq7-master-1 2024-07-30 114
...
etcd-ocp4-2nvq7-master-1 2024-08-12 579
etcd-ocp4-2nvq7-master-1 2024-08-13 805
etcd-ocp4-2nvq7-master-1 2024-08-14 887
etcd-ocp4-2nvq7-master-2 2024-07-28 98
etcd-ocp4-2nvq7-master-2 2024-07-29 58
etcd-ocp4-2nvq7-master-2 2024-07-30 152
etcd-ocp4-2nvq7-master-2 2024-07-31 144
etcd-ocp4-2nvq7-master-2 2024-08-01 63
...
etcd-ocp4-2nvq7-master-2 2024-08-12 627
etcd-ocp4-2nvq7-master-2 2024-08-13 744
etcd-ocp4-2nvq7-master-2 2024-08-14 952
----
====
Expand Down Expand Up @@ -270,17 +198,5 @@ POD COUNT
etcd-ocp4-2nvq7-master-0 14
etcd-ocp4-2nvq7-master-1 12
etcd-ocp4-2nvq7-master-2 15
Date: 05:31
POD COUNT
etcd-ocp4-2nvq7-master-0 15
etcd-ocp4-2nvq7-master-1 13
etcd-ocp4-2nvq7-master-2 11
Date: 05:54
POD COUNT
etcd-ocp4-2nvq7-master-0 7
etcd-ocp4-2nvq7-master-1 7
etcd-ocp4-2nvq7-master-2 14
----
====
10 changes: 5 additions & 5 deletions content/modules/ROOT/pages/module-10.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,15 @@ In this module we will utilize the `koff` tool to showcase get and inspect commn
====
[source,bash]
----
$ koff version
koff version: v1.0.1
hash: 01cddf8
https://github.com/gmeghnag/koff
cd ~/Module10/
----
[source,bash]
----
cd ~/Module10/
$ koff version
koff version: v1.0.1
hash: 01cddf8
https://github.com/gmeghnag/koff
----
[source,bash]
Expand Down

0 comments on commit 8c99567

Please sign in to comment.