Failed to get storage metrics spamming journal log causing higher load on k3s server

[JSN-1]

It started after upgrading from k3s 1.29.x to 1.30.x, problem still persist after upgrading to 1.31.4

my cluster is 1 control plane node and 2 worker nodes.

metrics-server is running fine and kubectl top pods | nodes is working as well.

message is written in journal log 5 - 6 times per minute causing higher load on k3s-server.

Jan 06 19:50:24 <redacted> k3s[335232]: I0106 19:50:24.818644  335232 metrics.go:299] "Failed to get storage metrics" storage_cluster_id="etcd-0" err="context deadline exceeded"
Jan 06 19:50:26 <redacted> k3s[335232]: I0106 19:50:26.654560  335232 metrics.go:299] "Failed to get storage metrics" storage_cluster_id="etcd-0" err="context deadline exceeded"
Jan 06 19:50:31 <redacted> k3s[335232]: I0106 19:50:31.717152  335232 metrics.go:299] "Failed to get storage metrics" storage_cluster_id="etcd-0" err="context deadline exceeded"
Jan 06 19:50:36 <redacted> k3s[335232]: I0106 19:50:36.068181  335232 metrics.go:299] "Failed to get storage metrics" storage_cluster_id="etcd-0" err="context deadline exceeded"

[brandond]

Storage metrics are not node/pod metrics. This has nothing to do with metrics-server. What you're seeing here are failures to collect resource count metrics from the datastore: https://github.com/kubernetes/kubernetes/blob/v1.32.0/staging/src/k8s.io/apiserver/pkg/storage/etcd3/metrics/metrics.go

This is probably a symptom of slowness on your node, not the cause. The apiserver caps time to collect all metrics to one second, if your node is performing poorly and cannot count all resources within one second then you'll get this error.

Figure out why your node is performing poorly and these messages will go away.

[JSN-1]

thanks for the explanation, i will debug further.