Helm chart err : prometheus.counter is not prometheus.Gauge: missing method Dec

[dimashenkov]

What happened:
Helm chart pod gets error

panic: interface conversion: *prometheus.counter is not prometheus.Gauge: missing method Dec
goroutine 1 [running]:
github.com/aws/amazon-vpc-cni-k8s/cmd/cni-metrics-helper/metrics.producePrometheusMetrics({0x2cf3990, 0xc00060f0e0}, 0xc000f93a70, 0xc000810d80)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/cmd/cni-metrics-helper/metrics/metrics.go:368 +0x88a

Environment: AWS EKS

• Kubernetes version (use kubectl version): 1.29
• CNI Version 1.19
• OS (e.g: cat /etc/os-release): Amazon Linux

[orsenthil]

@dimashenkov . Do you see this issue with 1.18.6. Want to verify if this change #3093 is causing this regression.

[dimashenkov]

@orsenthil I need to test, i tested with 1.19.0 its broken with 1.18.0 its works , v1.18.6 works fine

[dimashenkov]

@orsenthil @dshehbaj cni-metrics-helper 1.18.6 works fine , interesting that 1.19.0 also works on our other cluster which has Kubernetes version : 1.29
CNI Version 1.11.4

[dshehbaj]

Hi @dimashenkov,

Can you share the helm chart version of CNI Metrics Helper running both on the cluster where you see the error and the cluster where it runs fine?

[dshehbaj]

Hi @dimashenkov,

For both clusters where you see the error and do not see the error, can you share the versions CNI and CNI Metrics Helper that are running?

In CNI version 1.19, we updated the types of some metrics: https://github.com/aws/amazon-vpc-cni-k8s/pull/3093/files#diff-fea6b7a2003b8fe9b0f1f7c48d37848be31bdfffb86a7baaebd50bb645c031d5, and CNI Metrics Helper version 1.19 also imports the same types.

Due to the nature of this change, CNI Metrics Helper version 1.19 is incompatible with CNI versions 1.18.6 and older.

I am able to reproduce the error you reported if I pair CNI Metrics Helper version 1.19 with versions of CNI 1.18.6, or older.

[dimashenkov]

Hi @dshehbaj ,
devops cluster, cni v1.19.0-eksbuild.1 , helper 1.19.0 works
stage cluster , cni v1.16.0-eksbuild.1 , helper 1.19.0 does not work ,1.18.6 works
dev cluster , cni v1.11.4-eksbuild.1 , helper 1.19.0 works
prod cluster cni v1.10.4-eksbuild.1 , helper 1.19.0 works

[gnuletik]

We are also facing the same issue with:

• CNI: amazon-k8s-cni:v1.19.2-eksbuild.1
• Helper: cni-metrics-helper:v1.19.0

Both deployments seems to be up-to-date. Is there anything we can do to workaround this CrashLoopBackoff ?

Here's the helm chart (as a terraform resource), we use to deploy the helper:

resource "helm_release" "cni-metrics-helper" {
  name       = "cni-metrics-helper"
  namespace  = "kube-system"
  repository = "https://aws.github.io/eks-charts"
  chart      = "cni-metrics-helper"
  version    = "1.19.0"

  values = [
    <<-YAML
    env:
      USE_CLOUDWATCH: false
      USE_PROMETHEUS: "true"
      AWS_VPC_K8S_CNI_LOGLEVEL: INFO

    resources:
      requests:
        cpu: 100m
        memory: 256Mi
    YAML
  ]

Thanks!

[orsenthil]

Can you use the cni-metrics-helper with the same version as v1.19.2-eksbuild.1

[gnuletik]

cni-metrics-helper has no v1.19.2 release available on Helm:
https://artifacthub.io/packages/helm/aws/cni-metrics-helper

The CNI version is provided through the EKS add-on.

[orsenthil]

The helm charts are yet to be published. Will be published on Monday. I understand you updated the image directly. The image of cni metrics helper should be available.

[orsenthil]

cni-metrics-helper has no v1.19.2 release available on Helm.

It is available now - https://github.com/aws/eks-charts/releases/tag/v0.0.190
Please try this.

[gnuletik]

It works like a charm!
Thanks :)

[dimashenkov]

yes it works, closing .

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.