prometheus-kube-stack - ConfigMap dashboard download InvalidCredentials

[m17kea]

Describe the bug a clear and concise description of what the bug is.

I'm trying to add an observability stack to my kubernetes environment which is currently running RabbitMQ from the official RabbitMQ Operator. I have installed the kube-prometheus-stack as follows:

#!/usr/bin/env bash
KUBE_PROMETHEUS_STACK_VERSION='45.7.1'
KUBE_PROMETHEUS_STACK_NAME='prom'
KUBE_PROMETHEUS_STACK_NAMESPACE='kube-prometheus'
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm upgrade "$KUBE_PROMETHEUS_STACK_NAME" prometheus-community/kube-prometheus-stack \
  --version "$KUBE_PROMETHEUS_STACK_VERSION" \
  --install \
  --namespace "$KUBE_PROMETHEUS_STACK_NAMESPACE" \
  --create-namespace \
  --wait \
  --set "defaultRules.create=false" \
  --set "nodeExporter.enabled=false" \
  --set "prometheus.prometheusSpec.ruleSelectorNilUsesHelmValues=false" \
  --set "prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false" \
  --set "prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false" \
  --set "prometheus.prometheusSpec.probeSelectorNilUsesHelmValues=false" \
  --set "alertmanager.alertmanagerSpec.useExistingSecret=true" \
  --set "grafana.env.GF_INSTALL_PLUGINS=flant-statusmap-panel"

I've added the service monitor using the following yaml and can see the metrics in prometheus:

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: rabbitmq
  namespace: kube-prometheus
  # If labels are defined in spec.serviceMonitorSelector.matchLabels of your deployed Prometheus object, make sure to include them here.
  labels:
    release: prom
spec:
  endpoints:
  - port: prometheus
    scheme: http
    interval: 15s
    scrapeTimeout: 14s
  - port: prometheus-tls
    scheme: https
    interval: 15s
    scrapeTimeout: 14s
    tlsConfig:
      insecureSkipVerify: true # set to false and uncomment lines below to enable tls verification
        # ca:
        #   secret:
        #     key: ca.crt
        #     name: tls-secret # name of the secret containing the CA cert which signed the RabbitMQ Prometheus TLS cert
        # serverName: '*.RABBITMQ-INSTANCE-NAME.NAMESPACE.svc.cluster.local'
  - port: prometheus
    scheme: http
    path: /metrics/detailed
    params:
      family:
        - queue_coarse_metrics
        - queue_metrics
    interval: 15s
    scrapeTimeout: 14s
  - port: prometheus-tls
    scheme: https
    path: /metrics/detailed
    params:
      family:
        - queue_coarse_metrics
        - queue_metrics
    interval: 15s
    scrapeTimeout: 14s
    tlsConfig:
      insecureSkipVerify: true
  selector:
    matchLabels:
      app.kubernetes.io/component: rabbitmq
  namespaceSelector:
    any: true

I've attempted to load one of the RabbitMQ community dashboards using the following ConfigMap:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: erlang-distribution-dashboard
  namespace: kube-prometheus
  labels:
    grafana_dashboard: "1"
data:
  # https://grafana.com/grafana/dashboards/11352
  erlang-distribution-dashboard.json.url: https://grafana.com/api/dashboards/11352/revisions/7/download

However the dashboard does not appear and after tailing the grafana container logs using command:

kubectl logs prom-grafana-895847f4f-mbfzw  -n kube-prometheus --tail 1 --follow -c grafana

I saw the following error:

logger=provisioning.dashboard type=file name=sidecarProvider t=2023-03-24T17:26:08.554955848Z level=error msg="failed to load dashboard from " file=/tmp/dashboards/erlang-distribution-dashboard.json error="Dashboard title cannot be empty"

Once I connect to the container using command:

kubectl exec -it prom-grafana-895847f4f-mbfzw  /bin/sh -n kube-prometheus

and open the above file the contents is as follows:

{
  "code": "InvalidCredentials",
  "message": "Invalid password",
  "requestId": "4d1d0ea3-2629-44ec-9fd6-3d172d1bce1b"
}

If I try to download the file manually using wget:

wget https://grafana.com/api/dashboards/11352/revisions/7/download -O erlang-distribution-dashboard.json

it succeeds with correct content but after a short time it is overwritten by the above output again. It doesn't seem to be obvious where that output is coming from so any suggestions would be most welcome.

Thanks in advance.

What's your helm version?

version.BuildInfo{Version:"v3.10.3", GitCommit:"835b7334cfe2e5e27870ab3ed4135f136eecc704", GitTreeState:"clean", GoVersion:"go1.19.4"}

What's your kubectl version?

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff3fb78", GitTreeState:"clean", BuildDate:"2022-11-09T13:36:36Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"darwin/amd64"} Kustomize Version: v4.5.7 Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.8-eks-ffeb93d", GitCommit:"abb98ec0631dfe573ec5eae40dc48fd8f2017424", GitTreeState:"clean", BuildDate:"2022-11-29T18:45:03Z", GoVersion:"go1.18.8", Compiler:"gc", Platform:"linux/amd64"}

Which chart?

kube-prometheus-stack

What's the chart version?

45.7.1

What happened?

As above

What you expected to happen?

The dashboard should have been downloaded and configured correctly

How to reproduce it?

As above

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

As above

Anything else we need to know?

No response

[zeritti]

Dashboards coming from configmaps get collected by the dashboard sidecar which makes them available to Grafana via provisioning. The configmap has to include a full dashboard definition in JSON format underneath .data.KEY with the KEY being some_name.json.

For importing dashboards from various other sources, using grafana.dashboards looks like the way forward (Ref.).

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity.

[Fresa]

The problem is upstream: kiwigrid/k8s-sidecar#290