Peer already exists error in MetalLb #427

schrodit · 2023-07-17T07:25:14Z

The metallb of our Kubernetes cluster throws that error on startup:

{"branch":"HEAD","caller":"level.go:63","commit":"v0.12.1","goversion":"gc / go1.16.14 / amd64","level":"info","msg":"MetalLB controller starting version 0.12.1 (commit v0.12.1, branch HEAD)","ts":"2023-07-17T07:11:16.64900186Z","version":"0.12.1"}
{"caller":"level.go:63","configmap":"metallb-system/config","error":"peer #25 already exists","event":"configStale","level":"error","msg":"config (re)load failed, config marked stale","ts":"2023-07-17T07:11:24.348221116Z"}
{"caller":"level.go:63","event":"stateSynced","level":"info","msg":"controller synced, can allocate IPs now","ts":"2023-07-17T07:11:24.448924397Z"}

the respective peer config is:

    - hold-time: ""
      my-asn: xxxx
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: shoot--codesphere--fra-worker-67fdf-m76pk
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: default
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: default
      password: ""
      peer-address: 169.254.255.1
      peer-asn: xxxx
      peer-port: 0
      router-id: ""
      source-address: ""

In the equinix metalconsole I also see that we 2 nodes without a learned bgp route (might be related).

Environment:

K8s: 1.22.17
Equinix CCM: v3.6.2
MetalLb: 0.12.1

The text was updated successfully, but these errors were encountered:

ctreatma · 2023-07-17T17:22:52Z

Hi @schrodit! Is this causing your cluster and/or the applications running in it to malfunction in some way, or is the log message the only observable outcome? Did this issue happen after upgrading CPEM in an existing cluster, or is this happening in a new cluster that was set up from the beginning with the software versions listed in the issue description?

schrodit · 2023-07-17T19:23:38Z

Hey @ctreatma ,

Did this issue happen after upgrading CPEM in an existing cluster, or is this happening in a new cluster that was set up from the beginning with the software versions listed in the issue description?

the cpem and metallb were updated from metallb v9.5 and equinix ccm 3.3.0.

Is this causing your cluster and/or the applications running in it to malfunction in some way, or is the log message the only observable outcome?

An issue is that the bgp routes of some nodes are not correctly updated.
So only half of the nodes serve the IP according to the Metal Console. And less and less machines are serving the domain.
I think all nodes should serve the domain.

All speaker pods were no routes are assigned have the saem/similöar error message:

{"caller":"level.go:63","event":"nodeLabelsChanged","level":"info","msg":"Node labels changed, resyncing BGP peers","ts":"2023-07-17T13:45:53.944909421Z"}
{"caller":"level.go:63","configmap":"metallb-system/config","error":"peer #25 already exists","event":"configStale","level":"error","msg":"config (re)load failed, config marked stale","ts":"2023-07-17T13:45:53.948508927Z"}

Non-of them serving the ip.

ctreatma · 2023-07-17T21:01:55Z

Could you provide the config for the other peers as well? Are there 2 identical peer configs in metallb-system/config?

schrodit · 2023-07-18T07:02:46Z

I think its the best if provide the whole configmap:

MetalLb Configmap

apiVersion: v1
data:
  config: |
    address-pools:
    - addresses:
      - 1.1.1.1/32
      auto-assign: false
      avoid-buggy-ips: false
      bgp-advertisements: []
      name: my-ns/prerequisites-ingress-nginx-controller
      protocol: bgp
    - addresses:
      - 1.1.1.2/32
      auto-assign: false
      avoid-buggy-ips: false
      bgp-advertisements: []
      name: my-ns/workspace-reverse-proxy-ingress-nginx-controller
      protocol: bgp
    bgp-communities: {}
    peers:
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
kind: ConfigMap
metadata:
  name: config
  namespace: metallb-system

schrodit · 2023-07-18T07:44:17Z

We now had to completly roll back to equinix ccm v3.3.0 and metallb v0.9.5 as all bgp routes were deleted.

After the rollback everything was fine again

cprivitere · 2023-07-19T20:38:34Z

Hey @schrodit few questions for you:

Do you have the config map for CPEM that you tried using?
Do you have what the metallb config map looked like before the upgrade?
Have you tried letting metallb rebuild the config map for metallb (obviously you'd want to do this in a testing environment to see how hard it is to kick the cluster into rebuilding that config map for you).
In your testing did you try only upgrading metallb first and seeing if that went ok (looking through the dependencies here, I think the upgrade process would be metallb to 0.12.1 and then CPEM to 3.6.2, but it's hard to tell for sure what would work best, perhaps just some stair steps depending on how the config maps look along the way.
Do you already have a planned timeframe for getting to MetalLB 0.13 or later? If so, we may be able to side-step troubleshooting this given the 0.13 versions use a CRD based setup and the work of getting to 0.12.1 won't help with that.

For @ctreatma it looks like the config map they ended up with had 48 entries with 12 unique hostnames, so four entries each.

schrodit · 2023-07-19T22:08:43Z

Hey @cprivitere ,

Do you have the config map for CPEM that you tried using?

we do not use a configmap but rather the env var approach.
Our env vars are:

        env:
        - name: METAL_API_KEY
          valueFrom:
            secretKeyRef:
              key: apiToken
              name: cloudprovider
        - name: METAL_PROJECT_ID
          valueFrom:
            secretKeyRef:
              key: projectID
              name: cloudprovider
        - name: METAL_METRO_NAME
          value: fr
        - name: METAL_LOAD_BALANCER
          value: metallb:///

Do you have what the metallb config map looked like before the upgrade?

yes like it currently does (becasue we had to roll back):

peers:
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      password: ""
    bgp-communities: {}
    address-pools:
    - protocol: bgp
      name: my-ns/prerequisites-ingress-nginx-controller
      addresses:
      - 1.1.1.1/32
      avoid-buggy-ips: false
      auto-assign: false
      bgp-advertisements: []
    - protocol: bgp
      name: my-ns/workspace-reverse-proxy-ingress-nginx-controller
      addresses:
      - 1.1.1.2/32
      avoid-buggy-ips: false
      auto-assign: false
      bgp-advertisements: []peers:
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      password: ""
    bgp-communities: {}
    address-pools:
    - protocol: bgp
      name: my-ns/prerequisites-ingress-nginx-controller
      addresses:
      - 1.1.1.1/32
      avoid-buggy-ips: false
      auto-assign: false
      bgp-advertisements: []
    - protocol: bgp
      name: my-ns/workspace-reverse-proxy-ingress-nginx-controller
      addresses:
      - 1.1.1.2/32
      avoid-buggy-ips: false
      auto-assign: false
      bgp-advertisements: []

Have you tried letting metallb rebuild the config map for metallb (obviously you'd want to do this in a testing environment to see how hard it is to kick the cluster into rebuilding that config map for you).

I think it did that when the new ccm was started. This was because first we used metallb 0.9.5 with ccm 3.6.2 which resulted in an error in the metallb (source-address was unknown). After updating the config was correctly parsed but the described error occurred.

Do you already have a planned timeframe for getting to MetalLB 0.13 or later? If so, we may be able to side-step troubleshooting this given the 0.13 versions use a CRD based setup and the work of getting to 0.12.1 won't help with that.

We are not tight to version 0.12.1. But I guess the CRD approach needs more testing. And the question is if it will work better.
Because I tried to reproduce the issue in a fresh environment, but at least there it worked.

Maybe another sidenote to our setup:
we have two services of type Loadbalancer configured whereas one of them has externalTrafficPolicy: Local.
Not sure if that makes a difference.

Do you have an Idea how we ended up there and how we can upgrade.
If you say that the CRD approach is the safest way we could do that (But currently as I understood the configmap way is the default. so the stbale one?).
Are there any things we need to consider while updating or is it mostly updating metallb and the configuring the ccm as described here

cprivitere · 2023-07-21T16:59:20Z

Because I tried to reproduce the issue in a fresh environment, but at least there it worked.

Are you saying that this works properly in a fresh environment so the issue here has to do with upgrading? We don't have a lot of testing or feedback about the upgrade process.
I'm wondering if you need to upgrade in steps of metallb minor versions until you reach an issue and then cpem minor versions until you hit an issue and then back to metallb.

Is it possible to instead plan to build a fresh cluster and migrate the applications? I feel like a fresh cluster with 3.6.2 and MetalLB 0.13.X would be an easier route.

schrodit · 2023-07-25T11:49:44Z

Are you saying that this works properly in a fresh environment so the issue here has to do with upgrading?

It works in my simple test. But I'm not sure how to reproduce the situation in our prod cluster.

Is it possible to instead plan to build a fresh cluster and migrate the applications?

This is unfortunately not possible. We have customers running on that cluster and can not simple migrate them.

@cprivitere do you have an idea why the ccm generates a invalid metallb config?
You're suggest way foward is to upgrade metallb. then upgrade the equinix ccm. What if there are still issues?

cprivitere · 2023-08-01T21:42:02Z

@schrodit
Still working on reproducing this and figuring out how to best guide you. One thing that did come up in my testing. Between CPEM 3.3.0 and 3.6.2 we changed it from running as a deployment to running as a daemonset. Did you possibly have both the old and new version running at the same time? How did you go about stopping the old deployment and starting the new daemonset?

cprivitere · 2023-08-02T19:31:54Z

Ok we did some attempts to reproduce this and while we can't get the same exact issue you ran into, we did see several issues come up depending on how one tries to upgrade. Here's what we've found to be the best way to upgrade from MetalLB 0.9.5 and CPEM 3.3.0 to MetalLB 0.12.1 and CPEM 3.6.2. The biggest finding was that the Metal LB config map needs to be cleaned up to get rid of older entries that aren't formatted properly, CPEM won't over write these so the only option is to delete them.

This assumes your metallb lives in the metallb-system namespace and the metallb configmap is named "config".

# Remove current Metal LB
kubectl delete -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml
# Remove current CPEM
kubectl delete -f https://github.com/equinix/cloud-provider-equinix-metal/releases/download/v3.3.0/deployment.yaml
# Remove current MetalLB configmap (CPEM will regenerate it)
kubectl -n metallb-system delete cm config
# Install current Metal LB
kubectl  apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/metallb.yaml
# Install current CPEM
kubectl apply -f https://github.com/equinix/cloud-provider-equinix-metal/releases/download/v3.6.2/deployment.yaml

Are you able to test this out and see if the load balancers come up following your upgrade?

ocobles · 2023-08-04T10:04:01Z

Hi @schrodit, complementing @cprivitere ' comment and for other people seeking this information, during the tests, we found that version Metallb 0.9.5 doesn't work with CPEM 3.6.2, so upgrading both is necessary to ensure it works well. We also noticed that it is possible to upgrade them without deleting the configmap. However, we still recommend doing so when possible, as we encountered duplicated fields as in the config you shared when we perform multiple upgrades/rollbacks between different versions where changes were made to some data types and fields, and they were interpreted as different

schrodit · 2023-08-09T07:35:03Z

@cprivitere @ocobles thanks for the input.
Cleaning up the config.yaml seems to be the best option for us.
We still try that out and let you know.

k8s-triage-robot · 2024-01-26T02:25:00Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2024-02-25T02:25:25Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2024-03-26T03:03:33Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen
Mark this issue as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot · 2024-03-26T03:03:37Z

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen

Mark this issue as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

cprivitere · 2024-06-12T17:49:06Z

/reopen
/remove-lifecycle rotten
/triage accepted

k8s-ci-robot · 2024-06-12T17:49:11Z

@cprivitere: Reopened this issue.

In response to this:

/reopen
/remove-lifecycle rotten
/triage accepted

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 26, 2024

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 25, 2024

k8s-ci-robot closed this as not planned Won't fix, can't repro, duplicate, stale Mar 26, 2024

k8s-ci-robot added the triage/accepted Indicates an issue or PR is ready to be actively worked on. label Jun 12, 2024

k8s-ci-robot reopened this Jun 12, 2024

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Jun 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Peer already exists error in MetalLb #427

Peer already exists error in MetalLb #427

schrodit commented Jul 17, 2023

ctreatma commented Jul 17, 2023

schrodit commented Jul 17, 2023 •

edited

Loading

ctreatma commented Jul 17, 2023

schrodit commented Jul 18, 2023 •

edited

Loading

schrodit commented Jul 18, 2023

cprivitere commented Jul 19, 2023 •

edited

Loading

schrodit commented Jul 19, 2023 •

edited

Loading

cprivitere commented Jul 21, 2023 •

edited

Loading

schrodit commented Jul 25, 2023

cprivitere commented Aug 1, 2023

cprivitere commented Aug 2, 2023 •

edited

Loading

ocobles commented Aug 4, 2023

schrodit commented Aug 9, 2023

k8s-triage-robot commented Jan 26, 2024

k8s-triage-robot commented Feb 25, 2024

k8s-triage-robot commented Mar 26, 2024

k8s-ci-robot commented Mar 26, 2024

cprivitere commented Jun 12, 2024

k8s-ci-robot commented Jun 12, 2024

Peer already exists error in MetalLb #427

Peer already exists error in MetalLb #427

Comments

schrodit commented Jul 17, 2023

Environment:

ctreatma commented Jul 17, 2023

schrodit commented Jul 17, 2023 • edited Loading

ctreatma commented Jul 17, 2023

schrodit commented Jul 18, 2023 • edited Loading

schrodit commented Jul 18, 2023

cprivitere commented Jul 19, 2023 • edited Loading

schrodit commented Jul 19, 2023 • edited Loading

cprivitere commented Jul 21, 2023 • edited Loading

schrodit commented Jul 25, 2023

cprivitere commented Aug 1, 2023

cprivitere commented Aug 2, 2023 • edited Loading

ocobles commented Aug 4, 2023

schrodit commented Aug 9, 2023

k8s-triage-robot commented Jan 26, 2024

k8s-triage-robot commented Feb 25, 2024

k8s-triage-robot commented Mar 26, 2024

k8s-ci-robot commented Mar 26, 2024

cprivitere commented Jun 12, 2024

k8s-ci-robot commented Jun 12, 2024

schrodit commented Jul 17, 2023 •

edited

Loading

schrodit commented Jul 18, 2023 •

edited

Loading

cprivitere commented Jul 19, 2023 •

edited

Loading

schrodit commented Jul 19, 2023 •

edited

Loading

cprivitere commented Jul 21, 2023 •

edited

Loading

cprivitere commented Aug 2, 2023 •

edited

Loading