Can Sysbox help with `cpu.cfs_quota_us: invalid argument: unknown` error when running kind inside a k8s pod? #497

felipecrs · 2022-03-07T19:00:57Z

felipecrs
Mar 7, 2022

Hi,

The goal of this discussion is to evaluate if this is something Sysbox could help with, or not. I'm not requesting anything. For now, this is simply a question. :)

Currently, to be able to use a kind cluster within a k8s pod (with or without Sysbox as runtime), I need to mount /sys/fs/cgroup from host (which by itself is a potential risk for resources leaking).

EDIT: you only run into this problem if you try to deploy a resource constrained pod (cpu requests/limit) in the underlying kind cluster. Otherwise, all seems to work fine without needing to mount /sys/fs/cgroup.

I described this at kubernetes-sigs/kind#303 (comment), especially when I say:

The exact same thing also happens if I don't mount /sys/fs/cgroup (and do not use --cgroup-parent)

So, is this something Sysbox can help with? Kind currently recommends mounting this directory to dind pods (see kubernetes-sigs/kind#303).

Would be very nice if Sysbox could help to alleviate this requirement, and therefore, make running kind inside of a k8s dind pod more secure.

Answered by ctalledo

Mar 9, 2022

Hi @felipecrs, thanks for opening the issue.

EDIT: you only run into this problem if you try to deploy a resource constrained pod (cpu requests/limit) in the underlying kind cluster.

Can you provide a sample pod spec that would trigger this?

If we are doing things right in Sysbox, running kind inside a (rootless) Sysbox pod should work, regardless of whether you set CPU requests/limits on the pod, as long as you set them to a value that is large enough to accommodate the values that kind will need.

In general, I don't recommend mounting the host's /sys/fs/cgroup into the pod, as that essentially exposes the entire cgroup hierarchy of the host inside the container, breaking isolation.

View full answer

ctalledo · 2022-03-09T00:47:43Z

ctalledo
Mar 9, 2022
Maintainer

Hi @felipecrs, thanks for opening the issue.

EDIT: you only run into this problem if you try to deploy a resource constrained pod (cpu requests/limit) in the underlying kind cluster.

Can you provide a sample pod spec that would trigger this?

If we are doing things right in Sysbox, running kind inside a (rootless) Sysbox pod should work, regardless of whether you set CPU requests/limits on the pod, as long as you set them to a value that is large enough to accommodate the values that kind will need.

In general, I don't recommend mounting the host's /sys/fs/cgroup into the pod, as that essentially exposes the entire cgroup hierarchy of the host inside the container, breaking isolation.

9 replies

felipecrs Mar 10, 2022
Author

@ctalledo I was able to replicate the issue. Finally! But it's a bit complicated. This is what I did:

$ multipass launch bionic --cpus 8 --mem 16G --disk 24G --name primary

$ multipass shell

$ sudo apt update && sudo apt upgrade -y && sudo apt install -y --install-recommends linux-generic-hwe-18.04

$ sudo reboot

$ multipass shell

$ sudo apt update && sudo apt install -y ca-certificates curl gnupg lsb-release

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

$ echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

$ sudo mkdir -p /etc/docker

$ echo '{ "exec-opts": ["native.cgroupdriver=systemd"] }' | sudo tee /etc/docker/daemon.json

$ printf '%s\n' 'net/bridge/bridge-nf-call-ip6tables=1' \
  'net/bridge/bridge-nf-call-iptables=1' \
  'net/bridge/bridge-nf-call-arptables=1' | sudo tee -a /etc/ufw/sysctl.conf

$ sudo apt update && sudo apt install -y ebtables ethtool

$ sudo reboot

$ multipass shell

$ sudo apt update && sudo apt install -y docker-ce=5:19.03.15~3-0~ubuntu-bionic docker-ce-cli=5:19.03.15~3-0~ubuntu-bionic containerd.io=1.4.9-1

$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

$ echo 'deb http://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list

$ sudo apt update && sudo apt install -y kubeadm=1.20.10-00 kubelet=1.20.10-00 kubectl=1.20.10-00

$ sudo apt-mark hold docker-ce docker-ce-cli containerd.io kubeadm kubelet kubectl

$ sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --kubernetes-version 1.20.10

$ alias kubectl="KUBECONFIG=/etc/kubernetes/admin.conf sudo -E kubectl"

$ kubectl apply -f https://projectcalico.docs.tigera.io/manifests/calico.yaml

$ kubectl taint nodes --all node-role.kubernetes.io/master-

$ kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: dind
spec:
  containers:
  - name: dind
    image: ghcr.io/felipecrs/jenkins-agent-dind:latest
    imagePullPolicy: Always
    securityContext:
      privileged: true
    workingDir: /home/jenkins/agent
    volumeMounts:
      - mountPath: /home/jenkins/agent
        name: workspace-volume
      # - mountPath: /lib/modules
      #   name: lib-modules
      #   readOnly: true
      # - mountPath: /sys/fs/cgroup
      #   name: sys-fs-cgroup
    resources:
      requests:
        cpu: "1.75"
      limits:
        cpu: "1.75"
    args: [sleep, infinity]
  hostNetwork: false
  automountServiceAccountToken: false
  enableServiceLinks: false
  dnsPolicy: "None"
  dnsConfig:
    nameservers:
    - 1.1.1.1
    - 1.0.0.1
  restartPolicy: Never
  terminationGracePeriodSeconds: 60
  volumes:
    - name: workspace-volume
      emptyDir: {}
    # - name: lib-modules
    #   hostPath:
    #     path: /lib/modules
    #     type: Directory
    # - name: sys-fs-cgroup
    #   hostPath:
    #     path: /sys/fs/cgroup
    #     type: Directory
EOF

$ kubectl exec -it dind -- sudo -u jenkins bash

$ kind create cluster

$ kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
spec:
  containers:
  - name: ubuntu
    image: ubuntu
    imagePullPolicy: Always
    resources:
      requests:
        cpu: "50m"
      limits:
        cpu: "2000m"
    args: [sleep, infinity]
EOF

$ kubectl describe ubuntu

Warning  Failed     10s (x3 over 29s)  kubelet            Error: failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: failed to write "200000": write /sys/fs/cgroup/cpu,cpuacct/kubelet/kubepods/burstable/pod6248d28f-89da-46a2-85bb-d186f9b7d5fc/ubuntu/cpu.cfs_quota_us: invalid argument: unknown

$ kubectl delete pod ubuntu

# Installing sysbox with ghcr.io/nestybox/sysbox-deploy-k8s:issue_406
$ kubectl apply -f - <<'EOF'
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sysbox-label-node
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: sysbox-node-labeler
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: sysbox-label-node-rb
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: sysbox-node-labeler
subjects:
- kind: ServiceAccount
  name: sysbox-label-node
  namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: sysbox-deploy-k8s
  namespace: kube-system
spec:
  selector:
      matchLabels:
        sysbox-install: "yes"
  template:
    metadata:
        labels:
          sysbox-install: "yes"
    spec:
      serviceAccountName: sysbox-label-node
      nodeSelector:
        sysbox-install: "yes"
      containers:
      - name: sysbox-deploy-k8s
        image: ghcr.io/nestybox/sysbox-deploy-k8s:issue_406
        imagePullPolicy: Always
        command: [ "bash", "-c", "/opt/sysbox/scripts/sysbox-deploy-k8s.sh ce install" ]
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          privileged: true
        volumeMounts:
        - name: host-etc
          mountPath: /mnt/host/etc
        - name: host-osrelease
          mountPath: /mnt/host/os-release
        - name: host-dbus
          mountPath: /var/run/dbus
        - name: host-run-systemd
          mountPath: /run/systemd
        - name: host-lib-systemd
          mountPath: /mnt/host/lib/systemd/system
        - name: host-etc-systemd
          mountPath: /mnt/host/etc/systemd/system
        - name: host-lib-sysctl
          mountPath: /mnt/host/lib/sysctl.d
        - name: host-opt-lib-sysctl
          mountPath: /mnt/host/opt/lib/sysctl.d
        - name: host-usr-bin
          mountPath: /mnt/host/usr/bin
        - name: host-opt-bin
          mountPath: /mnt/host/opt/bin
        - name: host-usr-local-bin
          mountPath: /mnt/host/usr/local/bin
        - name: host-opt-local-bin
          mountPath: /mnt/host/opt/local/bin
        - name: host-usr-lib-mod-load
          mountPath: /mnt/host/usr/lib/modules-load.d
        - name: host-opt-lib-mod-load
          mountPath: /mnt/host/opt/lib/modules-load.d
        - name: host-run
          mountPath: /mnt/host/run
        - name: host-var-lib
          mountPath: /mnt/host/var/lib
      volumes:
        - name: host-etc
          hostPath:
            path: /etc
        - name: host-osrelease
          hostPath:
            path: /etc/os-release
        - name: host-dbus
          hostPath:
            path: /var/run/dbus
        - name: host-run-systemd
          hostPath:
            path: /run/systemd
        - name: host-lib-systemd
          hostPath:
            path: /lib/systemd/system
        - name: host-etc-systemd
          hostPath:
            path: /etc/systemd/system
        - name: host-lib-sysctl
          hostPath:
            path: /lib/sysctl.d
        - name: host-opt-lib-sysctl
          hostPath:
            path: /opt/lib/sysctl.d
        - name: host-usr-bin
          hostPath:
            path: /usr/bin/
        - name: host-opt-bin
          hostPath:
            path: /opt/bin/
        - name: host-usr-local-bin
          hostPath:
            path: /usr/local/bin/
        - name: host-opt-local-bin
          hostPath:
            path: /opt/local/bin/
        - name: host-usr-lib-mod-load
          hostPath:
            path: /usr/lib/modules-load.d
        - name: host-opt-lib-mod-load
          hostPath:
            path: /opt/lib/modules-load.d
        - name: host-run
          hostPath:
            path: /run
        - name: host-var-lib
          hostPath:
            path: /var/lib
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate
---
apiVersion: node.k8s.io/v1beta1
kind: RuntimeClass
metadata:
  name: sysbox-runc
handler: sysbox-runc
scheduling:
  nodeSelector:
    sysbox-runtime: running
---
EOF

$ sleep 60

$ kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: dind
  annotations:
    io.kubernetes.cri-o.userns-mode: "auto:size=65536"
spec:
  runtimeClassName: sysbox-runc
  containers:
  - name: dind
    image: ghcr.io/felipecrs/jenkins-agent-dind:latest
    imagePullPolicy: Always
    workingDir: /home/jenkins/agent
    volumeMounts:
      - mountPath: /home/jenkins/agent
        name: workspace-volume
      # - mountPath: /lib/modules
      #   name: lib-modules
      #   readOnly: true
      # - mountPath: /sys/fs/cgroup
      #   name: sys-fs-cgroup
    resources:
      requests:
        memory: "7G"
        ephemeral-storage: "8Gi"
        cpu: "1.75"
      limits:
        memory: "7G"
        ephemeral-storage: "8Gi"
        cpu: "1.75"
    args: [sleep, infinity]
  hostNetwork: false
  automountServiceAccountToken: false
  enableServiceLinks: false
  dnsPolicy: "None"
  dnsConfig:
    nameservers:
    - 1.1.1.1
    - 1.0.0.1
  restartPolicy: Never
  terminationGracePeriodSeconds: 60
  runtimeClass: sysbox-runc
  volumes:
    - name: workspace-volume
      emptyDir: {}
    # - name: lib-modules
    #   hostPath:
    #     path: /lib/modules
    #     type: Directory
    # - name: sys-fs-cgroup
    #   hostPath:
    #     path: /sys/fs/cgroup
    #     type: Directory
EOF

$ kubectl exec -it dind -- sudo -u jenkins bash

$ kind create cluster

$ kubectl apply -f - <<'EOF'
kind: Pod
metadata:
  name: ubuntu
spec:
  containers:
  - name: ubuntu
    image: ubuntu
    imagePullPolicy: Always
    resources:
      requests:
        memory: "1G"
        ephemeral-storage: "1Gi"
        cpu: "50m"
      limits:
        memory: "1G"
        ephemeral-storage: "1Gi"
        cpu: "2000m"
    args: [sleep, infinity]
EOF

$ kubectl describe pod ubuntu

# same issue

But if you mount /sys/fs/cgroup, there are no issues. With or without Sysbox.

ctalledo Mar 13, 2022
Maintainer

Thanks @felipecrs for the detailed steps. On a first look I suspect the problem is that the CPU limit on the outer container is actually smaller than the CPU limit on the inner one:

Outer container:

      limits:
        ...
        cpu: "1.75"

Inner container:

      limits:
       ...
        cpu: "2000m"    // Same as "2", which larger than "1.75"

Question: what happens if you make the CPU limit on the inner container to be <= the limit of the outer container?

felipecrs Mar 13, 2022
Author

@ctalledo wow that was a mistake from my side. Increasing the dind pod CPU limit or decreasing the ubuntu inner pod CPU limit indeed makes me unable to reproduce the issue again in my VM.

In my production cluster, I'm pretty sure the issue happens even when setting up proper limits.

But I will try again, just to make sure. Thank you!

felipecrs Mar 14, 2022
Author

I can confirm that this issue no longer occurs when using proper CPU limits even in my production cluster.

@ctalledo thank you so much for the attention, and sorry, this whole discussion was based on a wrong assumption from my side.

ctalledo Mar 14, 2022
Maintainer

Hi @felipecrs, thanks for confirming, and no problem at all, I learned from it and I am sure other users will bump into this too, so having the discussion is very helpful. Thanks!

ctalledo · 2022-03-14T15:46:57Z

ctalledo
Mar 14, 2022
Maintainer

To recap: problem was caused by a misconfiguration: the outer container was assigned less CPU than the inner containers via the Linux cgroups mechanism, causing Linux to block the CPU resource assignment for the inner container.

To avoid this, the outer container must be given more cgroup resources (CPU, mem, etc) than the inner container(s).

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can Sysbox help with `cpu.cfs_quota_us: invalid argument: unknown` error when running kind inside a k8s pod? #497

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 9 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Can Sysbox help with cpu.cfs_quota_us: invalid argument: unknown error when running kind inside a k8s pod? #497

felipecrs Mar 7, 2022

Replies: 2 comments · 9 replies

ctalledo Mar 9, 2022 Maintainer

felipecrs Mar 10, 2022 Author

ctalledo Mar 13, 2022 Maintainer

felipecrs Mar 13, 2022 Author

felipecrs Mar 14, 2022 Author

ctalledo Mar 14, 2022 Maintainer

ctalledo Mar 14, 2022 Maintainer

Can Sysbox help with `cpu.cfs_quota_us: invalid argument: unknown` error when running kind inside a k8s pod? #497

felipecrs
Mar 7, 2022

Replies: 2 comments 9 replies

ctalledo
Mar 9, 2022
Maintainer

felipecrs Mar 10, 2022
Author

ctalledo Mar 13, 2022
Maintainer

felipecrs Mar 13, 2022
Author

felipecrs Mar 14, 2022
Author

ctalledo Mar 14, 2022
Maintainer

ctalledo
Mar 14, 2022
Maintainer