Runc create failed: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed

Hello to everyone.

I’ve just installed ubuntu 22.04 on my jetson nano. Docker does not work on 22.04,but it works on ubuntu 18.04 and 20.04. I used the same kernel version for ubuntu 18,20 and 22. This :

Linux marietto-nano 4.9.299+ #0 SMP PREEMPT Wed Mar 29 14:22:17 CEST 2023 aarch64 aarch64 aarch64 GNU/Linux

so it’s not its fault if it does not work. I suppose there is some incompatibility between some component present only on ubuntu 22.04 and not on ubuntu 18 and 20. The error is the following :

# docker images

REPOSITORY TAG IMAGE ID CREATED SIZE
hello-world latest 46331d942d63 13 months ago 9.14kB

# docker run hello-world

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented: unknown.
ERRO[0004] error waiting for container: context canceled

I don’t know what to do. I even tried to upgrade the docker container files following this guide :

https://www.server-world.info/en/note?os=Ubuntu_22.04&p=nvidia&f=2

so,this is what I did :

# curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | apt-key add -
OK

# curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu22.04/nvidia-docker.list > /etc/apt/sources.list.d/nvidia-docker.list

# apt upgrade

Before the upgrade I had these versions :

nvidia-docker2/stable,now 2.8.0-1 all
nvidia-container-toolkit/stable,now 1.7.0-1 arm64

after :

nvidia-docker2/bionic 2.13.0-1 all
nvidia-container-toolkit/bionic 1.13.1-1 arm64
nvidia-container-toolkit-base/bionic 1.13.1-1 arm64

they have been upgraded,but I still see that those packages come from bionic,but I’ve used the repos of jammy

# curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu22.04/nvidia-docker.list > /etc/apt/sources.list.d/nvidia-docker.list

This is the content of the file /etc/docker/daemon.json :

{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

I’m using the JetPack 4.6.3 / L4T 32.7.3. runc version is :

# runc --version

runc version 1.1.4-0ubuntu1~22.04.1
spec: 1.0.2-dev
go: go1.18.1
libseccomp: 2.5.3

docker version is :

# docker --version
Docker version 20.10.21, build 20.10.21-0ubuntu1~22.04.3

And anyway,even with the packages upgraded,the error hasn’t been fixed.

As a further experiment,I have purged all the packages installed for ubuntu 22.04 and I have installed the versions of the same packages which works for ubuntu 20.04. They are called like this :

cgroup-tools_0.41-10_arm64.deb
docker.io_20.10.21-0ubuntu1~20.04.1_arm64.deb
containerd_1.6.12-0ubuntu1~20.04.1_arm64.deb
runc_1.1.4-0ubuntu1~20.04.1_arm64.deb

but,I’ve got the same exact error as before :

root@marietto-nano:/home/marietto# docker run hello-world

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented: unknown.
ERRO[0000] error waiting for container: context canceled

On the github two developers,after having checked the output of the “check-config.sh” script :

root@marietto-nano:/home/marietto/Scaricati# ./check-config.sh

info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: cgroupv2
  Controllers:
  **- cpu: missing
  - cpuset: missing**
  - io: available
  - memory: available
  - pids: available
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MARK: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CFQ_GROUP_IOSCHED: missing
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_SECURITY_SELINUX: missing
- CONFIG_SECURITY_APPARMOR: missing
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled
    - CONFIG_BRIDGE_VLAN_FILTERING: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled
  - "ipvlan":
    - CONFIG_IPVLAN: enabled
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: missing
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: available
    - zpool command: available

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

told me :

It looks like you have only partial support of cgroupv2. Can you try disabling it by setting the kernel command-line argument `systemd.unified_cgroup_hierarchy=0`?

ok. I did that. This is now my extlinux.conf :

TIMEOUT 30
DEFAULT primary

MENU TITLE L4T boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND ${cbootargs} root=PARTUUID=5ac80d7c-40fb-4796-bd56-4110e389819b rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0
      #APPEND ${cbootargs} root=/dev/sda1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0
       APPEND systemd.unified_cgroup_hierarchy=0

LABEL backup
    MENU LABEL backup kernel
    LINUX /boot/Image
    INITRD /boot/initrd
    APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0

But Jetson nano reboots in loop before to really boot. What’s missing ? I think that the latter could be the right thing to do.