8000 AWS Amazon Linux 2023 ECS optimised: Failed to create a manager: mountpoint for cpu not found · Issue #3658 · google/cadvisor · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

AWS Amazon Linux 2023 ECS optimised: Failed to create a manager: mountpoint for cpu not found #3658

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agre 8000 e to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
joe-alford opened this issue Feb 17, 2025 · 1 comment
Open

AWS Amazon Linux 2023 ECS optimised: Failed to create a manager: mountpoint for cpu not found #3658

joe-alford opened this issue Feb 17, 2025 · 1 comment

Comments

@joe-alford
Copy link
joe-alford commented Feb 17, 2025 •
edited
Loading

Hi,

I am seeing the Failed to create a manager: mountpoint for cpu not found error when trying to run cAdvisor on the ECS Optimised Amazon Linux 2023 images. It seems I'm not the only one, as per this stack overflow answer.

Both of these tasks used the same ECS Task Definition (see below) - the difference is their underlying AMI.

Steps to reproduce

Use the ECS Task Definition below in conjunction with the two different AMIs

Not working

AMI

al2023-ami-ecs-hvm-2023.0.20250206-kernel-6.1-arm64

Logs

17 February 2025 at 11:34 (UTC) | F0217 11:34:25.129145 1 cadvisor.go:137] Failed to create a manager: mountpoint for cpu not found
17 February 2025 at 11:34 (UTC) | I0217 11:34:25.055949 1 cadvisor.go:123] enabled metrics: app,cpu,cpuLoad,cpu_topology,cpuset,disk,diskIO,hugetlb,memory,memory_numa,network,oom_event,percpu,perf_event,process,resctrl,sched,tcp,udp
17 February 2025 at 11:34 (UTC) | I0217 11:34:25.056028 1 storagedriver.go:55] Caching stats in memory for 2m0s

Working

AMI

amzn2-ami-ecs-hvm-2.0.20250206-arm64-ebs

Logs

17 February 2025 at 11:34 (UTC) | I0217 11:34:06.938986 1 cadvisor.go:176] Starting cAdvisor version: v0.49.1-6f3f25ba on port 8080 | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.744105 1 manager.go:324] Recovery completed | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.436660 1 factory.go:103] Registering Raw factory | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.436825 1 manager.go:1196] Started watching for new ooms in manager | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.437294 1 manager.go:319] Starting recovery of all containers | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.402348 1 factory.go:352] Registering Docker factory | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.402397 1 factory.go:221] Registration of the docker container factory successfully | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.390452 1 factory.go:145] Registering containerd factory | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.390472 1 factory.go:221] Registration of the containerd container factory successfully | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.390478 1 factory.go:55] Registering systemd factory | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.390481 1 factory.go:221] Registration of the systemd container factory successfully | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.390564 1 factory.go:219] Registration of the crio container factory failed: Get "http://%2Fvar%2Frun%2Fcrio%2Fcrio.sock/info": dial unix /var/run/crio/crio.sock: connect: no such file or directory | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.391155 1 factory.go:219] Registration of the mesos container factory failed: unable to create mesos agent client: failed to get version | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.386532 1 fs.go:133] Filesystem UUIDs: map[642B-CCAF:/dev/nvme0n1p128 a329cfe5-8de9-49c2-b7ed-570855848d58:/dev/nvme0n1p1] | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.386606 1 fs.go:134] Filesystem partitions: map[/dev:{mountpoint:/dev major:0 minor:53 fsType:tmpfs blockSize:0} /dev/nvme0n1p1:{mountpoint:/var/lib/docker major:259 minor:1 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:50 fsType:tmpfs blockSize:0} /rootfs/dev/shm:{mountpoint:/rootfs/dev/shm major:0 minor:20 fsType:tmpfs blockSize:0} /rootfs/run:{mountpoint:/rootfs/run major:0 minor:22 fsType:tmpfs blockSize:0} /rootfs/sys/fs/cgroup:{mountpoint:/rootfs/sys/fs/cgroup major:0 minor:23 fsType:tmpfs blockSize:0} /rootfs/var/lib/docker/containers/36462b64943980ab0d51dd13c1da2e4adef581ddb1c4eeddc0afd9fb23b01801/mounts/shm:{mountpoint:/rootfs/var/lib/docker/containers/36462b64943980ab0d51dd13c1da2e4adef581ddb1c4eeddc0afd9fb23b01801/mounts/shm major:0 minor:42 fsType:tmpfs blockSize:0} /rootfs/var/lib/docker/containers/83f64f995fc136eec32b2615637af50b9b690807af9e4765402edcee09d9a76e/mounts/shm:{mountpoint:/rootfs/var/lib/docker/containers/83f64f995fc136eec32b2615637af50b9b690807af9e4765402edcee09d9a76e/mounts/shm major:0 minor:50 fsType:tmpfs blockSize:0} /rootfs/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/dev:{mountpoint:/rootfs/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/dev major:0 minor:53 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:22 fsType:tmpfs blockSize:0} /sys/fs/cgroup:{mountpoint:/sys/fs/cgroup major:0 minor:23 fsType:tmpfs blockSize:0} /var/lib/docker/containers/36462b64943980ab0d51dd13c1da2e4adef581ddb1c4eeddc0afd9fb23b01801/mounts/shm:{mountpoint:/var/lib/docker/containers/36462b64943980ab0d51dd13c1da2e4adef581ddb1c4eeddc0afd9fb23b01801/mounts/shm major:0 minor:42 fsType:tmpfs blockSize:0} /var/lib/docker/containers/83f64f995fc136eec32b2615637af50b9b690807af9e4765402edcee09d9a76e/mounts/shm:{mountpoint:/var/lib/docker/containers/83f64f995fc136eec32b2615637af50b9b690807af9e4765402edcee09d9a76e/mounts/shm major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/dev:{mountpoint:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/dev major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/dev/shm:{mountpoint:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42
8000
dcc2b2e08009d8e/merged/dev/shm major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/dev/shm:{mountpoint:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/dev/shm major:0 minor:20 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/run:{mountpoint:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/run major:0 minor:22 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/sys/fs/cgroup:{mountpoint:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/sys/fs/cgroup major:0 minor:23 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/var/lib/docker/containers/36462b64943980ab0d51dd13c1da2e4adef581ddb1c4eeddc0afd9fb23b01801/mounts/shm:{mountpoint:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/var/lib/docker/containers/36462b64943980ab0d51dd13c1da2e4adef581ddb1c4eeddc0afd9fb23b01801/mounts/shm major:0 minor:42 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/var/lib/docker/containers/83f64f995fc136eec32b2615637af50b9b690807af9e4765402edcee09d9a76e/mounts/shm:{mountpoint:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/var/lib/docker/containers/83f64f995fc136eec32b2615637af50b9b690807af9e4765402edcee09d9a76e/mounts/shm major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/dev:{mountpoint:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/dev major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/run:{mountpoint:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/run major:0 minor:22 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/sys/fs/cgroup:{mountpoint:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/sys/fs/cgroup major:0 minor:23 fsType:tmpfs blockSize:0} overlay_0-40:{mountpoint:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged major:0 minor:40 fsType:overlay blockSize:0} overlay_0-41:{mountpoint:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/var/lib/docker/overlay2/5ec6056d56b558272bb971961cb8d25e9d0c351e1dcc020dd631c4304a4beb33/merged major:0 minor:41 fsType:overlay blockSize:0}] | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.387421 1 sysinfo.go:242] Found node without cache information, nodeDir: /sys/devices/system/node/node0 | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.387914 1 manager.go:217] Machine: {Timestamp:2025-02-17 11:34:06.387612262 +0000 UTC m=+0.086987290 CPUVendorID: NumCores:2 NumPhysicalCores:2 NumSockets:1 CpuFrequency:0 MemoryCapacity:4121202688 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:2048 NumPages:0}] MachineID:ec2145a3a15f6374b1bb50b19bcd67af SystemUUID:EC2145A3-A15F-6374-B1BB-50B19BCD67AF BootID:693c000d-72f2-4fd4-a97e-800c9937fa94 Filesystems:[{Device:/rootfs/run DeviceMajor:0 DeviceMinor:22 Capacity:2060599296 Type:vfs Inodes:503076 HasInodes:true} {Device:overlay_0-41 DeviceMajor:0 DeviceMinor:41 Capacity:32190214144 Type:vfs Inodes:15722944 HasInodes:true} {Device:/rootfs/var/lib/docker/containers/83f64f995fc136eec32b2615637af50b9b690807af9e4765402edcee09d9a76e/mounts/shm DeviceMajor:0 DeviceMinor:50 Capacity:67108864 Type:vfs Inodes:503076 HasInodes:true} {Device:/sys/fs/cgroup DeviceMajor:0 DeviceMinor:23 Capacity:32190214144 Type:vfs Inodes:15722944 HasInodes:true} {Device:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/var/lib/docker/containers/36462b64943980ab0d51dd13c1da2e4adef581ddb1c4eeddc0afd9fb23b01801/mounts/shm DeviceMajor:0 DeviceMinor:42 Capacity:67108864 Type:vfs Inodes:503076 HasInodes:true} {Device:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/sys/fs/cgroup DeviceMajor:0 DeviceMinor:23 Capacity:32190214144 Type:vfs Inodes:15722944 HasInodes:true} {Device:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/run DeviceMajor:0 DeviceMinor:22 Capacity:2060599296 Type:vfs Inodes:503076 HasInodes:true} {Device:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/dev/shm DeviceMajor:0 DeviceMinor:50 Capacity:67108864 Type:vfs Inodes:503076 HasInodes:true} {Device:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/dev DeviceMajor:0 DeviceMinor:53 Capacity:67108864 Type:vfs Inodes:503076 HasInodes:true} {Device:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/dev DeviceMajor:0 DeviceMinor:53 Capacity:67108864 Type:vfs Inodes:503076 HasInodes:true} {Device:/dev/nvme0n1p1 DeviceMajor:259 DeviceMinor:1 Capacity:32190214144 Type:vfs Inodes:15722944 HasInodes:true} {Device:/rootfs/dev/shm DeviceMajor:0 DeviceMinor:20 Capacity:2060599296 Type:vfs Inodes:503076 HasInodes:true} {Device:/rootfs/sys/fs/cgroup DeviceMajor:0 DeviceMinor:23 Capacity:2060599296 Type:vfs Inodes:503076 HasInodes:true} {Device:/rootfs/var/lib/docker/containers/36462b64943980ab0d51dd13c1da2e4adef581ddb1c4eeddc0afd9fb23b01801/mounts/shm DeviceMajor:0 DeviceMinor:42 Capacity:67108864 Type:vfs Inodes:503076 HasInodes:true} {Device:/rootfs/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/dev DeviceMajor:0 DeviceMinor:53 Capacity:67108864 Type:vfs Inodes:503076 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:50 Capacity:67108864 Type:vfs Inodes:503076 HasInodes:true} {Device:/var/lib/docker/containers/36462b64943980ab0d51dd13c1da2e4adef581ddb1c4eeddc0afd9fb23b01801/mounts/shm DeviceMajor:0 DeviceMinor:42 Capacity:67108864 Type:vfs Inodes:503076 HasInodes:true} {Device:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/sys/fs/cgroup DeviceMajor:0 DeviceMinor:23 Capacity:2060599296 Type:vfs Inodes:503076 HasInodes:true} {Device:/var/lib/docker/containers/83f64f995fc136eec32b2615637af50b9b690807af9e4765402edcee09d9a76e/mounts/shm DeviceMajor:0 DeviceMinor:50 Capacity:67108864 Type:vfs Inodes:503076 HasInodes:true} {Device:overlay_0-40 DeviceMajor:0 DeviceMinor:40 Capacity:32190214144 Type:vfs Inodes:15722944 HasInodes:true} {Device:/dev DeviceMajor:0 DeviceMinor:53 Capacity:67108864 Type:vfs Inodes:503076 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:22 Capacity:2060599296 Type:vfs Inodes:503076 HasInodes:true} {Device:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/dev/shm DeviceMajor:0 DeviceMinor:20 Capacity:2060599296 Type:vfs Inodes:503076 HasInodes:true} {Device:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/run DeviceMajor:0 DeviceMinor:22 Capacity:2060599296 Type:vfs Inodes:503076 HasInodes:true} {Device:/var/lib/docker/overlay2/27701463d7a297e2e9c94ee11c85ac11de5f927de5f04aa42dcc2b2e08009d8e/merged/rootfs/var/lib/docker/containers/83f64f995fc136eec32b2615637af50b9b690807af9e4765402edcee09d9a76e/mounts/shm DeviceMajor:0 DeviceMinor:50 Capacity:67108864 Type:vfs Inodes:503076 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:32212254720 Scheduler:none}] NetworkDevices:[{Name:eth0 MacAddress:0a:11:95:44:eb:29 Speed:0 Mtu:9001}] Topology:[{Id:0 Memory:4121202688 HugePages:[{PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0] Caches:[] UncoreCaches:[] SocketID:60} {Id:1 Threads:[1] Caches:[] UncoreCaches:[] SocketID:60}] Caches:[] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.388075 1 manager.go:233] Version: {KernelVersion:4.14.355-275.582.amzn2.aarch64 ContainerOsVersion:Alpine Linux v3.18 DockerVersion: DockerAPIVersion: CadvisorVersion:v0.49.1 CadvisorRevision:6f3f25ba} | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.388907 1 factory.go:219] Registration of the podman container factory failed: failed to validate Podman info: response not present: Get "http://d/v1.0.0/info": dial unix /var/run/podman/podman.sock: connect: no such file or directory | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.346129 1 cadvisor.go:123] enabled metrics: app,cpu,cpuLoad,cpu_topology,cpuset,disk,diskIO,hugetlb,memory,memory_numa,network,oom_event,percpu,perf_event,process,resctrl,sched,tcp,udp | onvp-cadvisor
17 February 2025 at 11:34 (UTC) | I0217 11:34:06.346196 1 storagedriver.go:55] Caching stats in memory for 2m0s

ECS Task Definition (same between working/not working)

{
    "family": "onvp-cadvisor",
    "containerDefinitions": [
        {
            "name": "onvp-cadvisor",
            "image": "gcr.io/cadvisor/cadvisor:v0.49.1",
            "cpu": 0,
            "portMappings": [
                {
                    "containerPort": 8080,
                    "hostPort": 0,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "command": [
                "--docker_only=false",
                "--disable_metrics=advtcp,referenced_memory",
                "--housekeeping_interval=10s"
            ],
            "environment": [],
            "mountPoints": [
                {
                    "sourceVolume": "root",
                    "containerPath": "/rootfs",
                    "readOnly": true
                },
                {
                    "sourceVolume": "var_run",
                    "containerPath": "/var/run",
                    "readOnly": false
                },
                {
                    "sourceVolume": "sys",
                    "containerPath": "/sys",
                    "readOnly": true
                },
                {
                    "sourceVolume": "var_lib_docker",
                    "containerPath": "/var/lib/docker",
                    "readOnly": true
                },
                {
                    "sourceVolume": "cgroup",
                    "containerPath": "/sys/fs/cgroup",
                    "readOnly": true
                },
                {
                    "sourceVolume": "dev_disk",
                    "containerPath": "/dev/disk",
                    "readOnly": true
                }
            ],
            "volumesFrom": [],
            "user": "root",
            "privileged": true,
            "readonlyRootFilesystem": false,
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "/ecs/onvp-cadvisor",
                    "awslogs-create-group": "true",
                    "awslogs-region": "eu-west-1",
                    "awslogs-stream-prefix": "ecs-prometheus"
                }
            },
            "systemControls": []
        }
    ],
    "tags": [
        {
            "key": "Managed_by_Terraform",
            "value": "true"
        }
    ],
    "taskRoleArn": "arn:aws:iam::882131928231:role/onvp-cadvisorECSTaskRole",
    "executionRoleArn": "arn:aws:iam::882131928231:role/onvp-cadvisorECSExecutionRole",
    "networkMode": "bridge",
    "volumes": [
        {
            "name": "root",
            "host": {
                "sourcePath": "/"
            }
        },
        {
            "name": "var_run",
            "host": {
                "sourcePath": "/var/run"
            }
        },
        {
            "name": "cgroup",
            "host": {
                "sourcePath": "/cgroup"
            }
        },
        {
            "name": "dev_disk",
            "host": {
                "sourcePath": "/dev/disk"
            }
        },
        {
            "name": "sys",
            "host": {
                "sourcePath": "/sys"
            }
        },
        {
            "name": "var_lib_docker",
            "host": {
                "sourcePath": "/var/lib/docker"
            }
        }
    ],
    "placementConstraints": [],
    "requiresCompatibilities": [
        "EC2"
    ],
    "cpu": "256",
    "memory": "512"
}

An odd thing to note is that, if I run the container manually, it does start, using the same switches as in the ECS Task Definition even on the Amazon Linux 2023 image, which doesn't work via the ECS Task:

AMI ID: ami-0e905c785044a59f2
AMI name: al2023-ami-ecs-hvm-2023.0.20250206-kernel-6.1-arm64

[ssm-user@ip-172-20-110-20 bin]$ curl -s http://169.254.169.254/latest/meta-data/ami-id
ami-0e905c785044a59f2
[ssm-user@ip-172-20-110-20 bin]$
[ssm-user@ip-172-20-110-20 bin]$ sudo docker run --rm -it --privileged -v /var/lib/docker:/var/lib/docker -v /:/rootfs -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run:/var/run -v /sys:/sys -v /dev/disk:/dev/disk  gcr.io/cadvisor/cadvisor:v0.49.1 --docker_only=false --housekeeping_interval=10s --disable_metrics=referenced_memory
I0217 11:58:39.465775       1 cadvisor.go:123] enabled metrics: advtcp,app,cpu,cpuLoad,cpu_topology,cpuset,disk,diskIO,hugetlb,memory,memory_numa,network,oom_event,percpu,perf_event,process,resctrl,sched,tcp,udp
I0217 11:58:39.465855       1 storagedriver.go:55] Caching stats in memory for 2m0s
I0217 11:58:39.504303       1 fs.go:133] Filesystem UUIDs: map[39AB-F473:/dev/nvme0n1p128 ad9a82d4-ec9a-4eed-8ae3-914a2cbe86a5:/dev/nvme0n1p1]
I0217 11:58:39.504351       1 fs.go:134] Filesystem partitions: map[/dev:{mountpoint:/dev major:0 minor:56 fsType:tmpfs blockSize:0} /dev/nvme0n1p1:{mountpoint:/var/lib/docker major:259 minor:1 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:58 fsType:tmpfs blockSize:0} /rootfs/dev/shm:{mountpoint:/rootfs/dev/shm major:0 minor:23 fsType:tmpfs blockSize:0} /rootfs/run:{mountpoint:/rootfs/run major:0 minor:25 fsType:tmpfs blockSize:0} /rootfs/run/user/0:{mountpoint:/rootfs/run/user/0 major:0 minor:47 fsType:tmpfs blockSize:0} /rootfs/tmp:{mountpoint:/rootfs/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /rootfs/var/lib/docker/containers/7b2dff231835caaf45631d9ad79fe420ecfbe52c1cec889115f43c25373fbb72/mounts/shm:{mountpoint:/rootfs/var/lib/docker/containers/7b2dff231835caaf45631d9ad79fe420ecfbe52c1cec889115f43c25373fbb72/mounts/shm major:0 minor:63 fsType:tmpfs blockSize:0} /rootfs/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/dev:{mountpoint:/rootfs/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/dev major:0 minor:56 fsType:tmpfs blockSize:0} /rootfs/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/dev/shm:{mountpoint:/rootfs/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/dev/shm major:0 minor:58 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:25 fsType:tmpfs blockSize:0} /run/user/0:{mountpoint:/run/user/0 major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/docker/containers/7b2dff231835caaf45631d9ad79fe420ecfbe52c1cec889115f43c25373fbb72/mounts/shm:{mountpoint:/var/lib/docker/containers/7b2dff231835caaf45631d9ad79fe420ecfbe52c1cec889115f43c25373fbb72/mounts/shm major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/dev:{mountpoint:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/dev major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/dev/shm:{mountpoint:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/dev/shm major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/dev/shm:{mountpoint:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/dev/shm major:0 minor:23 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/run:{mountpoint:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/run major:0 minor:25 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/run/user/0:{mountpoint:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/run/user/0 major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/tmp:{mountpoint:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/var/lib/docker/containers/7b2dff231835caaf45631d9ad79fe420ecfbe52c1cec889115f43c25373fbb72/mounts/shm:{mountpoint:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/var/lib/docker/containers/7b2dff231835caaf45631d9ad79fe420ecfbe52c1cec889115f43c25373fbb72/mounts/shm major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/dev:{mountpoint:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/dev major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/dev/shm:{mountpoint:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/dev/shm major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/run:{mountpoint:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/run major:0 minor:25 fsType:tmpfs blockSize:0} /var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/run/user/0:{mountpoint:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/run/user/0 major:0 minor:47 fsType:tmpfs blockSize:0} overlay_0-49:{mountpoint:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-61:{mountpoint:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/var/lib/docker/overlay2/9ad19e14907f46a02f2761d17bf2cf91e6acaa7530a3aab19970838af46b514b/merged major:0 minor:61 fsType:overlay blockSize:0}]
I0217 11:58:39.506806       1 sysinfo.go:242] Found node without cache information, nodeDir: /sys/devices/system/node/node0
I0217 11:58:39.508057       1 manager.go:217] Machine: {Timestamp:2025-02-17 11:58:39.507313807 +0000 UTC m=+0.074889151 CPUVendorID: NumCores:2 NumPhysicalCores:2 NumSockets:1 CpuFrequency:0 MemoryCapacity:4022366208 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0} {PageSize:32768 NumPages:0} {PageSize:64 NumPages:0}] MachineID:ec23f4c096ba3178b891d89fc8c3677f SystemUUID:ec23f4c0-96ba-3178-b891-d89fc8c3677f BootID:d8cdc9bc-684c-4a6b-b534-86198680214e Filesystems:[{Device:/var/lib/docker/containers/7b2dff231835caaf45631d9ad79fe420ecfbe52c1cec889115f43c25373fbb72/mounts/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:491011 HasInodes:true} {Device:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/dev/shm DeviceMajor:0 DeviceMinor:23 Capacity:2011181056 Type:vfs Inodes:491011 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:32133591040 Type:vfs Inodes:15722944 HasInodes:true} {Device:/dev DeviceMajor:0 DeviceMinor:56 Capacity:67108864 Type:vfs Inodes:491011 HasInodes:true} {Device:/rootfs/run/user/0 DeviceMajor:0 DeviceMinor:47 Capacity:402235392 Type:vfs Inodes:98202 HasInodes:true} {Device:/rootfs/var/lib/docker/containers/7b2dff231835caaf45631d9ad79fe420ecfbe52c1cec889115f43c25373fbb72/mounts/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:491011 HasInodes:true} {Device:/rootfs/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/dev DeviceMajor:0 DeviceMinor:56 Capacity:67108864 Type:vfs Inodes:491011 HasInodes:true} {Device:/rootfs/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/dev/shm DeviceMajor:0 DeviceMinor:58 Capacity:67108864 Type:vfs Inodes:491011 HasInodes:true} {Device:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/run DeviceMajor:0 DeviceMinor:25 Capacity:804474880 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/run/user/0 DeviceMajor:0 DeviceMinor:47 Capacity:402235392 Type:vfs Inodes:98202 HasInodes:true} {Device:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/dev/shm DeviceMajor:0 DeviceMinor:58 Capacity:67108864 Type:vfs Inodes:491011 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:58 Capacity:67108864 Type:vfs Inodes:491011 HasInodes:true} {Device:/dev/nvme0n1p1 DeviceMajor:259 DeviceMinor:1 Capacity:32133591040 Type:vfs Inodes:15722944 HasInodes:true} {Device:/rootfs/run DeviceMajor:0 DeviceMinor:25 Capacity:804474880 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-61 DeviceMajor:0 DeviceMinor:61 Capacity:32133591040 Type:vfs Inodes:15722944 HasInodes:true} {Device:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/dev DeviceMajor:0 DeviceMinor:56 Capacity:67108864 Type:vfs Inodes:491011 HasInodes:true} {Device:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/var/lib/docker/containers/7b2dff231835caaf45631d9ad79fe420ecfbe52c1cec889115f43c25373fbb72/mounts/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:491011 HasInodes:true} {Device:/rootfs/dev/shm DeviceMajor:0 DeviceMinor:23 Capacity:2011181056 Type:vfs Inodes:491011 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:25 Capacity:804474880 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/user/0 DeviceMajor:0 DeviceMinor:47 Capacity:402235392 Type:vfs Inodes:98202 HasInodes:true} {Device:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/run/user/0 DeviceMajor:0 DeviceMinor:47 Capacity:402235392 Type:vfs Inodes:98202 HasInodes:true} {Device:/rootfs/tmp DeviceMajor:0 DeviceMinor:32 Capacity:2011185152 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/dev/shm DeviceMajor:0 DeviceMinor:58 Capacity:67108864 Type:vfs Inodes:491011 HasInodes:true} {Device:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/run DeviceMajor:0 DeviceMinor:25 Capacity:804474880 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/tmp DeviceMajor:0 DeviceMinor:32 Capacity:2011185152 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/rootfs/var/lib/docker/overlay2/d68cbd69124033ed72aabfa927bbab6e3e2c3ffbee54ff110870ac1fff6a11c6/merged/dev DeviceMajor:0 DeviceMinor:56 Capacity:67108864 Type:vfs Inodes:491011 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:32212254720 Scheduler:none}] NetworkDevices:[{Name:ens5 MacAddress:0a:ac:d1:a5:3e:c9 Speed:0 Mtu:9001}] Topology:[{Id:0 Memory:4022366208 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0} {PageSize:32768 NumPages:0} {PageSize:64 NumPages:0}] Cores:[{Id:0 Threads:[0] Caches:[] UncoreCaches:[] SocketID:60} {Id:1 Threads:[1] Caches:[] UncoreCaches:[] SocketID:60}] Caches:[] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
I0217 11:58:39.508443       1 manager.go:233] Version: {KernelVersion:6.1.127-135.201.amzn2023.aarch64 ContainerOsVersion:Alpine Linux v3.18 DockerVersion: DockerAPIVersion: CadvisorVersion:v0.49.1 CadvisorRevision:6f3f25ba}
I0217 11:58:39.508642       1 factory.go:55] Registering systemd factory
I0217 11:58:39.508723       1 factory.go:221] Registration of the systemd container factory successfully
I0217 11:58:39.508965       1 factory.go:219] Registration of the crio container factory failed: Get "http://%2Fvar%2Frun%2Fcrio%2Fcrio.sock/info": dial unix /var/run/crio/crio.sock: connect: no such file or directory
I0217 11:58:39.509946       1 factory.go:219] Registration of the mesos container factory failed: unable to create mesos agent client: failed to get version
I0217 11:58:39.523367       1 factory.go:352] Registering Docker factory
I0217 11:58:39.523403       1 factory.go:221] Registration of the docker container factory successfully
I0217 11:58:39.523792       1 factory.go:219] Registration of the podman container factory failed: failed to validate Podman info: response not present: Get "http://d/v1.0.0/info": dial unix /var/run/podman/podman.sock: connect: no such file or directory
I0217 11:58:39.525123       1 factory.go:145] Registering containerd factory
I0217 11:58:39.525145       1 factory.go:221] Registration of the containerd container factory successfully
I0217 11:58:39.525177       1 factory.go:103] Registering Raw factory
I0217 11:58:39.525192       1 manager.go:1196] Started watching for new ooms in manager
I0217 11:58:39.525876       1 manager.go:319] Starting recovery of all containers
I0217 11:58:39.555379       1 manager.go:324] Recovery completed
I0217 11:58:39.563002       1 cadvisor.go:176] Starting cAdvisor version: v0.49.1-6f3f25ba on port 8080
@maichouni-mitek
Copy link

It's the cgroup v2.
As a workaround, you can build a custom AL2023 ECS image to make docker use cgroup v1.
One way to do it is by modifying /etc/default/grub: in GRUB_CMDLINE_LINUX_DEFAULT=, append systemd.unified_cgroup_hierarchy=0.
Then run grub2-mkconfig -o /boot/grub2/grub.cfg
After a reboot, if you do docker info you will see Cgroup Version: 1.
To be clear, this workaround does not revert to cgroup v1, but instead both versions are used, and Docker automatically picks up v1 which solves the issue you encounter.
I have been using this for more than a year while waiting for cAdvisor to fix the issue.

@liuchisanren liuchisanren mentioned this issue May 8, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@joe-alford @maichouni-mitek
0