Docker Run Error with –storage-opt and Overlay2 on XFS with pquota #46823

rajat709 · 2023-11-17T03:53:55Z

Description

According to the documentation at docker run | Docker Docs, --storage-opt should work with overlay2, and the backing filesystem is XFS mounted with pquota. I’ve been trying for a while, but I keep encountering this error.

docker: Error response from daemon: --storage-opt is supported only for overlay over xfs with 'pquota' mount option.
See 'docker run --help'.

Reproduce

docker run -d it --storage-opt size=10G ubuntu

Expected behavior

No response

docker version

ubuntu@ip-172-31-15-152:~$ docker version
Client:
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.3
 Git commit:        24.0.5-0ubuntu1~22.04.1
 Built:             Mon Aug 21 19:50:14 2023
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.3
  Git commit:       24.0.5-0ubuntu1~22.04.1
  Built:            Mon Aug 21 19:50:14 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.2
  GitCommit:
 runc:
  Version:          1.1.7-0ubuntu1~22.04.1
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:

docker info

ubuntu@ip-172-31-15-152:~$ docker info
Client:
 Version:    24.0.5
 Context:    default
 Debug Mode: false

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version:
 runc version:
 init version:
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.2.0-1012-aws
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.32GiB
 Name: ip-172-31-15-152
 ID: 6b042b0f-e260-4b24-a9d3-1ed982cc6c53
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

The text was updated successfully, but these errors were encountered:

thaJeztah · 2023-11-17T13:13:59Z

Thanks for reporting; from your docker info output, it looks like your system is running extfs, not xfs

Backing Filesystem: extfs

Project quotas are currently only supported on xfs (as the error-message mentions), so this should currently be the expected behavior. There's a feature request for this, and there was a pull request, which got abandoned, so it was not implemented so far;

We should also look if we want to implement this as part of the graph-drivers, because we're transitioning to he containerd image-store and snapshotters, so it would be better to implement there (if not implemented yet)

I'll close this ticket because of the above, but feel free to continue the conversation

rajat709 · 2023-11-17T15:11:31Z

Hi @thaJeztah

I have tried with both file backing systems but did'nt work

you check the conversion here https://forums.docker.com/t/docker-run-error-with-storage-opt-and-overlay2-on-xfs-with-pquota/138489

thaJeztah · 2023-11-17T15:27:31Z

I know there's been some corner-cases around feature-detection for the overlay driver, and there's some work (but unfinished) to improve (see #45890), although I don't know if feature-detection for quotas is part of that /cc @neersighted

Feature detection for the overlay driver is initiated in

moby/daemon/graphdriver/overlay2/overlay.go

Lines 189 to 196 in 75324c4

    
           if backingFs == "xfs" { 
        
           	// Try to enable project quota support over xfs. 
        
           	if d.quotaCtl, err = quota.NewControl(home); err == nil { 
        
           		projectQuotaSupported = true 
        
           	} else if opts.quota.Size > 0 { 
        
           		return nil, fmt.Errorf("Storage option overlay2.size not supported. Filesystem does not support Project Quota: %v", err) 
        
           	} 
        
           } else if opts.quota.Size > 0 {

Which uses code from the quota package to perform the actual detection;

moby/quota/projectquota.go

Lines 107 to 132 in 75324c4

    
           // xfs_quota tool can be used to assign a project id to the driver home directory, e.g.: 
        
           // 
        
           //	echo 999:/var/lib/docker/overlay2 >> /etc/projects 
        
           //	echo docker:999 >> /etc/projid 
        
           //	xfs_quota -x -c 'project -s docker' /<xfs mount point> 
        
           // 
        
           // In that case, the home directory project id will be used as a "start offset" 
        
           // and all containers will be assigned larger project ids (e.g. >= 1000). 
        
           // This is a way to prevent xfs_quota management from conflicting with docker. 
        
           // 
        
           // Then try to create a test directory with the next project id and set a quota 
        
           // on it. If that works, continue to scan existing containers to map allocated 
        
           // project ids. 
        
           func NewControl(basePath string) (*Control, error) { 
        
           	// 
        
           	// If we are running in a user namespace quota won't be supported for 
        
           	// now since makeBackingFsDev() will try to mknod(). 
        
           	// 
        
           	if userns.RunningInUserNS() { 
        
           		return nil, ErrQuotaNotSupported 
        
           	} 
        
           	// 
        
           	// create backing filesystem device node 
        
           	// 
        
           	backingFsBlockDev, err := makeBackingFsDev(basePath)

That code involves CGO though, and depends on some headers;

moby/quota/projectquota.go

Lines 1 to 19 in 75324c4

    
           //go:build linux && !exclude_disk_quota && cgo 
        
           // 
        
           // projectquota.go - implements XFS project quota controls 
        
           // for setting quota limits on a newly created directory. 
        
           // It currently supports the legacy XFS specific ioctls. 
        
           // 
        
           // TODO: use generic quota control ioctl FS_IOC_FS{GET,SET}XATTR 
        
           //       for both xfs/ext4 for kernel version >= v4.5 
        
           // 
        
           package quota // import "github.com/docker/docker/quota" 
        
           /* 
        
           #include <stdlib.h> 
        
           #include <dirent.h> 
        
           #include <linux/fs.h> 
        
           #include <linux/quota.h> 
        
           #include <linux/dqblk_xfs.h>

I also seem to recall that feature detection may happen on /var/lib/docker; in your case, is the whole filesystem xfs, or are you using a separate mount for it? Are you using a custom configuration to use a non-default path for docker's storage?

From the output of your docker version and docker info, I suspect you're running a version of docker that was built and packaged by your distro (Ubuntu);

Git commit:        24.0.5-0ubuntu1~22.04.1

That may not be relevant, but we know that distro-maintainers tend to make modifications in the build-process (including different versions of the Go toolchain and build-time dependencies), which has resulted in (sometimes subtly) broken packages; if you have a system to test on, do you also see the same issue when installing packages from download.docker.com? (instructions for installing in https://docs.docker.com/engine/install/ubuntu/).

MohamedKarrab · 2024-02-28T11:58:24Z

I need this on extfs I also have this problem

tung06021992 · 2024-03-20T07:00:50Z

Dear @thaJeztah
Do you have any update for this issue ?
My system is running extfs so I need resolve it.
Thank you so much.

thaJeztah · 2024-03-20T09:47:12Z

@tung06021992 it's not supported on extfs currently; see #29364

SkyperTHC · 2024-09-20T13:56:52Z

I've been using storage-opts for 2 years (pquota worked just fine and well), now upgraded docker and the fight is on.

Docker fails:

docker run --rm --storage-opt size=1g alpine id
docker: Error response from daemon: --storage-opt is supported only for overlay over xfs with 'pquota' mount option.
See 'docker run --help'.

Backing FS and Root-Dir is correct

# docker info | grep -E '(Root|Back)'
  Backing Filesystem: xfs
 Docker Root Dir: /sf/docker

Mount points and options are correct:

# grep xfs /etc/fstab
UUID=ea522e3a-e9a9-4f28-8553-d0157029364a     /sf    xfs    defaults,nofail,noatime,usrquota,pquota 1 2

# mount | grep xfs
/dev/mapper/vg0-sf on /sf type xfs (rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=1024,swidth=2048,usrquota,prjquota)

Other:

Linux fb 6.1.0-25-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.106-3 (2024-08-26) x86_64 GNU/Linux
Docker version 20.10.24+dfsg1, build 297e128

thaJeztah · 2024-09-20T15:19:01Z

Docker 20.10 reached EOL in this project; based on the version output (20.10.24+dfsg1), you're probably running a version of docker that's packaged and maintained by the Debian packagers; if this was an existing installation, and you previously ran a version of 20.10 as well, it's worth reporting with the debian maintainers; it's possible that either something's broken in their packages, or that other updates on your system caused issues.

SkyperTHC · 2024-09-22T20:25:28Z

thanks for your reply. The problem remains even when upgrading to the latest docker version.

docker version
Client: Docker Engine - Community
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:41:11 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Fri Sep 20 11:41:11 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.22
  GitCommit:        7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc:
  Version:          1.1.14
  GitCommit:        v1.1.14-0-g2c9f560
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

thaJeztah · 2024-09-23T08:41:34Z

I had a quick peek at the code; it looks like detection for quota support is handled during initialization of the graphdriver;

moby/daemon/graphdriver/overlay2/overlay.go

Lines 192 to 202 in 96ea6e0

    
           if backingFs == "xfs" { 
        
           	// Try to enable project quota support over xfs. 
        
           	if d.quotaCtl, err = quota.NewControl(home); err == nil { 
        
           		projectQuotaSupported = true 
        
           	} else if opts.quota.Size > 0 { 
        
           		return nil, fmt.Errorf("Storage option overlay2.size not supported. Filesystem does not support Project Quota: %v", err) 
        
           	} 
        
           } else if opts.quota.Size > 0 { 
        
           	// if xfs is not the backing fs then error out if the storage-opt overlay2.size is used. 
        
           	return nil, fmt.Errorf("Storage Option overlay2.size only supported for backingFS XFS. Found %v", backingFs) 
        
           }

The quota.NewControl(home) there returns an error when quota could not be supported, which could be either a sentinel ErrQuotaNotSupported error, for example, if the daemon is running with user-namespaces enabled;

moby/quota/projectquota.go

Lines 125 to 127 in 96ea6e0

    
           if userns.RunningInUserNS() { 
        
           	return nil, ErrQuotaNotSupported 
        
           }

But could also return an error for other reasons;

moby/quota/projectquota.go

Lines 139 to 142 in 96ea6e0

    
           hasQuotaSupport, err := hasQuotaSupport(backingFsBlockDev) 
        
           if err != nil { 
        
           	return nil, err 
        
           }

However, those errors are discarded unless the daemon is started with overlay2.size is set as option; otherwise only the result ("not supported") is kept;

moby/daemon/graphdriver/overlay2/overlay.go

Lines 196 to 197 in 96ea6e0

    
           } else if opts.quota.Size > 0 { 
        
           	return nil, fmt.Errorf("Storage option overlay2.size not supported. Filesystem does not support Project Quota: %v", err)

In that last case, no error is produced during daemon startup, and when trying to create a container with a size set, a fixed error is returned, which assumes it's not supported because xfs (which may be the case, but could also be for any of the other reasons);

moby/daemon/graphdriver/overlay2/overlay.go

Lines 331 to 333 in 96ea6e0

    
           if _, ok := opts.StorageOpt["size"]; ok && !projectQuotaSupported { 
        
           	return fmt.Errorf("--storage-opt is supported only for overlay over xfs with 'pquota' mount option") 
        
           }

Based on the above, at least some improvements could be made;

I think at least the error-handling part can be improved; make sure that any error encountered is surfaced. The error can be logged when starting the daemon, but given that quota support is an optional feature, that should not be logged as an error, but either a "warning", or even an "info" message.
The error that's produced when creating a container can be made less ambiguous; if that code doesn't know whether xfs is used (and the quota not being supported because of that) it should not make that assumption (i.e., the --storage-opt is supported only for overlay over xfs with 'pquota' mount option may be for a very different reason).

As to the cause of this issue on your machine (more so because you indicated this was after upgrading your host), I wonder if there's some change in newer kernel versions; I see there's some CGO code involved, and wondering if there's some incompatibility with newer kernel versions. Unfortunately, due to the error potentially being swallowed, that information is missing in this ticket currently;

moby/quota/projectquota.go

Lines 14 to 22 in 96ea6e0

    
           /* 
        
           #include <stdlib.h> 
        
           #include <dirent.h> 
        
           #include <linux/fs.h> 
        
           #include <linux/quota.h> 
        
           #include <linux/dqblk_xfs.h> 
        
           #ifndef FS_XFLAG_PROJINHERIT 
        
           struct fsxattr {

thaJeztah · 2024-09-23T08:46:35Z

@SkyperTHC if you have an environment that reproduces this issue, and if that environment can be used for testing; would you be able to try starting the daemon with the overlay2.size storage-option set?

First start the docker service so that the daemon isn't running;

systemctl stop docker

Then manually start the daemon with the storage-option set (overlay2.size or size - I think either works for this);

dockerd --storage-opt size=1g

SkyperTHC · 2024-09-24T11:55:50Z

sorry for the late reply. No Luck.

dockerd --storage-opt overlay2.size=1g
failed to start daemon: error initializing graphdriver: Storage option overlay2.size not supported. Filesystem does not support Project Quota: failed to open dir: /sf/docker/overlay2/3aaae444b60d9904ab43e83c2d04a015ff0f8495bde588ffdb76?69318b548ae

FYI, no prj-flag is set (and that's correct):

lsattr -dp /sf/docker/overlay2/
    0 ---------------------- /sf/docker/overlay2/

whereas the XFS quota is certainly working and works well when I use xfs_quota:

xfs_quota -x -c "limit -p bhard=128T 31337"
xfs_quota -x -c "report" | grep 31337
#31337      164690436          0 137438953472     00 [--------]

I set a dummy project-id on /sf/docker/overlay2. Did not help but the error message changed:

xfs_quota -x -c "project -s -p /sf/docker/overlay2 31337"
lsattr -dp /sf/docker/overlay2
31337 -------------------P-- /sf/docker/overlay2

The error message changed to:

dockerd --storage-opt overlay2.size=1g
failed to start daemon: error initializing graphdriver: driver not supported

Removing the prj-id from /sf/docker/overlay2 and all is back to the normal error:

chattr -RP /sf/docker/overlay2
dockerd --storage-opt overlay2.size=1g
failed to start daemon: error initializing graphdriver: Storage option overlay2.size not supported. Filesystem does not support Project Quota: failed to open dir: /sf/docker/overlay2/3aaae444b60d9904ab43e83c2d04a015ff0f8495bde588ffdb76?69318b548ae

I dont have access to an env to reproduce this (unless I find more spare time) but happy to help as much as I can.

thaJeztah · 2024-09-24T14:04:01Z

dockerd --storage-opt overlay2.size=1g
failed to start daemon: error initializing graphdriver: Storage option overlay2.size not supported. Filesystem does not support Project Quota: failed to open dir: /sf/docker/overlay2/3aaae444b60d9904ab43e83c2d04a015ff0f8495bde588ffdb76?69318b548ae

Hm; interesting; so that error looks to originate from some CGO code here;

moby/quota/projectquota.go

Lines 382 to 388 in c7e42d8

    
           func openDir(path string) (*C.DIR, error) { 
        
           	Cpath := C.CString(path) 
        
           	defer free(Cpath) 
        
           	dir := C.opendir(Cpath) 
        
           	if dir == nil { 
        
           		return nil, errors.Errorf("failed to open dir: %s", path)

At least that slightly narrows down where to look.

rajat709 · 2024-10-03T11:14:34Z

Hi @thaJeztah ,

Due to all this, I am using devicemapper, but it does not support Docker Engine versions above 24.0.9. It worked fine for a long time, but now, with a fresh installation, when I pull any image, I get the following error:

docker: failed to register layer: devicemapper: error running deviceSuspend dm_task_run failed

mmusenbr · 2025-03-06T09:42:15Z

I was currently investigating the same issue on a setup, turns out nodev was set as mount option which caused the issue. After removing the nodev option from the xfs-mount project quota was working.

rajat709 added kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. status/0-triage labels Nov 17, 2023

thaJeztah closed this as not planned Won't fix, can't repro, duplicate, stale Nov 17, 2023

ayushr2 mentioned this issue May 15, 2025

Overlay size, as alternative to --storage-opt size=30G google/gvisor#11504

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Docker Run Error with –storage-opt and Overlay2 on XFS with pquota #46823

Docker Run Error with –storage-opt and Overlay2 on XFS with pquota #46823

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Docker Run Error with –storage-opt and Overlay2 on XFS with pquota #46823

Docker Run Error with –storage-opt and Overlay2 on XFS with pquota #46823

Comments

Description

Reproduce

Expected behavior

docker version

docker info

Additional Info

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!