Add suppport for PuzzleFS images in the oci template #4483

ariel-miculas · 2024-09-18T10:16:10Z

PuzzleFS images (media type 'application/vnd.puzzlefs.image.rootfs.v1') can be mounted in a similar way to squashfs images, we just have to detect the type and reuse the existing code for providing a mount helper. PuzzleFS is a next-generation container filesystem [1] with several benefits, such as reduced duplication, reproducible image builds, direct mounting support and memory safety guarantees.

Since PuzzleFS currently doesn't provide an image config, also add support for empty image configs, they are supported by the OCI spec [2].

[1] https://github.com/project-machine/puzzlefs
[2] https://github.com/opencontainers/image-spec/blob/main/manifest.md#image-manifest

lxc-jenkins · 2024-09-18T10:16:14Z

This pull request didn't trigger Jenkins as its author isn't in the allow list.

An organization member must perform one of the following:

To have this branch tested by Jenkins, use the "ok to test" command.
To have a one time test done, use the "test this please" command.

Those commands are simple Github comments of the format: "jenkins: COMMAND"

ariel-miculas · 2024-09-18T10:16:40Z

@hallyn could you please take a look?

hallyn · 2024-09-18T17:17:25Z

If this is all it takes, that's great. Huge +1 from me on making testing that much easier.

ariel-miculas · 2024-09-18T17:34:16Z

I'm curious how @smoser uses this template with atomfs images. It would seem that it needs the persist flag in order to create an overlay. The PuzzleFS mount helper also has to setup an overlay because lxc write configuration files into the rootfs of the container.

ariel-miculas · 2024-09-18T17:51:15Z

It seems like the atomfs semantics changed in this commit, so I guess the atomfs mounts no longer work out of the box.

@hallyn I'm wondering if I should add an --upperdir parameter to the generic mount command. It would be nice if atomfs/puzzlefs could work out of the box, without needing to pass --mount-helper. The mount helper could be detected from the media type.

hallyn · 2024-09-18T18:14:39Z

Agreed, mount helper should be easily detected from media type. We could also support a simple toml file somewhere to customize over our defauls.

To be clear: that's a nice-to-have, but not required for a commit to enable puzzlefs.

--upper-dir (or, I think I prefer --persist-to, but maybe that's weird sounding) would be nice.

Anyway let me set up a vm to test this in.

ariel-miculas · 2024-09-18T18:17:25Z

Also I've noticed we set lxc.execute.cmd which is used mostly by lxc-execute, but not lxc.init.cmd, which is used by lxc-start. So I cannot start my container because it doesn't have the default /sbin/init path that lxc uses.

hallyn · 2024-09-21T02:37:45Z

Also I've noticed we set lxc.execute.cmd which is used mostly by lxc-execute, but not lxc.init.cmd, which is used by lxc-start. So I cannot start my container because it doesn't have the default /sbin/init path that lxc uses.

That's intentional: traditionally, OCI images are used for application containers. Of course we've been using them to ship RFS for hardware, so it may be worth revisiting that decision.

hallyn · 2024-09-21T02:41:10Z

Do you mind adding the pf-mount-helper binary and logic to choose the right helper based on the application/vnd..image.layer.squashfs vs application/vnd.puzzlefs.image.rootfs.*`to this patch?

ariel-miculas · 2024-09-21T09:05:05Z

For context, this it my presentation at LPC 2024 where I show how to run an LXC container with PuzzleFS images.

@hallyn I'm wondering what's the best approach here:

if we put the mount helper in LXC, I think it will be more difficult to make changes to it; it something changes on the PuzzleFS, we're stuck waiting for the next version of LXC in order to make changes to the mount helper; and the issue is that we still rely on an external program (the puzzlefs binary itself), so it's not like having the mount helper in LXC will make the template work without external dependencies
should we actually just pass the puzzlefs binary itself as the mount-helper and let it deal with creating the overlay, like atomfs does? I don't think the OCI template works with new versions of atomfs now, because it doesn't pass the upperdir option
which takes me to my next point: how do we design this to work the same for puzzlefs and atomfs? should we have a generic mount helper for both that just creates an overlay on top of the atomfs/puzzlefs mountpoint? I think we already need to change the mount-helper arguments, as it doesn't account for the additional --upperdir option that we need to pass to atomfs

I think the solution with the least amount of changes would be to just call the atomfs/puzzlefs binaries (or maybe atomfs-lxc/puzzlefs-lxc so we can have wrapper scripts that do the overlay mounting) when we detect their corresponding media type and to change the mount-helper arguments to accept an overlay. That way, we don't need to add puzzlefs specific mount helpers in LXC.

But maybe the solution with a generic overlay mount helper is more elegant, or even implement this logic into LXC itself. I mean, there already is overlayfs:/lower:/upper syntax for lxc.rootfs.path. Hmm, or maybe just using this existing syntax to create the overlay.

@mihalicyn any thoughts on this?

ariel-miculas · 2024-09-23T11:29:30Z

But maybe the solution with a generic overlay mount helper is more elegant, or even implement this logic into LXC itself. I mean, there already is overlayfs:/lower:/upper syntax for lxc.rootfs.path. Hmm, or maybe just using this existing syntax to create the overlay.

Having looked at the atomfs codebase, this wouldn't work because there are too many mountpoints being created.

Having said this, I would either go with option 2 and add the necessary logic for creating an overlay in puzzlefs, or use the overlayfs syntax in lxc.rootfs.path for puzzlefs (if it's even possible), but not atomfs. I still don't like storing the puzzlefs mount helper script in lxc.

hallyn · 2024-09-26T03:08:04Z

Ok, no puzzlefs mount helper in lxc.

What do you mean by atomfs makes too many mounts? It does hide them behind the final mount, right?

ariel-miculas · 2024-09-26T06:04:19Z

What do you mean by atomfs makes too many mounts? It does hide them behind the final mount, right?

Right, I just meant that we couldn't implement a generic overlayfs functionality in LXC because of the complex requirements that atomfs has. But now that I think about it, atomfs could mount a read-only overlay and then LXC could mount a writable overlay on top. I don't know whether we should go for this or not.

use the overlayfs syntax in lxc.rootfs.path for puzzlefs

This could work for starting the container, but we also need to mount the rootfs in the LCX OCI template script itself, so it's not a good solution.

What do you think we should do, @hallyn ?

hallyn · 2024-10-01T14:12:55Z

Honestly I think we are over complicating the issue. From lxc side, we should just say mount-helper api has to be: mount ocidir:image [-w writeable-dir] destdir. We make atomfs and puzzlefs honor those, and anybother mount helper which doesn't can provide its own wrapper script.

ariel-miculas · 2024-10-01T15:32:30Z

This sounds reasonable, I'll go ahead and implement it this way.

The mount-helper is called like this: mount-helper mount --writable <oci_dir>:<oci_name> <mountpoint> mount-helper umount <mountpoint> I've added the support for the `--writable` flag that creates a writable overlay on top of a PuzzleFS mountpoint (used as a lowerdir). This requires root privileges. I've also added the support for the `umount` subcommand which detects whether the puzzlefs filesystem was mounted as a read-only fuse driver or whether an overlay was created on top of it, and handles the unmounting accordingly. Unmounting an overlay requires root privileges. I've implemented the changes on the LXC side in [1]. [1] lxc/lxc#4483 Signed-off-by: Ariel Miculas-Trif <amiculas@cisco.com>

ariel-miculas · 2024-10-03T20:08:12Z

@hallyn this should work with the upcoming puzzlefs version (not yet merged, PR is here). Let me know if this is what you had in mind for the mount-helper.

hallyn · 2024-10-04T14:53:00Z

It needs to be spelled out somewhere what the writeable directory will be.

It looks like puzzlefs is putting it under $mountpoint/upper. I think it would be better to put it, for lxc containers, under ${lxc_path}/${container_name}/upper. So the lxc-oci template would likely have to pass the upperdir as a value for the -writable argument. That way when I create an unprivileged (atomfs) container, i know that I can find the delta under $HOME/.local/share/lxc/${container}/upper.

ariel-miculas · 2024-10-04T15:07:37Z

For a privileged container mycontainer, the upperdir is in /var/lib/lxc/mycontainer/rootfs/upper because the mountpoint is /var/lib/lxc/mycontainer/rootfs. Here is also where the overlay gets mounted. You want it moved outside of rootfs, in /var/lib/lxc/mycontainer/upper?

hallyn · 2024-10-04T16:45:03Z

For a privileged container mycontainer, the upperdir is in /var/lib/lxc/mycontainer/rootfs/upper because the mountpoint is /var/lib/lxc/mycontainer/rootfs. Here is also where the overlay gets mounted. You want it moved outside of rootfs, in /var/lib/lxc/mycontainer/upper?

Right.

ariel-miculas · 2024-10-04T18:46:09Z

I have the following error with the latest changes:

lxc-start: mycontainer: ../src/lxc/conf.c: turn_into_dependent_mounts: 3948 No such file or directory - Failed to recursively turn old root mount tree into dependent mount. Continuing...

This occurs when I start the container:

$ sudo build/src/lxc/tools/lxc-start --name mycontainer --foreground /bin/bash

The mount-helper is called like this: mount-helper mount --writable <oci_dir>:<oci_name> <mountpoint> mount-helper umount <mountpoint> I've added the support for the `--writable` flag that creates a writable overlay on top of a PuzzleFS mountpoint (used as a lowerdir). This requires root privileges. I've also added the support for the `umount` subcommand which detects whether the puzzlefs filesystem was mounted as a read-only fuse driver or whether an overlay was created on top of it, and handles the unmounting accordingly. Unmounting an overlay requires root privileges. I've implemented the changes on the LXC side in [1]. [1] lxc/lxc#4483 Signed-off-by: Ariel Miculas-Trif <amiculas@cisco.com>

PuzzleFS images (media type 'application/vnd.puzzlefs.image.rootfs.v1') can be mounted in a similar way to squashfs images, we just have to detect the type and reu 8000 se the existing code for providing a mount helper. PuzzleFS is a next-generation container filesystem [1] with several benefits, such as reduced duplication, reproducible image builds, direct mounting support and memory safety guarantees. Since PuzzleFS currently doesn't provide an image config, also add support for empty image configs, they are supported by the OCI spec [2]. The MOUNT_HELPER is now passed a `--persist <upperdir>` flag, so it knows that it needs to create an overlay. This is needed because LXC expects a writable rootfs and both atomfs and puzzlefs are read-only filesystems. Example: ``` $ sudo env PATH=$PATH build/src/lxc/tools/lxc-create --name mycontainer -t \ oci -- --url oci:/$HOME/.local/share/puzzlefs/pfs_ubuntu:eg --no-cache $ sudo build/src/lxc/tools/lxc-start --name mycontainer --foreground /bin/bash ``` --no-cache is needed for puzzlefs until [3] is solved [1] https://github.com/project-machine/puzzlefs [2] https://github.com/opencontainers/image-spec/blob/main/manifest.md#image-manifest [3] project-machine/puzzlefs#131 Signed-off-by: Ariel Miculas-Trif <amiculas@cisco.com>

hallyn · 2024-10-04T19:08:50Z

Can you show the full logs? (lxc-start [....] -l trace -o debug.out)

The error you quote says "continuing", so I don't think it's the actual problem...

ariel-miculas · 2024-10-04T19:13:15Z

Well the container does start...
Here's the log: https://pastebin.com/Z9pxqGV5

ariel-miculas · 2024-10-04T19:27:08Z

Here's the log at level TRACE: https://pastebin.com/Xhmaf9Wr

hallyn · 2024-10-04T22:20:14Z

Hm, so it did start, appears to have immediately exited:

lxc-start mycontainer 20241004191128.159 DEBUG    start - ../src/lxc/start.c:signal_handler:464 - Container init process 2677157 exited

Hm, looking at the puzzlefs and lxc patches, the mount-helper is not being passed --writable, but puzzlefs still requires --writable as well as --persist. So my guess is your container is actually readonly right now?

I was actually suggesting that --writable not be a bool, but itself accept the string for the upperdir. That's all bikeshedding though, I'm fine if you want to say that

--writable --persist /a/b will put the upperdir at /a/b
--writable (without --persist) will pick and create its own upperdir.

ariel-miculas · 2024-10-05T08:13:37Z

Hm, so it did start, appears to have immediately exited:

That's because I pressed Ctrl-D right after it started

Hm, looking at the puzzlefs and lxc patches, the mount-helper is not being passed --writable, but puzzlefs still requires --writable as well as --persist. So my guess is your container is actually readonly right now?

It's enough to pass --persist <upperdir>.

❯ sudo target/release/puzzlefs mount --persist /tmp/upper /tmp/pfs:eg /tmp/mnt
❯ mount  | tail -n 2
/dev/fuse on /tmp/mnt/ro type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0)
overlay on /tmp/mnt type overlay (rw,relatime,lowerdir=/tmp/mnt/ro,upperdir=/tmp/upper,workdir=/tmp/mnt/work,uuid=on,xino=off,nouserxattr)
❯ sudo touch /tmp/mnt/asdoijfoidsaf
❯ ls /tmp/upper
asdoijfoidsaf

I was actually suggesting that --writable not be a bool, but itself accept the string for the upperdir. That's all bikeshedding though, I'm fine if you want to say that

I just copied the flags from atomfs, so we can keep the existing atomfs implementation as it is. Of course, if there's a better way, I'm not opposed to changing the flags.

The current puzzlefs behavior is (not need to pass --writable, --persist <upperdir> is enough):

--persist /a/b will create an overlay and put the upperdir at /a/b
--writable will pick and create its own upperdir under the specified mountpoint.

ariel-miculas · 2024-10-05T08:35:15Z

Turns out the message

lxc-start: mycontainer: ../src/lxc/conf.c: turn_into_dependent_mounts: 3948 No such file or directory - Failed to recursively turn old root mount tree into dependent mount. Continuing...

wasn't related to my changes, but to another mountpoint I had in my system. So all works well.

Do you want me to make any changes to the --persist <upperdir> flag that is being passed to the mount helper (like renaming it to upper or upperdir)? The current state of this PR works with both puzzlefs and atomfs.

hallyn · 2024-10-06T00:10:36Z

No, while I would prefer that behavior, let's keep changing that as a separate task.

So just be clear, we can merge both this and the puzzlefs patch and all will work?

hallyn · 2024-10-06T00:12:34Z

Thanks!

ariel-miculas · 2024-10-06T12:39:28Z

Thanks, @hallyn !

let's keep changing that as a separate task.

you mean renaming --persist to --upperdir?

So just be clear, we can merge both this and the puzzlefs patch and all will work?

Yes, it all works.

hallyn · 2024-10-07T01:26:51Z

let's keep changing that as a separate task.

you mean renaming --persist to --upperdir?

Yeah. It's fine as is. If we decide it's confusing we can rename it separately later.

The mount-helper is called like this: mount-helper mount --writable <oci_dir>:<oci_name> <mountpoint> mount-helper umount <mountpoint> I've added the support for the `--writable` flag that creates a writable overlay on top of a PuzzleFS mountpoint (used as a lowerdir). This requires root privileges. I've also added the support for the `umount` subcommand which detects whether the puzzlefs filesystem was mounted as a read-only fuse driver or whether an overlay was created on top of it, and handles the unmounting accordingly. Unmounting an overlay requires root privileges. I've implemented the changes on the LXC side in [1]. [1] lxc/lxc#4483 Signed-off-by: Ariel Miculas-Trif <amiculas@cisco.com>

ariel-miculas mentioned this pull request Sep 22, 2024

Cannot create an LXC OCI template with an atomfs rootfs project-machine/atomfs#15

Open

ariel-miculas force-pushed the support_puzzlefs branch from 2368b8c to 44cded3 Compare October 3, 2024 19:52

ariel-miculas mentioned this pull request Oct 3, 2024

Enable puzzlefs to be used as a MOUNT_HELPER in the LXC OCI template project-machine/puzzlefs#133

Merged

ariel-miculas force-pushed the support_puzzlefs branch from 44cded3 to 3764c19 Compare October 4, 2024 18:51

hallyn merged commit bf33e9b into lxc:main Oct 6, 2024
8 checks passed

ariel-miculas mentioned this pull request Oct 16, 2024

Ideas - Feel free to post ideas. youki-dev/youki#10

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add suppport for PuzzleFS images in the oci template #4483

Add suppport for PuzzleFS images in the oci template #4483

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Add suppport for PuzzleFS images in the oci template #4483

Add suppport for PuzzleFS images in the oci template #4483

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!