Add ability to scan snaps (as a source) #3929

wagoodman · 2025-05-21T18:35:09Z

This adds support for cataloging single snap files, cataloging all contents. This can scan a local snap file:

syft /Users/wagoodman/OrbStack/ubuntu/home/wagoodman/snaps/etcd.snap

or grab the snap from snapcraft:

# assumes latest stable with amd64 arch
syft --from snap etcd

# specify a channel and arch
syft --from snap etcd@3.1/stable --platform i386

Note: if --platform is given an OS then the source will error out and not continue (as this is not a valid form).

Fixes add native support for snap packages #1088

Type of change

New feature (non-breaking change which adds functionality)

Checklist

I have added unit tests that cover changed behavior
I have tested my code in common scenarios and confirmed there are no regressions
I have added comments to my code, particularly in hard-to-understand sections

TODO

Add support for remote snaps
Add LZO support (pending Add LZO decompressor diskfs/go-diskfs#298)
Capture snap/manifest.yaml and surrounding data onto source.Metadata
Confirm that only the install cataloger set is being used for this new source
Rebase after merging fix: bump stereoscope to fix symlink performance issue #3953

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

popey · 2025-05-27T14:35:18Z

I'm seeing this a lot:

[0002] ERROR could not determine source: an error occurred attempting to resolve 'adguard-home': snap: unable to create snap file resolver: failed to walk squashfs file="/tmp/syft-snap-653810807/UXZIkJfJT2SPCGejjnSjOBqJ71yHk8bw_8117.snap": path="/" already exists but is NOT a regular file

Yet:

ls -l /tmp/syft-snap-653810807/UXZIkJfJT2SPCGejjnSjOBqJ71yHk8bw_8117.snap
-rw-rw-r-- 1 alan alan 9334784 May 27 15:31 /tmp/syft-snap-653810807/UXZIkJfJT2SPCGejjnSjOBqJ71yHk8bw_8117.snap

file /tmp/syft-snap-653810807/UXZIkJfJT2SPCGejjnSjOBqJ71yHk8bw_8117.snap
/tmp/syft-snap-653810807/UXZIkJfJT2SPCGejjnSjOBqJ71yHk8bw_8117.snap: Squashfs filesystem, little endian, version 4.0, xz compressed, 9332165 bytes, 8 inodes, blocksize: 131072 bytes, created: Tue May 27 12:20:03 2025

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

wagoodman · 2025-06-02T17:28:24Z

@popey pointed out that there are region locked snaps that fail with obtuse errors:

[0000] ERROR could not determine source: an error occurred attempting to resolve 'jp-ledger': snap: unable to open snap manifest file: failed to resolve request: failed to get download URL for snap "jp-ledger": API request failed with status code 404

We should probably use the search API to resolve an exact name match then extract the download link from there (search with q=NAME and only take results that locally match the given name exactly... or something similar). Then we could get an error message more like:

found snap 'jp-ledgrer' (id=jyDlMmifyQhSWGPM9fnKc1HSD7E6c47e) but it is unavailable for download

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

internal/file/getter.go

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

syft/format/common/spdxhelpers/to_format_model.go

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

cmd/syft/internal/commands/scan.go

syft/get_source_test.go

kzantow · 2025-06-05T14:35:01Z

syft/internal/unionreader/union_reader.go

+	r.mu.Lock()
+	defer r.mu.Unlock()
+
+	currentPos, err := r.Seek(0, io.SeekCurrent)


This should probably be tracked when the reader is created -- I don't think getting current position here will always be the right thing to do. I think this existing seekable reader behavior is close, albeit it's not ReaderAt but ReadSeeker:

syft/syft/format/internal/stream/seekable_reader.go

Line 14 in c36c697

func SeekableReader(reader io.Reader) (io.ReadSeeker, error) {

This is attempting to adapt the seeking ability (stateful) with a reader-at like experience (stateless), but doing so affects the pointer into the file, which would affect subsequent reads and seeks, so the first step is to keep the original position in order to restore it later in this call.

I think there is a problem in the sense that we are only locking for ReadAt, which in hindsight doesn't make sense because we're not locking for other operations.

Let's say I have this reader, and I call .Read(bytes) and then I call .ReadAt(0,...), it will not have the correct starting position of the reader. I think, if we track the starting position when this custom reader is created, it should be reasonably assumed to be the "start of the stream". Happy to chat on this further, I was just pointing out it's a similar problem to the other reader I mentioned above

The definition of ReaderAt is that the offset is absolute, not relative to any other read calls --we're only preserving the pointer state for other read/seek calls. That is, it could be that the file pointer is specifically there to subset a selection of the file, in which case the underlying reader should be wrapped with io.SectionReader or similar, where subsetting is explicit (and you can get the underlying non-section reader with .Outer()). However, us creating this wrapper around a io.ReadSeeker should not implicitly mean that we are intending to act like a io.SectionReader.

tangent, I did update this to account for locking for other read/seek calls

syft/pkg/cataloger/python/parse_wheel_egg_record.go

syft/source/snap_metadata.go

syft/source/snapsource/snap_source.go

kzantow · 2025-06-05T20:24:57Z

syft/source/snapsource/snap_source.go

+}
+
+func deriveID(path, name, version string) artifact.ID {
+	info := fmt.Sprintf("%s:%s@%s", digestOfFileContents(path), name, version)


seems like path.Base(path) might be a useful default here somewhere? and usually config aliases aren't set so name and version will be empty. e.g. <name | filename>@<version | hash> or something? this may not make a bit of difference, I didn't quite understand what the id is used for

the goal is to get a stable ID for the source when making relationships between other elements (packages / files) and the source.

Ideally this is content addressable -- regardless of the name of the file or any aliases used, we will get the same ID for the same file every time. However, we've found that for other kinds of sources that if there is a different logical name or version then we should have separate IDs.

This is mimicking the ID derivation from the file source.

syft/source/sourceproviders/source_providers.go

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

kzantow

🎉

wagoodman force-pushed the add-snap-source branch from 8c085cf to f96fce4 Compare May 21, 2025 18:36

add snap source

c83be08

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

wagoodman force-pushed the add-snap-source branch from a9e940d to c83be08 Compare May 22, 2025 23:18

wagoodman added 2 commits May 23, 2025 14:45

port to diskfs lib

c5c785e

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

fix tests

f7771f3

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

associate openers for squashfs files

4c20fa6

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

wagoodman added this to OSS May 29, 2025

wagoodman moved this to In Progress in OSS May 29, 2025

wagoodman self-assigned this May 29, 2025

wagoodman added 3 commits May 29, 2025 13:50

correct manifest path

dc54bc1

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

dont add opener for dirs

1d66df0

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

use upstream (not fork)

bbbedff

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

wagoodman added 3 commits June 3, 2025 09:38

Merge remote-tracking branch 'origin/main' into add-snap-source

4fd731d

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

better reporting of snap name resolution

17b4bf3

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

pull in stereoscope performance fix

88c997c

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

wagoodman moved this from In Progress to In Review in OSS Jun 4, 2025

wagoodman marked this pull request as ready for review June 4, 2025 13:38

wagoodman commented Jun 4, 2025

View reviewed changes

internal/file/getter.go Outdated Show resolved Hide resolved

wagoodman added 2 commits June 4, 2025 11:07

do not read file entries further on read errors

9f7130b

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

add tests for walk filesystem

8e106cc

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

wagoodman force-pushed the add-snap-source branch from e12ee6f to 8e106cc Compare June 4, 2025 15:37

wagoodman added 2 commits June 4, 2025 12:30

Merge remote-tracking branch 'origin/main' into add-snap-source

d127f9c

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

fix python record reader errors

98eb9cc

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

wagoodman commented Jun 4, 2025

View reviewed changes

syft/format/common/spdxhelpers/to_format_model.go Outdated Show resolved Hide resolved

review comments

c97d88a

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

wagoodman requested a review from a team June 4, 2025 17:00

wagoodman added 2 commits June 4, 2025 14:17

change snap primary purpose to container

e5be2dc

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into add-snap-source

e3644ab

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

This was referenced Jun 5, 2025

Catalog git repos natively #3246

Open

Catalog VM images directly #3245

Open

kzantow reviewed Jun 5, 2025

View reviewed changes

address review comments

d39234e

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

wagoodman changed the title ~~Add source that represents snaps~~ Add ability to scan snaps (as a source) Jun 9, 2025

wagoodman added 5 commits June 25, 2025 12:09

split local and remote snap source

01d3670

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into add-snap-source

d08f65f

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

address review comments

24863e3

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

add locking around other read/seek ops in readerAtAdapter

781e063

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

prevent deadlocks for other Read/Seek ops within ReadAt()

60f503c

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

kzantow approved these changes Jun 25, 2025

View reviewed changes

wagoodman merged commit 2bda086 into main Jun 25, 2025
12 checks passed

wagoodman deleted the add-snap-source branch June 25, 2025 20:53

github-project-automation bot moved this from In Review to Done in OSS Jun 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add ability to scan snaps (as a source) #3929

Add ability to scan snaps (as a source) #3929

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Add ability to scan snaps (as a source) #3929

Add ability to scan snaps (as a source) #3929

Uh oh!

Conversation

Uh oh!

Type of change

Checklist

TODO

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!