context: use actual git ref instead of commit sha #677

crazy-max · 2025-04-23T11:19:52Z

Currently the git reference used with git context for push events (branch or tag) is always set to commit sha. This is problematic as fetching by commit sha does not retrieve tags.

For example in https://github.com/docker/buildx/actions/runs/14474897650/job/40598541434#step:7:495

#3 [internal] load git source https://github.com/docker/buildx.git#28c90eadc4c12cc78155ad59ca5f486220241d2a
#3 0.012 Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/3/fs/.git/
#3 0.015 commit
#3 1.322 From file:///var/lib/buildkit/runc-overlayfs/snapshots/snapshots/1/fs
#3 1.322  * branch            refs/buildkit/et2ai7t745oioyi79d3m3hs1c -> FETCH_HEAD
#3 2.120 Note: switching to 'FETCH_HEAD'.
#3 2.120 
#3 2.120 You are in 'detached HEAD' state. You can look around, make experimental
#3 2.120 changes and commit them, and you can discard any commits you make in this
#3 2.120 state without impacting any branches by switching back to a branch.
#3 2.120 
#3 2.120 If you want to create a new branch to retain commits you create, you may
#3 2.120 do so (now or later) by using -c with the switch command. Example:
#3 2.120 
#3 2.120   git switch -c <new-branch-name>
#3 2.120 
#3 2.120 Or undo this operation with:
#3 2.120 
#3 2.120   git switch -
#3 2.120 
#3 2.120 Turn off this advice by setting config variable advice.detachedHead to false
#3 2.120 
#3 2.120 HEAD is now at 28c90ea Merge pull request #3116 from crazy-max/v0.23-picks-0.23.0
#3 DONE 2.1s
...
 #32 [linux/amd64 buildx-build 1/1] RUN --mount=type=bind,target=.   --mount=type=cache,target=/root/.cache   --mount=type=cache,target=/go/pkg/mod   --mount=type=bind,from=buildx-version,source=/buildx-version,target=/buildx-version <<EOT (set -e...)
#32 0.340 + CGO_ENABLED=0 go build -mod vendor -trimpath -ldflags '-X github.com/docker/buildx/version.Version=28c90ea -X github.com/docker/buildx/version.Revision=28c90eadc4c12cc78155ad59ca5f486220241d2a -X github.com/docker/buildx/version.Package=github.com/docker/buildx -s -w' -o /usr/bin/docker-buildx ./cmd/buildx
#32 ...

This workflow runs on push event with refs/tags/v0.23.0 git reference but here version.Version is set to 28c90ea because BuildKit fetches by commit sha and no tags are fetched.

It leads to inconsistency with actions/checkout action that fetches by git reference with tags.

With this change we are setting the actual git reference and fallback to commit sha if GitHub context does not have a ref (should never happen).

Before:

$ docker buildx bake https://github.com/docker/buildx.git#28c90eadc4c12cc78155ad59ca5f486220241d2a
...
#2 [internal] load git source https://github.com/docker/buildx.git#28c90eadc4c12cc78155ad59ca5f486220241d2a
#2 0.040 Initialized empty Git repository in /var/lib/docker/overlay2/hh69bksmo36g5zue054yzl7pu/diff/.git/
#2 0.043 commit
#2 0.460 From file:///var/lib/docker/overlay2/3035vhcu76jdybl0vivmtyedy/diff
#2 0.460  * branch            refs/buildkit/vymteqx5uv90gn7pa5kqjl1ra -> FETCH_HEAD
#2 1.298 Note: switching to 'FETCH_HEAD'.
#2 1.298
#2 1.298 You are in 'detached HEAD' state. You can look around, make experimental
#2 1.298 changes and commit them, and you can discard any commits you make in this
#2 1.298 state without impacting any branches by switching back to a branch.
#2 1.298
#2 1.298 If you want 
8000
to create a new branch to retain commits you create, you may
#2 1.298 do so (now or later) by using -c with the switch command. Example:
#2 1.298
#2 1.298   git switch -c <new-branch-name>
#2 1.298
#2 1.298 Or undo this operation with:
#2 1.298
#2 1.298   git switch -
#2 1.298
#2 1.298 Turn off this advice by setting config variable advice.detachedHead to false
#2 1.298
#2 1.298 HEAD is now at 28c90ea Merge pull request #3116 from crazy-max/v0.23-picks-0.23.0
#2 DONE 1.3s
...
#13 [buildx-build 1/1] RUN --mount=type=bind,target=.   --mount=type=cache,target=/root/.cache   --mount=type=cache,target=/go/pkg/mod   --mount=type=bind,from=buildx-version,source=/buildx-version,target=/buildx-version <<EOT (set -e...)
#13 0.378 + CGO_ENABLED=0 go build -mod vendor -trimpath -ldflags '-X github.com/docker/buildx/version.Version=28c90ea -X github.com/docker/buildx/version.Revision=28c90eadc4c12cc78155ad59ca5f486220241d2a -X github.com/docker/buildx/version.Package=github.com/docker/buildx -s -w' -o /usr/bin/docker-buildx ./cmd/buildx

After:

$ docker buildx bake https://github.com/docker/buildx.git#refs/tags/v0.23.0
...
#2 [internal] load git source https://github.com/docker/buildx.git#refs/tags/v0.23.0
#2 0.633 28c90eadc4c12cc78155ad59ca5f486220241d2a       refs/tags/v0.23.0
#2 0.828 Initialized empty Git repository in /var/lib/docker/overlay2/0ppr3of1n71yuli08v2gx2i8n/diff/.git/
#2 0.831 commit
#2 1.271 From file:///var/lib/docker/overlay2/3035vhcu76jdybl0vivmtyedy/diff
#2 1.271  * [new tag]         refs/tags/v0.23.0 -> v0.23.0
#2 1.272  * [new tag]         refs/tags/v0.23.0 -> refs/tags/v0.23.0
#2 2.230 Note: switching to 'FETCH_HEAD'.
#2 2.230
#2 2.230 You are in 'detached HEAD' state. You can look around, make experimental
#2 2.230 changes and commit them, and you can discard any commits you make in this
#2 2.230 state without impacting any branches by switching back to a branch.
#2 2.230
#2 2.230 If you want to create a new branch to retain commits you create, you may
#2 2.230 do so (now or later) by using -c with the switch command. Example:
#2 2.230
#2 2.230   git switch -c <new-branch-name>
#2 2.230
#2 2.230 Or undo this operation with:
#2 2.230
#2 2.230   git switch -
#2 2.230
#2 2.230 Turn off this advice by setting config variable advice.detachedHead to false
#2 2.230
#2 2.230 HEAD is now at 28c90ea Merge pull request #3116 from crazy-max/v0.23-picks-0.23.0
#2 DONE 2.9s
...
#16 [buildx-build 1/1] RUN --mount=type=bind,target=.   --mount=type=cache,target=/root/.cache   --mount=type=cache,target=/go/pkg/mod   --mount=type=bind,from=buildx-version,source=/buildx-version,target=/buildx-version <<EOT (set -e...)
#16 0.354 + CGO_ENABLED=0 go build -mod vendor -trimpath -ldflags '-X github.com/docker/buildx/version.Version=v0.23.0 -X github.com/docker/buildx/version.Revision=28c90eadc4c12cc78155ad59ca5f486220241d2a -X github.com/docker/buildx/version.Package=github.com/docker/buildx -s -w' -o /usr/bin/docker-buildx ./cmd/buildx

In the future with CSV support for git context moby/buildkit#5903, we could set both the git reference and commit sha.

Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>

fiam · 2025-04-23T11:39:09Z

This is problematic as fetching by commit sha does not retrieve tags.

Shouldn't this be fixed on the BuildKit side?

crazy-max · 2025-04-23T11:54:05Z

src/context.ts

+    if (!ref) {
+      if (!sha) {
+        throw new Error('Git reference or commit SHA is required');
+      }


Similar to actions/checkout behavior: https://github.com/actions/checkout/blob/85e6279cec87321a52edac9c87bce653a07cf6c2/src/ref-helper.ts#L22-L24

crazy-max · 2025-04-23T12:03:00Z

This is problematic as fetching by commit sha does not retrieve tags.

Shouldn't this be fixed on the BuildKit side?

I think the logic in BuildKit makes sense. GitHub does support fetching tags with commit sha: https://github.com/actions/checkout/blob/85e6279cec87321a52edac9c87bce653a07cf6c2/src/ref-helper.ts#L79-L128

With the example in PR description it would be +28c90eadc4c12cc78155ad59ca5f486220241d2a:refs/tags/v0.23.0 but not all providers support this afaik.

And BuildKit could not use this kind of ref anyway with the : char that is currently used to set context to a subdir: https://docs.docker.com/build/concepts/context/#git-repositories

$ docker buildx bake https://github.com/docker/buildx.git#+28c90eadc4c12cc78155ad59ca5f486220241d2a:refs/tags/v0.23.0
#0 building with "default" instance using docker driver

#1 [internal] load git source https://github.com/docker/buildx.git#+28c90eadc4c12cc78155ad59ca5f486220241d2a:refs/tags/v0.23.0
#1 ERROR: repository does not contain ref +28c90eadc4c12cc78155ad59ca5f486220241d2a, output: ""

I think moby/buildkit#5903 would help for such case.

tonistiigi · 2025-04-23T17:42:02Z

We need to check how this works with pull/ refs and merge queues. Eg. I've seen this dynamic ref getting deleted when PR is added to merge queue.

Ultimately moby/buildkit#5903 is a better fix and we can also see if we can just keep the tags in the git source.

crazy-max · 2025-04-24T08:14:02Z

After discussing offline with @tonistiigi, tags are fetched with commit sha as ref: https://github.com/moby/buildkit/blob/f2ce016376e5e61ff87d95cd20ccdf108fa77230/source/git/source.go#L482-L489

The issue we have in https://github.com/docker/buildx/actions/runs/14474897650/job/40598541434#step:7:495 is because we first parse the definition using the git context that effectively fetch the tags: https://github.com/docker/buildx/actions/runs/14474897650/job/40598541434#step:7:213

But as the source is now cached with only commit as ref, other subsequent checkout don't retain tags as there is no remote lookup: https://github.com/docker/buildx/actions/runs/14474897650/job/40598541434#step:7:499

context: use actual git ref instead of commit sha

fd2010a

Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>

crazy-max requested a review from tonistiigi April 23, 2025 11:19

crazy-max self-assigned this Apr 23, 2025

crazy-max commented Apr 23, 2025

View reviewed changes

vvoland approved these changes Apr 23, 2025

View reviewed changes

crazy-max marked this pull request as draft April 24, 2025 08:07

This was referenced Apr 24, 2025

ci: fix bin-image job docker/buildx#3155

Open

ci(bin-image): use distribute reusable workflow moby/moby#49474

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

context: use actual git ref instead of commit sha #677

context: use actual git ref instead of commit sha #677

context: use actual git ref instead of commit sha #677

Are you sure you want to change the base?

context: use actual git ref instead of commit sha #677

Conversation

Choose a reason for hiding this comment