cmd/go: embed directory fails when at / #49570

thomaspeugeot · 2021-11-13T03:30:12Z

What version of Go are you using (`go version`)?

/ # go version
go version go1.16.10 linux/amd64

Does this issue reproduce with the latest release?

I do not know

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
 # go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.10"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2312574173=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Training for docker with go, I forked github.com/olliefr/docker-gs-ping and added embed directive

//go:embed foo
var foo embed.FS

it worked fined in my working directory.
But when I configured Dockerfile for having everything at root, it did not work.

# Set destination for COPY
WORKDIR /

To reproduce the bug

git clone https://github.com/fullstack-lang/docker-gs-ping.git
cd docker-gs-ping
docker build --tag docker-gs-ping .

What did you expect to see?

a successfull go build

What did you see instead?

 > [7/8] RUN go build -o /docker-gs-ping:
#11 2.002 # github.com/olliefr/docker-gs-ping
#11 2.002 /main.go:14:5: embed oo/foo.txt: open /oo/foo.txt: no such file or directory

A workaround is to have the WORKDIR different from "/"

The text was updated successfully, but these errors were encountered:

seankhliao · 2021-11-13T08:13:52Z

cc @bcmills @matloob

ronaldpetty · 2022-02-26T01:15:27Z

This fails for 1.18rc1 as well (using root directory). Trying to learn a little here, seems to fail almost immediately in the build process. Shows truncating the leading / if it is in root.

Here we are in "/ (root)":

root@82481f861c36:/# go build -x -work -o a2 app.go 
WORK=/tmp/go-build2643592769
mkdir -p $WORK/b001/
cat >$WORK/b001/importcfg << 'EOF' # internal
# import config
packagefile embed=/usr/local/go/pkg/linux_amd64/embed.a
packagefile fmt=/usr/local/go/pkg/linux_amd64/fmt.a
packagefile log=/usr/local/go/pkg/linux_amd64/log.a
packagefile net/http=/usr/local/go/pkg/linux_amd64/net/http.a
packagefile time=/usr/local/go/pkg/linux_amd64/time.a
packagefile runtime=/usr/local/go/pkg/linux_amd64/runtime.a
EOF
cat >$WORK/b001/embedcfg << 'EOF' # internal
{
	"Patterns": {
		"images/*": [
			"mages/me.jpg"
		],
		"index.html": [
			"index.html"
		]
	},
	"Files": {
		"index.html": "/index.html",
		"mages/me.jpg": "/mages/me.jpg"
	}
}EOF
/usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/b001/_pkg_.a -trimpath "$WORK/b001=>" -p main -complete -buildid iQqxv8YiKWDrDnzfPYmP/iQqxv8YiKWDrDnzfPYmP -goversion go1.18rc1 -c=2 -nolocalimports -importcfg $WORK/b001/importcfg -embedcfg $WORK/b001/embedcfg -pack /app.go
# command-line-arguments
/app.go:13:5: embed mages/me.jpg: open /mages/me.jpg: no such file or directory
root@82481f861c36:/# export WORK=/tmp/go-build2643592769
root@82481f861c36:/# /usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/b001/_pkg_.a -trimpath "$WORK/b001=>" -p main -complete -buildid iQqxv8YiKWDrDnzfPYmP/iQqxv8YiKWDrDnzfPYmP -goversion go1.18rc1 -c=2 -nolocalimports -importcfg $WORK/b001/importcfg -embedcfg $WORK/b001/embedcfg -pack /app.go
/app.go:13:5: embed mages/me.jpg: open /mages/me.jpg: no such file or directory
root@82481f861c36:/# ls images/
me.jpg
root@82481f861c36:/# ls mages
ls: cannot access 'mages': No such file or directory

Symlink fixes it.

root@82481f861c36:/# ln -s /images /mages
root@82481f861c36:/# /usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/b001/_pkg_.a -trimpath "$WORK/b001=>" -p main -complete -buildid iQqxv8YiKWDrDnzfPYmP/iQqxv8YiKWDrDnzfPYmP -goversion go1.18rc1 -c=2 -nolocalimports -importcfg $WORK/b001/importcfg -embedcfg $WORK/b001/embedcfg -pack /app.go
root@82481f861c36:/#

I have been digging around, this code seems to fail when the pkgdir is "/".

> cmd/go/internal/load.resolveEmbed() .usr/local/go/src/cmd/go/internal/load/pkg.go:2077 (hits goroutine(1):1 total:1) (PC: 0x831cec)
Warning: debugging optimized function
  2072:					}
  2073:				}
  2074:	
  2075:				switch {
  2076:				default:
=>2077:					return nil, nil, fmt.Errorf("cannot embed irregular file %s", rel)
  2078:	
  2079:				case info.Mode().IsRegular():
  2080:					if have[rel] != pid {
  2081:						have[rel] = pid
  2082:						list = append(list, rel)
(dlv) stack
0  0x0000000000831cec in cmd/go/internal/load.resolveEmbed
   at .usr/local/go/src/cmd/go/internal/load/pkg.go:2077
1  0x000000000082e454 in cmd/go/internal/load.(*Package).load
   at .usr/local/go/src/cmd/go/internal/load/pkg.go:1871
2  0x0000000000827b3d in cmd/go/internal/load.loadImport
   at .usr/local/go/src/cmd/go/internal/load/pkg.go:722
3  0x0000000000836cf2 in cmd/go/internal/load.PackagesAndErrors
   at .usr/local/go/src/cmd/go/internal/load/pkg.go:2706
4  0x00000000008c8433 in cmd/go/internal/list.runList
   at .usr/local/go/src/cmd/go/internal/list/list.go:507
5  0x00000000009035ae in main.invoke
   at .usr/local/go/src/cmd/go/main.go:218
6  0x0000000000902f8e in main.main
   at .usr/local/go/src/cmd/go/main.go:175
7  0x0000000000438212 in runtime.main
   at .usr/local/go/src/runtime/proc.go:250
8  0x0000000000465661 in runtime.goexit
   at .usr/local/go/src/runtime/asm_amd64.s:1571
(dlv) args
pkgdir = "/"
patterns = []string len: 2, cap: 0, [...]
files = []string len: 0, cap: 0, nil
pmap = map[string][]string []
(err) = error nil
(dlv) locals
pattern = "images/*"
have = (unreadable empty OP stack)
dirOK = (unreadable empty OP stack)
pid = 1
glob = "images/*"
all = false
~R0 = (unreadable empty OP stack)
match = (unreadable read out of bounds)
(err) = (unreadable could not find loclist entry at 0x366a1b for address 0x831cec)
list = []string len: 0, cap: 0, nil
file = "/images/mages"
rel = ""
what = "file"
info = io/fs.FileInfo(*os.fileStat) 0xbeef000000000608
err = (unreadable could not find loclist entry at 0x366cf0 for address 0x831cec)
(dlv)

I had thought I caused "go list all" to also fail, but after cleaning things up its seems to no longer fail in the same way.

tenkoh · 2022-03-25T21:36:52Z

Hi, I have investigated about this issue.

My guess:

This issue would be caused by resolveEmbed function in src/cmd/go/internal/load/pkg.go.

Detail:

Reproduce and check the problem:

First of all, I had confirmed go list output changing working directory. I focused on EmbedPattern and EmbedFiles fields.

./
 |- Dockerfile
 |- main.go
 |- go.mod
 |- foo/
     |- foo.txt

main.go

package main

import (
	"embed"
	"fmt"
)

//go:embed foo
var foo embed.FS

func main() {
    // do something
}

Dockerfile

FROM golang:1.18-alpine
# git is required to run `go list`
RUN apk add git --no-cache

WORKDIR /
COPY ./ ./

CMD ["go", "list", "-f", "'pattern: {{.EmbedPatterns}}, files: {{.EmbedFiles}}'"]

Result:

EmbedPatterns are same in both cases. But EmbedFiles is different. So, some codes generating EmbedFiles would have a problem.

working directory	EmbedPattern	EmbedFiles
/	foo	oo/foo.txt
/app	foo	foo/foo.txt

Read codes:

I found a suspicious part in resolveEmbed function in src/cmd/go/internal/load/pkg.go.

https://cs.opensource.google/go/go/+/refs/tags/go1.18:src/cmd/go/internal/load/pkg.go;l=2089;drc=c016133c50512e9a83e7442bd7ac614fe7ca62de

err := fsys.Walk(file, func(path string, info os.FileInfo, err error) error {
    if err != nil {
      return err
    }
    rel := filepath.ToSlash(path[len(pkgdir)+1:])

This function recursively walks around in the working directory, then picks up files. The files' paths are stored as rel shown in the code above.

Here, pkgdir varies by working directory. It is set in GoFilesPackage in src/cmd/go/internal/load/pkg.go

https://cs.opensource.google/go/go/+/refs/tags/go1.18:src/cmd/go/internal/load/pkg.go;l=2929;drc=c016133c50512e9a83e7442bd7ac614fe7ca62de

if dir == "" {
    dir = base.Cwd()
}
dir, err = filepath.Abs(dir)
if err != nil {
    base.Fatalf("%s", err)
}
bp, err := ctxt.ImportDir(dir, 0)

The dir is equal to pkgdir and it differs from the working directory. It is trivial.

working directory	dir
/	/
/app	/app

Considering all above, the rel (i.e. EmbedFile) becomes as below. Wow, we got oo/foo.txt !

pkgdir	path (just an example)	rel = filepath.ToSlash(path[len(pkgdir)+1:])
/	/foo/foo.txt	oo/foo.txt
/app	/app/foo/foo.txt	foo/foo.txt

https://go.dev/play/p/ySwcRSEYVd9

Solution (guess):

So, solution could be adding an exception when pkgdir='/'.

If I have a time, I will challenge a bugfix. But it's the first time to try, so it will take a little long...

tenkoh · 2022-03-30T07:09:58Z

I have fixed resolveEmbed and confirm the result.

https://github.com/tenkoh/fix-resolveEmbed-on-root

gopherbot · 2022-03-30T07:12:17Z

Change https://go.dev/cl/396694 mentions this issue: load: fix resolveEmbed to avoid a build error on a specific situation.

seankhliao changed the title ~~go embed fails at root level (first character is mangled)~~ cmd/go: embed directory fails when module root is at / Nov 13, 2021

seankhliao changed the title ~~cmd/go: embed directory fails when module root is at /~~ cmd/go: embed directory fails when at / Nov 13, 2021

seankhliao added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 13, 2021

gopherbot closed this as completed in 5bf6c97 Apr 12, 2022

tyzbit added a commit to tyzbit/bitcoin-balance-notifier that referenced this issue May 1, 2022

fix: fix for bug golang/go#49570

ee11e55

seankhliao mentioned this issue Nov 22, 2022

cmd/go: support a -C flag value to not use any current directory #56868

Open

golang locked and limited conversation to collaborators Apr 12, 2023

gopherbot added the FrozenDueToAge label Apr 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: embed directory fails when at / #49570

cmd/go: embed directory fails when at / #49570

cmd/go: embed directory fails when at / #49570

cmd/go: embed directory fails when at / #49570

Comments

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

My guess:

Detail:

Reproduce and check the problem:

Read codes:

Solution (guess):

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?