Closed
Description
Due to a bug in docker moby/moby#36789, when we repeat doing
docker tag image:v1 image:latest
docker push image:latest
. It will create the same image with different digests like this
*/fesu/tess-build-jenkins latest sha256:349c662bc29fd98a608a5da773afd50b068a8f624b142c2df3060aa5626783fc 95eaa6e8aae1 12 days
*/fesu/tess-build-jenkins latest sha256:54a37661a521c4f69832b11ea76fd062fe67c6182382a2d291c93fd6eecb47f7 95eaa6e8aae1 12 days
For the pod with imagePullPolicy: true, it will pull the image every time
The flow will be like this, image:v1 is not changed during this flow
the image.ID (ref) sha256:af280c8b0a6945cca90755d2931c849e525c6aca1212ee934191bf9046352786 is not changed, but this image's target is changed.
But for the imageStore in cri for this ref, it will not be updated for this image.
https://github.com/containerd/cri/blob/master/pkg/store/image/image.go#L110
if oldExist {
if oldID == img.ID {
return nil
}
*
}
So when we create the container using this ref, cri will guide us to find the old target blobs. It may be deleted by gc.
Jul 03 22:40:18 containerd[30986]: time="2019-07-03T22:40:18.453608104-07:00" level=debug msg="create image" name="sha256:af280c8b0a6945cca90755d2931c849e525c6aca1212ee934191bf9046352786" target=sha256:22e1171a33b41af2ef71270c57092dffb8fb2b0148f72fa9082e26d44f59e223
Jul 03 22:40:18 containerd[30986]: time="2019-07-03T22:40:18.454204263-07:00" level=debug msg="event published" ns=k8s.io topic="/images/update" type=containerd.services.images.v1.ImageUpdate
Jul 03 22:40:18 containerd[30986]: time="2019-07-03T22:40:18.456067932-07:00" level=debug msg="event published" ns=k8s.io topic="/images/update" type=containerd.services.images.v1.ImageUpdate
Jul 03 22:40:18 containerd[30986]: time="2019-07-03T22:40:18.457312091-07:00" level=debug msg="Received containerd event timestamp - 2019-07-04 05:40:18.454199554 +0000 UTC, namespace - "k8s.io", topic - "/images/update""
Jul 03 22:40:18 containerd[30986]: time="2019-07-03T22:40:18.457334235-07:00" level=info msg="ImageUpdate event &ImageUpdate{Name:sha256:af280c8b0a6945cca90755d2931c849e525c6aca1212ee934191bf9046352786,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Jul 03 22:40:18 containerd[30986]: time="2019-07-03T22:4
6419
0:18.458512744-07:00" level=debug msg="Received containerd event timestamp - 2019-07-04 05:40:18.456038532 +0000 UTC, namespace - "k8s.io", topic - "/images/update""
Jul 03 22:40:18 containerd[30986]: time="2019-07-03T22:40:18.463379368-07:00" level=debug msg="create image" name="sha256:af280c8b0a6945cca90755d2931c849e525c6aca1212ee934191bf9046352786" target=sha256:22e1171a33b41af2ef71270c57092dffb8fb2b0148f72fa9082e26d44f59e223
Jul 03 22:40:18 containerd[30986]: time="2019-07-03T22:40:18.463972653-07:00" level=debug msg="event published" ns=k8s.io topic="/images/update" type=containerd.services.images.v1.ImageUpdate
Jul 03 22:40:18 containerd[30986]: time="2019-07-03T22:40:18.464111622-07:00" level=debug msg="Received containerd event timestamp - 2019-07-04 05:40:18.463966691 +0000 UTC, namespace - "k8s.io", topic - "/images/update""
Jul 03 22:40:18 containerd[30986]: time="2019-07-03T22:40:18.464134535-07:00" level=info msg="ImageUpdate event &ImageUpdate{Name:sha256:af280c8b0a6945cca90755d2931c849e525c6aca1212ee934191bf9046352786,Labels:map[string]string{io.cri-containerd.image: managed,},}"
Jul 03 22:40:18 containerd[30986]: time="2019-07-03T22:40:18.467466380-07:00" level=debug msg="event published" ns=k8s.io topic="/images/update" type=containerd.services.images.v1.ImageUpdate
Jul 03 22:40:18 containerd[30986]: time="2019-07-03T22:40:18.468823401-07:00" level=debug msg="Received containerd event timestamp - 2019-07-04 05:40:18.467461016 +0000 UTC, namespace - "k8s.io", topic - "/images/update""
Jul 03 22:40:18 containerd[30986]: time="2019-07-03T22:40:18.472973421-07:00" level=info msg="CreateContainer within sandbox "5a84fe1bd02af2696a86354aaa68c8ee44012eb9f377dae922ecdd0e69630203" for container &ContainerMetadata{Name:imagetestpod,Attempt:0,}"
Jul 03 22:40:18 containerd[30986]: time="2019-07-03T22:40:18.472973421-07:00" level=info msg="CreateContainer within sandbox "5a84fe1bd02af2696a86354aaa68c8ee44012eb9f377dae922ecdd0e69630203" for container &ContainerMetadata{Name:imagetestpod,Attempt:0,}"
Jul 03 22:40:18 containerd[30986]: time="2019-07-03T22:40:18.473258583-07:00" level=debug msg="Container config &ContainerConfig{Metadata:&ContainerMetadata{Name:imagetestpod,Attempt:0,},Image:&ImageSpec{Image:sha256:af280c8b0a6945cca90755d2931c849e525c6aca1212ee934191bf9046352786,},Command:[sleep 1000],Args:[],WorkingDir:,Envs:[&KeyValue{Key:PEER_REVIEW_BOT_PORT_80_TCP_ADDR,Value:192.168.232.109,} &KeyValue{Key:CIMASTER_SERVICE_PORT,Value:80,} *"
Jul 03 22:40:18 containerd[30986]: time="2019-07-03T22:40:18.487511605-07:00" level=debug msg="Generated id "0e244328e924d69cb1092bc216220b943a22e52079cba771b22db5ffb29cd81d" for container "imagetestpod_imagetestpod-5b498d997f-rnnf8_ci_26735643-9e1e-11e9-a793-74dbd1805c29_0""
Jul 03 22:40:18 containerd[30986]: time="2019-07-03T22:40:18.488851362-07:00" level=debug msg="Container "0e244328e924d69cb1092bc216220b943a22e52079cba771b22db5ffb29cd81d" spec: (*specs.Spec)(0xc0020b9880){Version:(string)1.0.1-d
Jul 03 22:40:18 containerd[30986]: time="2019-07-03T22:40:18.491375571-07:00" level=error msg="CreateContainer within sandbox "5a84fe1bd02af2696a86354aaa68c8ee44012eb9f377dae922ecdd0e69630203" for &ContainerMetadata{Name:imagetestpod,Attempt:0,} failed" error="failed to create containerd container: error unpacking image: content digest sha256:1657edafda9d973766b6ba5a748da3c957023ee682636a918b28a2460e3278eb: not found"