iottymux: Store logs for kubelet in the appropriate location #3798

nhlfr · 2017-09-18T15:13:49Z

This is the change made for rktlet. Kubelet expect the CRI-formatted
logs to be in specific directory and it provides that information with
CRI.

CRI-compatible logs are generated by iottymux. That specific directory
which kubelet is looking at, exists in the host filesystem. In order to
make it available for iottymux, we need to mount it into the stage1
container. That's why this commit introduces the new annotation called
coreos.com/rkt/experiment/kubernetes-log-dir. This annotation is read
by stage1 init and the directory by it is bind mounted to the stage1
nspawn container (the mountpoint inside container is /rkt/kubernetes/log).

Kubelet provides the name of a logfile of a container via CRI. Since the
container concept in Kubernetes refers to an app in rkt, we use the
annotation named coreos.com/rkt/experiment/kubernetes-log-path to pass it
from rktlet. Then stage1 init redirects that value in KUBERNETES_LOG_PATH
environment variable for iottymux.

Then finally, iottymux saves the logs to /rkt/kubernetes/log/$KUBERNETES_LOG_PATH
file, i.e. /rkt/kubernetes/log/0a8f619b-df34-4c57-86e5-84a551452db1/etcd_0.log.

Fixes #3795

iaguis

Some comments.

iaguis · 2017-09-19T10:13:33Z

rkt/run.go

@@ -155,6 +156,7 @@ func init() {
 	cmdRun.Flags().StringVar(&flagHostname, "hostname", "", `pod's hostname. If empty, it will be "rkt-$PODUUID"`)
 	cmdRun.Flags().Var((*appsVolume)(&rktApps), "volume", "volumes to make available in the pod")
 	cmdRun.Flags().StringVar(&flagIPCMode, "ipc", "", `whether to stay in the host IPC namespace. Syntax: --ipc=[auto|private|parent]`)
+	cmdRun.Flags().StringVar(&flagKubeletLogDir, "kubelet-log-dir", "", "path to kubelet log dir")


Why is this flag in rkt run? It should be for rkt app sandbox, right?

This is a function addAppFlags which is used for almost every app command, and for sure it's used in rkt app add:

https://github.com/rkt/rkt/blob/master/rkt/app_add.go#L41

iaguis · 2017-09-19T10:14:30Z

stage1/iottymux/iottymux.go

@@ -411,7 +411,10 @@ func actionIOMux(statusFile string) error {
 	case "k8s-plain":
 		var err error
 		// TODO(lucab): check what should be the target path with Euan
-		logTarget := filepath.Join(pathPrefix, appName, "logfile")
+		logDir := os.Getenv("KUBELET_LOG_DIR")
+		kubernetesIndex := os.Getenv("KUBERNETES_INDEX")


If this is a per-app identifier, how can it work with only one variable?

lucab · 2017-09-20T12:33:03Z

rkt/app_sandbox.go

+	flagAppPorts      appPortList
+	flagAnnotations   kvMap
+	flagLabels        kvMap
+	flagKubeletLogDir string


Nit: here and everywhere else please s/kubelet/kubernetes/

lucab · 2017-09-20T12:40:18Z

rkt/app_sandbox.go

@@ -62,6 +63,8 @@ func init() {
 	flagAppPorts = appPortList{}
 	cmdAppSandbox.Flags().Var(&flagAnnotations, "user-annotation", "optional, set the pod's annotations in the form of key=value")
 	cmdAppSandbox.Flags().Var(&flagLabels, "user-label", "optional, set the pod's label in the form of key=value")
+
+	cmdAppSandbox.Flags().StringVar(&flagKubeletLogDir, "kubelet-log-dir", "", "Optional, path to kubelet log dir. Should be used only by rktlet.")


Can we have a default matching the kubernetes one here?

Not really, because when this string is non-empty, we bind mount it to the stage1 container. We don't want to do it when we don't use rktlet.

lucab · 2017-09-20T13:02:15Z

stage1/init/common/units.go

+				}
+				file.WriteString(fmt.Sprintf("KUBELET_LOG_DIR=%s\n", p.KubeletLogDir))
+
+				kubeletLogPath, ok := ra.Annotations.Get("coreos.com/rkt/experiment/kubelet-log-path")


This name is a bit confusing, as it isn't a proper path, just a suffix/name (according to my understanding of your explanation yesterday). What about kubernetes-log-filename or something similar?

lucab · 2017-09-20T13:05:17Z

stage1/iottymux/iottymux.go

-		logFile, err = os.OpenFile(logTarget, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, os.ModePerm)
+		logDir := os.Getenv("KUBELET_LOG_DIR")
+		logPath := os.Getenv("KUBELET_LOG_PATH")
+		logFullPath := filepath.Join(logDir, logPath)


This just works because there is an implicit invariant that "kubelet logpath on host == logpath in stage1". This should be either documented somewhere, or the logDir part should be dropped from here and scoped instead somewhere under /rkt or similar (probably better).

/cc @alban

I would prefer the latter, to choose a fixed path inside stage1 so that the kubelet will not be able to overshadow some other file/directory by mistake.

Then, --bind= in stage1/init/init.go must be updated accordingly using 2 paths.

nhlfr · 2017-09-21T17:40:40Z

Way to test that change:

$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app sandbox --kubelet-log-dir=/home/vador/playground/rkt --stage1-path=/home/vador/go/src/github.com/rkt/rkt/build-rkt/target/bin/stage1-coreos.aci --insecure-options=image --user-annotation=coreos.com/rkt/experiment/logmode=k8s-plain
$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt list
UUID		APP	IMAGE NAME	STATE	CREATED		STARTED		NETWORKS
5a878340	-	-		running	27 seconds ago	27 seconds ago	default:ip4=172.16.28.2
$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app add 5a878340 coreos.com/etcd:v2.0.0 --name=etcd --user-annotation=coreos.com/rkt/experiment/kubelet-log-path=etcd_0.log --stdin stream --stdout stream --stderr stream
$ cat /home/vador/playground/rkt/etcd_0.log 
2017-09-21T19:35:55.730366351+02:00 stderr 2017/09/21 17:35:55 no data-dir provided, using default data-dir ./default.etcd
2017-09-21T19:35:55.730747783+02:00 stderr 2017/09/21 17:35:55 etcd: listening for peers on http://localhost:2380
2017-09-21T19:35:55.730767547+02:00 stderr 2017/09/21 17:35:55 etcd: listening for peers on http://localhost:7001
2017-09-21T19:35:55.730839161+02:00 stderr 2017/09/21 17:35:55 etcd: listening for client requests on http://localhost:2379
2017-09-21T19:35:55.730842466+02:00 stderr 2017/09/21 17:35:55 etcd: listening for client requests on http://localhost:4001
2017-09-21T19:35:55.730924571+02:00 stderr 2017/09/21 17:35:55 etcdserver: name = default
2017-09-21T19:35:55.730929816+02:00 stderr 2017/09/21 17:35:55 etcdserver: data dir = default.etcd
2017-09-21T19:35:55.73093377+02:00 stderr 2017/09/21 17:35:55 etcdserver: heartbeat = 100ms
2017-09-21T19:35:55.730936372+02:00 stderr 2017/09/21 17:35:55 etcdserver: election = 1000ms
2017-09-21T19:35:55.73093996+02:00 stderr 2017/09/21 17:35:55 etcdserver: snapshot count = 10000
2017-09-21T19:35:55.730942201+02:00 stderr 2017/09/21 17:35:55 etcdserver: advertise client URLs = http://localhost:2379,http://localhost:4001
2017-09-21T19:35:55.730949459+02:00 stderr 2017/09/21 17:35:55 etcdserver: initial advertise peer URLs = http://localhost:2380,http://localhost:7001
2017-09-21T19:35:55.730954968+02:00 stderr 2017/09/21 17:35:55 etcdserver: initial cluster = default=http://localhost:2380,default=http://localhost:7001
2017-09-21T19:35:55.736569397+02:00 stderr 2017/09/21 17:35:55 etcdserver: start member ce2a822cea30bfca in cluster 7e27652122e8b2ae
2017-09-21T19:35:55.736597943+02:00 stderr 2017/09/21 17:35:55 raft: ce2a822cea30bfca became follower at term 0
2017-09-21T19:35:55.736610637+02:00 stderr 2017/09/21 17:35:55 raft: newRaft ce2a822cea30bfca [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2017-09-21T19:35:55.736621077+02:00 stderr 2017/09/21 17:35:55 raft: ce2a822cea30bfca became follower at term 1
2017-09-21T19:3
8000
5:55.741344981+02:00 stderr 2017/09/21 17:35:55 etcdserver: added local member ce2a822cea30bfca [http://localhost:2380 http://localhost:7001] to cluster 7e27652122e8b2ae
2017-09-21T19:35:57.037299019+02:00 stderr 2017/09/21 17:35:57 raft: ce2a822cea30bfca is starting a new election at term 1
2017-09-21T19:35:57.03762314+02:00 stderr 2017/09/21 17:35:57 raft: ce2a822cea30bfca became candidate at term 2
2017-09-21T19:35:57.037784912+02:00 stderr 2017/09/21 17:35:57 raft: ce2a822cea30bfca received vote from ce2a822cea30bfca at term 2
2017-09-21T19:35:57.037975485+02:00 stderr 2017/09/21 17:35:57 raft: ce2a822cea30bfca became leader at term 2
2017-09-21T19:35:57.038018404+02:00 stderr 2017/09/21 17:35:57 raft.node: ce2a822cea30bfca elected leader ce2a822cea30bfca at term 2
2017-09-21T19:35:57.04330613+02:00 stderr 2017/09/21 17:35:57 etcdserver: published {Name:default ClientURLs:[http://localhost:2379 http://localhost:4001]} to cluster 7e27652122e8b2ae

nhlfr · 2017-09-22T12:37:29Z

@alban @iaguis @lucab PTAL

iaguis · 2017-09-22T13:31:19Z

This works:

$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app sandbox --kubernetes-log-dir=/home/iaguis/temp/rkt --user-annotation=coreos.com/rkt/experiment/logmode=k8s-plain
...
$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app add 7d9a73d9 coreos.com/etcd:v2.0.10 --name=etcd --user-annotation=coreos.com/rkt/experiment/kubernetes-log-path=etcd_0.log --stdin stream --stdout stream --stderr stream
$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app start 7d9a73d9 --app=etcd

Now if I do:

$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app add 7d9a73d9 coreos.com/etcd:v2.0.0 --name=etcd20 --user-annotation=coreos.com/rkt/experiment/kubernetes-log-path=etcd_0.log --stdin stream --stdout stream --stderr stream
$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app start 7d9a73d9 --app=etcd20

I get this on the sandbox:

[16720.759217] etcd20[30]: panic: send on closed channel
[16720.760165] etcd20[30]: goroutine 11 [running]:
[16720.761325] etcd20[30]: main.bufferLine(0xa087c0, 0xc42000e0b8, 0xc42006a9c0, 0xc42006a720)
[16720.761985] etcd20[30]: 	/home/iaguis/work/go/src/github.com/rkt/rkt/build-rkt/gopath/src/github.com/rkt/rkt/stage1/iottymux/iottymux.go:491 +0x10d
[16720.762308] etcd20[30]: created by main.actionIOMux
[16720.762552] etcd20[30]: 	/home/iaguis/work/go/src/github.com/rkt/rkt/build-rkt/gopath/src/github.com/rkt/rkt/stage1/iottymux/iottymux.go:447 +0x1014
[16720.767100] etcd20[30]: panic: send on closed channel
[16720.767792] etcd20[30]: goroutine 8 [running]:
[16720.767992] etcd20[30]: main.bufferLine(0xa087c0, 0xc42000e0b0, 0xc42006a8a0, 0xc42006a720)
[16720.768152] etcd20[30]: 	/home/iaguis/work/go/src/github.com/rkt/rkt/build-rkt/gopath/src/github.com/rkt/rkt/stage1/iottymux/iottymux.go:491 +0x10d
[16720.768440] etcd20[30]: created by main.actionIOMux
[16720.768607] etcd20[30]: 	/home/iaguis/work/go/src/github.com/rkt/rkt/build-rkt/gopath/src/github.com/rkt/rkt/stage1/iottymux/iottymux.go:434 +0x11c9

Maybe because the log file has the same name? If I remove the app and add it again with a different log file:

$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app rm 7d9a73d9 --app=etcd20
$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app add 7d9a73d9 coreos.com/etcd:v2.0.0 --name=etcd20 --user-annotation=coreos.com/rkt/experiment/kubernetes-log-path=etcd_1.log --stdin stream --stdout stream --stderr stream
$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app start 7d9a73d9 --app=etcd20

I see this now:

[16828.630846] etcd20[54]: iottymux: runtime failure: unable to open stdin listener: listen unix /rkt/iottymux/etcd20/sock-stdin: bind: address already in use

Doing the same but with different log files seems to work 👍

$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app sandbox --kubernetes-log-dir=/home/iaguis/temp/rkt --user-annotation=coreos.com/rkt/experiment/logmode=k8s-plain
...
$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app add 4bb103be coreos.com/etcd:v2.0.10 --name=etcd --user-annotation=coreos.com/rkt/experiment/kubernetes-log-path=etcd_0.log --stdin stream --stdout stream --stderr stream
$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app start 4bb103be --app=etcd
$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app add 4bb103be coreos.com/etcd:v2.0.0 --name=etcd20 --user-annotation=coreos.com/rkt/experiment/kubernetes-log-path=etcd_1.log --stdin stream --stdout stream --stderr stream
$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app start 4bb103be --app=etcd20
$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app add 4bb103be kinvolk.io/aci/busybox --name=busybox --user-annotation=coreos.com/rkt/experiment/kubernetes-log-path=busybox_0.log --stdin stream --stdout stream --stderr stream -- -c 'while true; do date; sleep 1; done'
$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app start 4bb103be --app=busybox

$ ls ~/temp/rkt
busybox_0.log  etcd_0.log     etcd_1.log 
$ cat ~/temp/rkt/busybox_0.log 
2017-09-22T15:27:47.281125037+02:00 stdout Fri Sep 22 13:27:47 UTC 2017
2017-09-22T15:27:48.283188723+02:00 stdout Fri Sep 22 13:27:48 UTC 2017
2017-09-22T15:27:49.285577388+02:00 stdout Fri Sep 22 13:27:49 UTC 2017
2017-09-22T15:27:50.287567238+02:00 stdout Fri Sep 22 13:27:50 UTC 2017
2017-09-22T15:27:51.290272537+02:00 stdout Fri Sep 22 13:27:51 UTC 2017
2017-09-22T15:27:52.292623861+02:00 stdout Fri Sep 22 13:27:52 UTC 2017
2017-09-22T15:27:53.294853812+02:00 stdout Fri Sep 22 13:27:53 UTC 2017

lucab · 2017-09-22T13:21:36Z

stage1/init/common/units.go

@@ -320,10 +320,20 @@ func (uw *UnitWriter) SetupAppIO(p *stage1commontypes.Pod, ra *schema.RuntimeApp
 			// streaming mode brings in a `iomux@.service` before-dependency
 			opts = append(opts, unit.NewUnitOption("Unit", "Requires", fmt.Sprintf("iomux@%s.service", ra.Name)))
 			opts = append(opts, unit.NewUnitOption("Unit", "Before", fmt.Sprintf("iomux@%s.service", ra.Name)))
-			logMode, ok := p.Manifest.Annotations.Get("coreos.com/rkt/experiment/logmode")
+			logMode, ok := p.Manifest.UserAnnotations["coreos.com/rkt/experiment/logmode"]


I think this was correct before as an annotation, not a user-annotation. Same for the one you added below. See https://github.com/appc/spec/blob/master/spec/pods.md.

I think the concern with this is how to pass a pod annotation in rktlet if we use the current rkt CLI. Since we're not using a pod manifest, we'd need to add an option to pass annotations to rkt run and rkt app sandbox

Exaclty, using non-user annotations is just impossible.

I'm not sure I'm following, I'm just commenting on stage1 logic here.
It shouldn't matter how you get this information from the user in stage0 (CLI flag, env flag, hardcoded value), you can still pass it as an annotation to stage1.
Unless you are trying to tell me that you want to pass this to stage0 CLI as --user-annotation=, in which case I'm saying that you shouldn't (because of spec, and because it will appear back in k8s among user-facing annotations).

lucab · 2017-09-22T13:28:43Z

stage1/init/init.go

@@ -135,6 +135,7 @@ func parseFlags() *stage1commontypes.RuntimePod {
 	})
 	flag.Var(dnsConfMode, "dns-conf-mode", "DNS config file modes")
 	flag.StringVar(&rp.IPCMode, "ipc", "", "IPC mode --ipc=[auto|private|parent]")
+	flag.StringVar(&rp.KubernetesLogDir, "kuber
A373
netes-log-dir", "", "Mount the kubernetes log directory to write CRI logs")


This is a bit heavy and needs to be put in the stage1 interface protocol. Can't we instead just generate a volume+mount pair in stage0 at app_add time? If not, then an annotation on the pod is probably fine for the time being.

lucab · 2017-09-22T13:30:03Z

stage1/iottymux/iottymux.go

-		logTarget := filepath.Join(pathPrefix, appName, "logfile")
-		logFile, err = os.OpenFile(logTarget, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, os.ModePerm)
+		logPath := os.Getenv("KUBERNETES_LOG_PATH")
+		logFullPath := filepath.Join("/rkt/kubernetes", logPath)


As this is coming from the outside, I suggest to Clean it before use and make sure the result is still within the prefix dir after.

nhlfr · 2017-09-22T13:54:16Z

@iaguis This is expected, you cannot log to the same file from two apps. And kubelet generates the another filename for each container, so that's totally fine. Or you mean that we should check whether the file isn't used already before doing anything? That could be cool in terms of UX.

iaguis · 2017-09-22T13:55:53Z

Or you mean that we should check whether the file isn't used already before doing anything? That could be cool in terms of UX.

If it's not too difficult to implement I rather see errors than panics :)

nhlfr · 2017-09-22T15:02:00Z

Yeah, can do that. I will just check whether the file exists in the dir we mount from host.

iaguis · 2017-09-27T10:34:35Z

#3814 adds the --annotation flag to be able to pass annotations to stage1. This PR needs to be updated when that's merged.

nhlfr · 2017-10-05T14:47:43Z

Since we changed the implementation a little bit and now everything relies on annotations (instead of user annotations or new CLI arguments), the current way of testing that branch is:

$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app sandbox --annotation=coreos.com/rkt/experiment/logmode=k8s-plain --annotation=coreos.com/rkt/experiment/kubernetes-log-dir=/home/vador/playground/rkt
$ sudo rkt list
UUID		APP	IMAGE NAME	STATE	CREATED		STARTED		NETWORKS
fc392781	-	-		running	3 seconds ago	3 seconds ago	default:ip4=172.16.28.2
$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app add fc392781 coreos.com/etcd:v2.0.0 --name=etcd --annotation=coreos.com/rkt/experiment/kubernetes-log-path=etcd_0.log --stdin=stream --stdout=stream --stderr=stream
$ sudo RKT_EXPERIMENT_APP=true RKT_EXPERIMENT_ATTACH=true rkt app start fc392781 --app=etcd
$ cat /home/vador/playground/rkt/etcd_0.log 
2017-10-05T15:29:44.556024467+02:00 stderr 2017/10/05 13:29:44 no data-dir provided, using default data-dir ./default.etcd
2017-10-05T15:29:44.558498474+02:00 stderr 2017/10/05 13:29:44 etcd: listening for peers on http://localhost:2380
2017-10-05T15:29:44.558641344+02:00 stderr 2017/10/05 13:29:44 etcd: listening for peers on http://localhost:7001
2017-10-05T15:29:44.55866307+02:00 stderr 2017/10/05 13:29:44 etcd: listening for client requests on http://localhost:2379
2017-10-05T15:29:44.55883772+02:00 stderr 2017/10/05 13:29:44 etcd: listening for client requests on http://localhost:4001
2017-10-05T15:29:44.558884614+02:00 stderr 2017/10/05 13:29:44 etcdserver: name = default
2017-10-05T15:29:44.558889978+02:00 stderr 2017/10/05 13:29:44 etcdserver: data dir = default.etcd
2017-10-05T15:29:44.558902749+02:00 stderr 2017/10/05 13:29:44 etcdserver: heartbeat = 100ms
2017-10-05T15:29:44.558905701+02:00 stderr 2017/10/05 13:29:44 etcdserver: election = 1000ms
2017-10-05T15:29:44.558914016+02:00 stderr 2017/10/05 13:29:44 etcdserver: snapshot count = 10000
2017-10-05T15:29:44.558917099+02:00 stderr 2017/10/05 13:29:44 etcdserver: advertise client URLs = http://localhost:2379,http://localhost:4001
2017-10-05T15:29:44.558925815+02:00 stderr 2017/10/05 13:29:44 etcdserver: initial advertise peer URLs = http://localhost:2380,http://localhost:7001
2017-10-05T15:29:44.558933524+02:00 stderr 2017/10/05 13:29:44 etcdserver: initial cluster = default=http://localhost:2380,default=http://localhost:7001
2017-10-05T15:29:44.564150576+02:00 stderr 2017/10/05 13:29:44 etcdserver: start member ce2a822cea30bfca in cluster 7e27652122e8b2ae
2017-10-05T15:29:44.564364362+02:00 stderr 2017/10/05 13:29:44 raft: ce2a822cea30bfca became follower at term 0
2017-10-05T15:29:44.564378005+02:00 stderr 2017/10/05 13:29:44 raft: newRaft ce2a822cea30bfca [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2017-10-05T15:29:44.564496681+02:00 stderr 2017/10/05 13:29:44 raft: ce2a822cea30bfca became follower at term 1
2017-10-05T15:29:44.57064913+02:00 stderr 2017/10/05 13:29:44 etcdserver: added local member ce2a822cea30bfca [http://localhost:2380 http://localhost:7001] to cluster 7e27652122e8b2ae
2017-10-05T15:29:45.864876947+02:00 stderr 2017/10/05 13:29:45 raft: ce2a822cea30bfca is starting a new election at term 1
2017-10-05T15:29:45.864963867+02:00 stderr 2017/10/05 13:29:45 raft: ce2a822cea30bfca became candidate at term 2
2017-10-05T15:29:45.865349694+02:00 stderr 2017/10/05 13:29:45 raft: ce2a822cea30bfca received vote from ce2a822cea30bfca at term 2
2017-10-05T15:29:45.865415126+02:00 stderr 2017/10/05 13:29:45 raft: ce2a822cea30bfca became leader at term 2
2017-10-05T15:29:45.865547038+02:00 stderr 2017/10/05 13:29:45 raft.node: ce2a822cea30bfca elected leader ce2a822cea30bfca at term 2
2017-10-05T15:29:45.870924952+02:00 stderr 2017/10/05 13:29:45 etcdserver: published {Name:default ClientURLs:[http://localhost:2379 http://localhost:4001]} to cluster 7e27652122e8b2ae

nhlfr · 2017-10-05T14:47:55Z

@lucab @iaguis PTAL

iaguis

Some small changes. Otherwise looks good.

iaguis · 2017-10-05T15:18:26Z

stage1/iottymux/iottymux.go

+
+		match, err := filepath.Match("/rkt/kubernetes/*", logFullPath)
+		if err != nil {
+			return fmt.Errorf("Couldn't analyze the full log path %s: %s", logFullPath, err)


Errors should be lower case

iaguis · 2017-10-05T15:18:29Z

stage1/iottymux/iottymux.go

+		if err != nil {
+			return fmt.Errorf("Couldn't analyze the full log path %s: %s", logFullPath, err)
+		} else if !match {
+			return fmt.Errorf("The log path isn't inside /rkt/kubernetes, which most probably means that user tried to escape that directory by using '..'")


Errors should be lower case

iaguis · 2017-10-05T15:20:03Z

stage1/iottymux/iottymux.go

+		if err != nil {
+			return fmt.Errorf("Couldn't analyze the full log path %s: %s", logFullPath, err)
+		} else if !match {
+			return fmt.Errorf("The log path isn't inside /rkt/kubernetes, which most probably means that user tried to escape that directory by using '..'")


I'd just say:

log path is not inside /rkt/kubernetes, refusing path traversal

iaguis · 2017-10-05T15:33:44Z

tests/rkt_app_sandbox_test.go

+const (
+	// Number of retries to read CRI log file.
+	criLogsReadRetries = 5
+	// Delay between each retry attemtp in reading CRI log file.


iaguis · 2017-10-05T15:44:21Z

stage1/init/common/units.go

@@ -328,6 +328,16 @@ func (uw *UnitWriter) SetupAppIO(p *stage1commontypes.Pod, ra *schema.RuntimeApp
 			if ok {
 				file.WriteString(fmt.Sprintf("STAGE1_LOGMODE=%s\n", logMode))
 			}
+			switch logMode {
+			case "k8s-plain":
+				kubernetesLogPath, ok := ra.Annotations.Get("coreos.com/rkt/experiment/kubernetes-log-path")


Can we make this kubernetes-log-filename

log path is the term that CRI and kubelet use and I just wanted to be consistent. But I don't care if you seriously care about it. ;)

Right. In the CRI they use LogDirectory and LogPath, which I find confusing. Since these annotations are k8s-specific I guess staying consistent with their names is ok. Disregard the comment then.

iaguis · 2017-10-05T15:44:31Z

stage1/init/common/units.go

+					uw.err = fmt.Errorf("kubernetes-log-path annotation needs to be specified when k8s-plain logging mode is used")
+					return nil
+				}
+				file.WriteString(fmt.Sprintf("KUBERNETES_LOG_PATH=%s\n", kubernetesLogPath))


Here too, KUBERNETES_LOG_FILENAME

iaguis · 2017-10-05T15:44:57Z

stage1/iottymux/iottymux.go

-		// TODO(lucab): check what should be the target path with Euan
-		logTarget := filepath.Join(pathPrefix, appName, "logfile")
-		logFile, err = os.OpenFile(logTarget, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, os.ModePerm)
+		logPath := os.Getenv("KUBERNETES_LOG_PATH")


Then this would be logFilename

iaguis · 2017-10-05T15:45:04Z

stage1/iottymux/iottymux.go

-		logTarget := filepath.Join(pathPrefix, appName, "logfile")
-		logFile, err = os.OpenFile(logTarget, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, os.ModePerm)
+		logPath := os.Getenv("KUBERNETES_LOG_PATH")
+		logFullPath := filepath.Clean(filepath.Join("/rkt/kubernetes", logPath))


And this logPath

iaguis · 2017-10-05T15:49:22Z

stage1/init/init.go

@@ -362,6 +362,11 @@ func getArgsEnv(p *stage1commontypes.Pod, flavor string, canMachinedRegister boo
 			args = append(args, fmt.Sprintf("--register=false"))
 		}

+		kubernetesLogDir, ok := p.Manifest.Annotations.Get("coreos.com/rkt/experiment/kubernetes-log-dir")
+		if ok {
+			args = append(args, fmt.Sprintf("--bind=%s:/rkt/kubernetes", kubernetesLogDir))


/rkt/kubernetes seems too broad, maybe we can make it /rkt/kubernetes/log or such

iaguis · 2017-10-05T16:02:58Z

stage1/iottymux/iottymux.go

+			return fmt.Errorf("The log path isn't inside /rkt/kubernetes, which most probably means that user tried to escape that directory by using '..'")
+		}
+
+		logFile, err = os.OpenFile(logFullPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, os.ModePerm)


It seems that according to the CRI KUBERNETES_LOG_PATH can be a real path and not just a file name so this would fail if it turns out to be more than a file name.

We can just add a comment mentioning this because I don't think it's an issue for now.

Hmm... I don't think so, logPath is always a file name. I didn't see any code in kubelet or fragment of spec which proves otherwise. The name is just confusing. But yes, I can add a comment.

iaguis

Hm, actually can you re-work the commit message, it doesn't seem to reflect the contents of the commit anymore.

nhlfr · 2017-10-06T09:29:50Z

@iaguis done

iaguis · 2017-10-06T09:49:49Z

Thanks!

iaguis · 2017-10-06T11:57:53Z

Tests are failing:

    rkt_app_sandbox_test.go:613: Expected CRI logs to contain 'stdout HelloFromAppInSandbox', instead got:

Probably because in the test ioutil.ReadFile() succeeds but returns nothing. You could continue in the loop if the content is empty.

nhlfr · 2017-10-06T13:08:35Z

@iaguis checking whether the log file contains the actual output is important, we cannot just accept empty logs.

I assume that rkt is just working slightly slower in semaphoreci and we need to add some recheks and timeouts.

iaguis · 2017-10-06T13:19:01Z

Yes, I mean doing that in the test so we give some more time for the logs to appear

…

On Oct 6, 2017 15:08, "Michal Rostecki" ***@***.***> wrote: @iaguis <https://github.com/iaguis> checking whether the log file contains the actual output is important, we cannot just accept empty logs. I assume that rkt is just working slightly slower in semaphoreci and we need to add some recheks and timeouts. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3798 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAe92gT8Cjpw_sY_xzyxwFOJOAXLIzJBks5spiZWgaJpZM4PbESX> .

This is the change made for rktlet. Kubelet expect the CRI-formatted logs to be in specific directory and it provides that information with CRI. CRI-compatible logs are generated by iottymux. That specific directory which kubelet is looking at, exists in the host filesystem. In order to make it available for iottymux, we need to mount it into the stage1 container. That's why this commit introduces the new annotation called coreos.com/rkt/experiment/kubernetes-log-dir. This annotation is read by stage1 init and the directory by it is bind mounted to the stage1 nspawn container (the mountpoint inside container is /rkt/kubernetes/log). Kubelet provides the name of a logfile of a container via CRI. Since the container concept in Kubernetes refers to an app in rkt, we use the annotation named coreos.com/rkt/experiment/kubernetes-log-path to pass it from rktlet. Then stage1 init redirects that value in KUBERNETES_LOG_PATH environment variable for iottymux. Then finally, iottymux saves the logs to /rkt/kubernetes/log/$KUBERNETES_LOG_PATH file, i.e. /rkt/kubernetes/log/0a8f619b-df34-4c57-86e5-84a551452db1/etcd_0.log. Fixes rkt#3795

rktlet was using journal2cri binary for converting journald logs to CRI format. Container with that binary was injected into each pod. Unfortunately, it couldn't disginguish stdout and stderr output. It also wasn't very stable and reliable. Currently rkt is able to store logs in CRI format in provided directory[1]. Therefore we can remove journal2cri completely and just point rkt to the directory which CRI provides. Fixes kubernetes-retired#107 [1] rkt/rkt#3798

rktlet was using journal2cri binary for converting journald logs to CRI form 72A8 at. Container with that binary was injected into each pod. Unfortunately, it couldn't disginguish stdout and stderr output. It also wasn't very stable and reliable. Currently rkt is able to store logs in CRI format in provided directory[1]. Therefore we can remove journal2cri completely and just point rkt to the directory which CRI provides. Fixes kubernetes-retired#107 [1] rkt/rkt#3798

rktlet was using journal2cri binary for converting journald logs to CRI format. Container with that binary was injected into each pod. Unfortunately, it couldn't disginguish stdout and stderr output. It also wasn't very stable and reliable. Currently rkt is able to store logs in CRI format in provided directory[1]. Therefore we can remove journal2cri completely and just point rkt to the directory which CRI provides. Fixes kubernetes-retired#107 [1] rkt/rkt#3798

nhlfr force-pushed the nhlfr/expose-cri-logs-to-kubelet branch 2 times, most recently from cc86f34 to 972fa62 Compare September 19, 2017 06:36

iaguis reviewed Sep 19, 2017

View reviewed changes

iaguis added the area/cri label Sep 20, 2017

nhlfr force-pushed the nhlfr/expose-cri-logs-to-kubelet branch from 972fa62 to 99890d4 Compare September 20, 2017 11:21

lucab reviewed Sep 20, 2017

View reviewed changes

nhlfr force-pushed the nhlfr/expose-cri-logs-to-kubelet branch from 99890d4 to faa2b64 Compare September 21, 2017 17:37

nhlfr changed the title ~~[WIP] iottymux: Store logs for kubelet in the appropriate location~~ iottymux: Store logs for kubelet in the appropriate location Sep 21, 2017

nhlfr force-pushed the nhlfr/expose-cri-logs-to-kubelet branch 2 times, most recently from 1e30ae2 to 08ba149 Compare September 22, 2017 12:35

lucab reviewed Sep 22, 2017

View reviewed changes

nhlfr mentioned this pull request Sep 22, 2017

rkt run, add sandbox and add should be able to set annotations #3811

Closed

nhlfr force-pushed the nhlfr/expose-cri-logs-to-kubelet branch from 08ba149 to bb434bc Compare September 28, 2017 10:45

nhlfr changed the title ~~iottymux: Store logs for kubelet in the appropriate location~~ [WIP] iottymux: Store logs for kubelet in the appropriate location Sep 28, 2017

nhlfr force-pushed the nhlfr/expose-cri-logs-to-kubelet branch 2 times, most recently from d440d6d to 26a3e25 Compare October 5, 2017 14:20

nhlfr changed the title ~~[WIP] iottymux: Store logs for kubelet in the appropriate location~~ iottymux: Store logs for kubelet in the appropriate location Oct 5, 2017

nhlfr force-pushed the nhlfr/expose-cri-logs-to-kubelet branch from 26a3e25 to 75531ff Compare October 5, 2017 14:42

iaguis suggested changes Oct 5, 2017

View reviewed changes

nhlfr force-pushed the nhlfr/expose-cri-logs-to-kubelet branch from 75531ff to 690d6f5 Compare October 5, 2017 16:03

iaguis reviewed Oct 5, 2017

View reviewed changes

iaguis suggested changes Oct 5, 2017

View reviewed changes

nhlfr force-pushed the nhlfr/expose-cri-logs-to-kubelet branch from eb4b52f to 4aaedfe Compare October 6, 2017 09:20

nhlfr force-pushed the nhlfr/expose-cri-logs-to-kubelet branch from 4aaedfe to c302c34 Compare October 6, 2017 09:41

iaguis approved these changes Oct 6, 2017

View reviewed changes

iaguis added the reviewed/merge-on-green label Oct 6, 2017

nhlfr force-pushed the nhlfr/expose-cri-logs-to-kubelet branch 3 times, most recently from 37d1b4d to 64e165e Compare October 6, 2017 13:34

nhlfr force-pushed the nhlfr/expose-cri-logs-to-kubelet branch from 64e165e to b710725 Compare October 6, 2017 14:21

nhlfr mentioned this pull request Oct 6, 2017

Replace journal2cri with solution based on stdout/stderr stream kubernetes-retired/rktlet#147

Merged

iaguis merged commit d8550e5 into rkt:master Oct 9, 2017

samm81 mentioned this pull request Dec 11, 2017

iottymux: add support for json formatted logs akin to docker #3873

Open

iaguis added this to the 1.30.0 milestone Apr 11, 2018

iottymux: Store logs for kubelet in the appropriate location #3798

iottymux: Store logs for kubelet in the appropriate location #3798

Uh oh!

Conversation

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment