werf · distorhead · Oct 2, 2019 · Sep 13, 2019
diff --git a/.dockerignore b/.dockerignore
@@ -0,0 +1,2 @@
+.git
+**/.git
diff --git a/docs/_data/breadcrumbs.yml b/docs/_data/breadcrumbs.yml
@@ -43,3 +43,6 @@
 /documentation/cli/other:
   title: Other
   url: /documentation/cli/other/completion.html
+/documentation/development:
+  title: Development
+  url: /documentation/development/stapel.html
diff --git a/docs/_data/sidebars/documentation.yml b/docs/_data/sidebars/documentation.yml
@@ -308,3 +308,8 @@ entries:
 
             - title: version
               url: /documentation/cli/other/version.html
+
+    - title: Development
+      sf:
+        - title: Stapel
+          url: /documentation/development/stapel.html
diff --git a/docs/pages/development/stapel.md b/docs/pages/development/stapel.md
@@ -0,0 +1,64 @@
+---
+title: Stapel
+sidebar: documentation
+permalink: documentation/development/stapel.html
+author: Timofey Kirillov <timofey.kirillo@flant.com>
+---
+
+## Overview
+
+Stapel is an [LFS](http://www.linuxfromscratch.org/lfs/view/stable) based linux distribution, which contains:
+
+ * Glibc;
+ * Gnu cli tools (install, patch, find, wget, grep, rsync and other);
+ * Git cli util.
+ * Bash.
+ * Python interpreter and Ansible.
+
+Stapel tools built in non-standard root location. Binaries, libraries and other related files are located in the following dirs:
+
+ * `/.werf/stapel/(etc|lib|lib64|libexec)`;
+ * `/.werf/stapel/x86_64-lfs-linux-gnu/bin`;
+ * `/.werf/stapel/embedded/(bin|etc|lib|libexec|sbin|share|ssl)`.
+
+The base of stapel is a Glibc library and linker (`/.werf/stapel/lib/libc-VERSION.so` and `/.werf/stapel/x86_64-lfs-linux-gnu/bin/ld`). All tools from `/.werf/stapel` are compiled and linked only agains libraries contained in `/.werf/stapel`. So stapel is a self-contained distribution of tools and libraries without external dependencies with an independent Glibc, which allows running these tools in arbitrary environment (independently of linux distribution and libraries versions in this distribution).
+
+Stapel filesystem `/.werf/stapel` is intent to be mounted into build container based on some base image. Tools from stapel can then be used for some purposes. As stapel tools does not have external dependencies stapel image can be mounted into any base image (alpine linux with musl libc, or ubuntu with glibc — does not matter) and tools will work as expected.
+
+Werf mounts _stapel image_ into each build container when building docker images with _stapel builder_ to enable ansible, git service operations and for other service purposes. More info about _stapel builder_ are available [in the article]({{ site.baseurl }}/documentation/reference/build_process.html#stapel-image-and-artifact).
+
+## Rebuild stapel
+
+Stapel image needs to be updated time to time to update ansible or when new version of [LFS](http://www.linuxfromscratch.org/lfs/view/stable) is available.
+
+1. Make necessary changes to build instructions in `stapel` directory.
+2. Update omnibus bundle:
+    ```bash
+    cd stapel
+    bundle update
+    git add -p Gemfile Gemfile.lock
+    ```
+3. Pull previous stapel images cache:
+    ```bash
+    scripts/stapel/pull_cache.sh
+    ```
+4. Increment current version in `scripts/stapel/version.sh` environment variable `CURRENT_STAPEL_VERSION`.
+5. Build new stapel images:
+    ```bash
+    scripts/stapel/build.sh
+    ```
+6. Publish new stapel images:
+    ```bash
+    scripts/stapel/publish.sh
+    ```
+7. As new stapel version is done and published, then change `PREVIOUS_STAPEL_VERSION` environment variable in `scripts/stapel/version.sh` to the same value as `CURRENT_STAPEL_VERSION` and commit changes to the repo.
+
+## How build works
+
+Stapel image builder consists of 2 docker stages: base and final.
+
+Base stage is an ubuntu based image used to build LFS system and additional omnibus packages.
+
+Final stage is a pure scratch image which imports only necessary `/.werf/stapel` files.
+
+NOTE: Both base and final docker stages are being published during `scripts/stapel/publish.sh` procedure to enable cache during next stapel rebuild.
diff --git a/docs/pages/documentation.md b/docs/pages/documentation.md
@@ -49,3 +49,5 @@ Also, this section includes articles with base primitives and general tools:
 * [Toolbox]({{ site.baseurl }}/documentation/reference/toolbox/slug.html).
 
 Werf is a CLI utility, so if you want to find a description of both basic commands needed to provide the CI/CD process and service commands that provide advanced functionality — use **CLI Commands** section.
+
+[**Development** section]({{ site.baseurl }}/documentation/development/stapel.html) contains service and maintenance manuals and other docs which help developers to understand how some werf subsystem works, how to maintain some subsystem in the actual state, how to write and build new code for the werf, etc.
diff --git a/docs/pages/reference/build_process.md b/docs/pages/reference/build_process.md
@@ -47,14 +47,7 @@ All generated instructions to build current stage are supposed to be run in a co
 
 To build a stage werf runs prepared instructions list in the build container based on the previous stage. The resulting container state is then committed as a new stage and saved into the [stages storage]({{ site.baseurl }}/documentation/reference/stages_and_images.html#stages-storage).
 
-Werf has a special service image called `flant/werf-stapel` which contains a chroot `/.werf/stapel` with all tools and libraries needed to build images with stapel builder, such as:
- * Glibc.
- * Gnu cli tools (install, patch, find, wget, grep, rsync and other).
- * Git cli util.
- * Bash.
- * Python interpreter and ansible.
-
-All these tools are compiled with an independent glibc, which allows running these tools in arbitrary environment independently of used base image.
+Werf has a special service image called `flant/werf-stapel` which contains a chroot `/.werf/stapel` with all tools and libraries needed to build images with stapel builder. More info about stapel image are available [in the article]({{ site.baseurl }}/documentation/development/stapel.html).
 
 `flant/werf-stapel` is mounted into every build container so that all precompiled tools are available in every stage build and may be used in instructions list.
 

diff --git a/go.mod b/go.mod
@@ -36,7 +36,6 @@ require (
 	github.com/flant/go-containerregistry v0.0.0-20190712094650-0cfc503dc51a
 	github.com/flant/kubedog v0.3.5-0.20190923111717-5fda2f77f960
 	github.com/flant/logboek v0.2.6-0.20190918091020-d00ba619a349
-	github.com/flant/logboek_py v0.0.0-20190418220715-388556f27301
 	github.com/flynn-archive/go-shlex v0.0.0-20150515145356-3f9db97f8568
 	github.com/ghodss/yaml v0.0.0-20180820084758-c7ce16629ff4
 	github.com/google/btree v1.0.0

diff --git a/go.sum b/go.sum
@@ -180,8 +180,6 @@ github.com/flant/logboek v0.0.0-20190416104940-91fee3a3fc8d/go.mod h1:WirgB39yMT
 github.com/flant/logboek v0.2.6-0.20190726104558-c32b60bb4a37/go.mod h1:eEQXX0UOWpyC8iaA9QOvzx8drZ64u0SRESiwQKnxF+I=
 github.com/flant/logboek v0.2.6-0.20190918091020-d00ba619a349 h1:ANKnp53Kl4sspROTuIai9aAVRed4ZEX/dZI9Tj0gEPE=
 github.com/flant/logboek v0.2.6-0.20190918091020-d00ba619a349/go.mod h1:eEQXX0UOWpyC8iaA9QOvzx8drZ64u0SRESiwQKnxF+I=
-github.com/flant/logboek_py v0.0.0-20190418220715-388556f27301 h1:RJBJ5fgQ9LsK68PZ3PxExIsMvG0zIgJm/QACuGw+bxA=
-github.com/flant/logboek_py v0.0.0-20190418220715-388556f27301/go.mod h1:jvqdwUDE7x6dvYu6eLCb7+O8xDjFp51wfHql9NHB03w=
 github.com/flynn-archive/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:rZfgFAXFS/z/lEd6LJmf9HVZ1LkgYiHx5pHhV5DR16M=
 github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=

diff --git a/pkg/build/builder/ansible.go b/pkg/build/builder/ansible.go
@@ -63,12 +63,13 @@ func (b *Ansible) stage(userStageName string, container Container) error {
 
 	container.AddEnv(
 		map[string]string{
-			"ANSIBLE_CONFIG":              filepath.Join(b.containerWorkDir(), "ansible.cfg"),
-			"WERF_DUMP_CONFIG_DOC_PATH":   filepath.Join(b.containerWorkDir(), "dump_config.json"),
-			"PYTHONPATH":                  "",
-			"LOGBOEK_SO_PATH":             filepath.Join(b.containerWorkDir(), "lib", "werf", ".logboek.so"),
-			"PYTHONIOENCODING":            "utf-8",
-			"ANSIBLE_PREPEND_SYSTEM_PATH": stapel.SystemPATH(),
+			"ANSIBLE_CONFIG":            filepath.Join(b.containerWorkDir(), "ansible.cfg"),
+			"WERF_DUMP_CONFIG_DOC_PATH": filepath.Join(b.containerWorkDir(), "dump_config.json"),
+			"PYTHONIOENCODING":          "utf-8",
+			"PATH":                      fmt.Sprintf("%s:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:%s", stapel.AnsibleToolsOverlayPATH(), stapel.SystemPATH()),
+			"LD_LIBRARY_PATH":           stapel.AnsibleLibsOverlayLDPATH(),
+			"LANG":                      "C.UTF-8",
+			"LC_ALL":                    "C.UTF-8",
 		},
 	)
 

diff --git a/pkg/build/builder/ansible/callback/live.py b/pkg/build/builder/ansible/callback/live.py
@@ -28,8 +28,8 @@
 import json, re
 from collections import Iterable
 
+import logboek
 from werf.live_stdout import LiveStdoutListener
-from werf import logboek
 
 # Taken from Dstat
 class vt100:
@@ -376,7 +376,7 @@ def v2_playbook_on_play_start(self, play):
         #cols=60
         self.HEADER_NAME_INFO_LEN = cols-2
         logboek.SetTerminalWidth(cols)
-        logboek.FittedStreamsOutputOn()
+        logboek.EnableFitMode()
         #logboek.LogProcessStart(play.name)
         self._live_stdout_listener.start()
 

diff --git a/pkg/build/builder/ansible/sitecustomize.py b/pkg/build/builder/ansible/sitecustomize.py
@@ -2,16 +2,6 @@
 import sys, re
 import os
 
-path = os.environ.get('PATH', '')
-prepend_path = os.environ.get('ANSIBLE_PREPEND_SYSTEM_PATH', '')
-
-if path == '':
-    path = prepend_path
-elif prepend_path != '':
-    path = prepend_path + os.pathsep + path
-
-os.environ['PATH'] = path
-
 
 # __del__ is called when python assigns real arguments into sys.argv
 class IsAnsiballZ(object):
Original file line number	Diff line number	Diff line change
Expand Up		@@ -49,3 +49,5 @@ Also, this section includes articles with base primitives and general tools:
		* [Toolbox]({{ site.baseurl }}/documentation/reference/toolbox/slug.html).

		Werf is a CLI utility, so if you want to find a description of both basic commands needed to provide the CI/CD process and service commands that provide advanced functionality — use CLI Commands section.

		[Development section]({{ site.baseurl }}/documentation/development/stapel.html) contains service and maintenance manuals and other docs which help developers to understand how some werf subsystem works, how to maintain some subsystem in the actual state, how to write and build new code for the werf, etc.