From f0ca747ed1767078fa42a659d2df3a77b73b4348 Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Sun, 15 Dec 2019 00:35:22 +0100 Subject: [PATCH 001/333] Fix automatic version detection after tagging 3.15.0 Our build system version detection is based on assumption that if we are on any 3.[0-9]*.x branch, then we're not on master branch. When 3.15.0 branch points to a commit in master branch and doesn't have any extra commits, it wrongfully assumes that we're on this branch even if we're actually on master. This empty commit fixes it by moving 3.15.0 branch one step away from master. Changelog: None Signed-off-by: Aleksei Shpakovskii From d5fecdbb0cfa165f7451ea3a3c811edcc66c83ac Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Mon, 16 Dec 2019 14:35:38 +0100 Subject: [PATCH 002/333] Update libntech to current master We need the fixed file locking test. (cherry picked from commit f4f7a4119a184d4342b24d541c4ceda37b972e15) --- libntech | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libntech b/libntech index a6ca9fb599..60a2cdd015 160000 --- a/libntech +++ b/libntech @@ -1 +1 @@ -Subproject commit a6ca9fb599e3560b74b16c97339c2f443c5a71fd +Subproject commit 60a2cdd0158ef236c3b27de989f0be834c1a8b36 From 4bc0dd7c39a477ba173fcf05ed884c68cf76c03b Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Mon, 16 Dec 2019 21:49:05 +0100 Subject: [PATCH 003/333] Fix automatic version detection after tagging 3.15.0-build3 Our build system version detection is based on assumption that all repos either have same tag at HEAD, or not. If some repos have a tag, and others - don't, build aborts. To prevent build issues after adding commits to some (but not all) repos, we add this empty commit. Changelog: None Signed-off-by: Aleksei Shpakovskii From 07a7a15aeda122791646b2ea0df31a4e82f85d5c Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 18 Dec 2019 15:51:26 +0100 Subject: [PATCH 004/333] Fixed small memory leak in policy evaluation This memory leak appears when the LHS of a promise constraint is `data` or `template_data`, and the RHS is a string literal. If the string literal is not valid JSON, perhaps because of unexpanded variables, a `parsejson` function call is inserted instead. The name of the function call (for example `parsejson`) is leaked in this case. Not picked up by our CI, since valgrind test only runs community parts of masterfiles. Changelog: Title Ticket: None Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 2132aeb5b58ed01c8cb2810e10bea1a1edbd9929) --- libpromises/cf3parse.y | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libpromises/cf3parse.y b/libpromises/cf3parse.y index 9fea664a98..6c4ccbe560 100644 --- a/libpromises/cf3parse.y +++ b/libpromises/cf3parse.y @@ -661,7 +661,7 @@ constraint: constraint_id /* BUNDLE ONLY */ RlistAppendScalar(&synthetic_args, xstrdup(P.rval.item)); RvalDestroy(P.rval); - P.rval = (Rval) { FnCallNew(xstrdup(fname), synthetic_args), RVAL_TYPE_FNCALL }; + P.rval = (Rval) { FnCallNew(fname, synthetic_args), RVAL_TYPE_FNCALL }; } else { From c504dfdabb81f420f35569c72ac962f2f4a16bbe Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Thu, 19 Dec 2019 19:02:16 +0100 Subject: [PATCH 005/333] Fix automatic version detection after tagging 3.15.0 Our build system version detection is based on assumption that all repos either have same tag at HEAD, or not. If some repos have a tag, and others - don't, build aborts. To prevent build issues after adding commits to some (but not all) repos, we add this empty commit. Changelog: None Signed-off-by: Aleksei Shpakovskii From 396248803324bab8f50b4fbdb1cca5d7a1ba36c5 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Thu, 19 Dec 2019 13:35:05 -0600 Subject: [PATCH 006/333] Update badges for 3.15.0 release --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f1055388c4..4f27270b26 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,9 @@ | Version | [Core](https://github.com/cfengine/core) | [MPF](https://github.com/cfengine/masterfiles) | |------------|--------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------| | master | [![Core Build Status](https://travis-ci.org/cfengine/core.svg?branch=master)](https://travis-ci.org/cfengine/core) | [![MPF Build Status](https://travis-ci.org/cfengine/masterfiles.svg?branch=master)](https://travis-ci.org/cfengine/masterfiles) | +| 3.15.x LTS | [![Core Build Status](https://travis-ci.org/cfengine/core.svg?branch=3.15.x)](https://travis-ci.org/cfengine/core) | [![MPF Build Status](https://travis-ci.org/cfengine/masterfiles.svg?branch=3.15.x)](https://travis-ci.org/cfengine/masterfiles) | | 3.12.x LTS | [![Core Build Status](https://travis-ci.org/cfengine/core.svg?branch=3.12.x)](https://travis-ci.org/cfengine/core) | [![MPF Build Status](https://travis-ci.org/cfengine/masterfiles.svg?branch=3.12.x)](https://travis-ci.org/cfengine/masterfiles) | | 3.10.x LTS | [![Core Build Status](https://travis-ci.org/cfengine/core.svg?branch=3.10.x)](https://travis-ci.org/cfengine/core) | [![MPF Build Status](https://travis-ci.org/cfengine/masterfiles.svg?branch=3.10.x)](https://travis-ci.org/cfengine/masterfiles) | -| 3.7.x LTS | [![Core Build Status](https://travis-ci.org/cfengine/core.svg?branch=3.7.x)](https://travis-ci.org/cfengine/core) | [![MPF Build Status](https://travis-ci.org/cfengine/masterfiles.svg?branch=3.7.x)](https://travis-ci.org/cfengine/masterfiles) | [![codecov](https://codecov.io/gh/cfengine/core/branch/master/graph/badge.svg)](https://codecov.io/gh/cfengine/core) [![Language grade: C](https://img.shields.io/lgtm/grade/cpp/g/cfengine/core.svg?logo=lgtm&logoWidth=18&label=code%20quality)](https://lgtm.com/projects/g/cfengine/core/) From 83afe2191659ee7bc6e419db9aa3ff43020a5e28 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Thu, 19 Dec 2019 10:05:19 -0600 Subject: [PATCH 007/333] Moved error reading file in countlinesmatching() from verbose to error countlinesmatching() did not error when the file being counted does not exist or was unable to be accessed. The result of 0 matching lines is correct, but there is also an expectation of an error message to indicate the file is missing. Added test for countlinesmatching() erroring on absent file. This test simply runs an existing test and inspects the agent run output for an error. Ticket: CFE-3234 Changelog: Title (cherry picked from commit 5b4ab064165b7f35ac46cde37c8fb7dd096fc61c) --- libpromises/evalfunction.c | 2 +- ...ountlinesmatching_errors_on_absent_file.cf | 51 +++++++++++++++++++ ...atching_returns_0_on_inaccessable_file.cf} | 4 ++ 3 files changed, 56 insertions(+), 1 deletion(-) create mode 100644 tests/acceptance/01_vars/02_functions/countlinesmatching_errors_on_absent_file.cf rename tests/acceptance/01_vars/02_functions/{101.cf => countlinesmatching_returns_0_on_inaccessable_file.cf} (87%) diff --git a/libpromises/evalfunction.c b/libpromises/evalfunction.c index f66e6ce9e1..39e33858f4 100644 --- a/libpromises/evalfunction.c +++ b/libpromises/evalfunction.c @@ -3000,7 +3000,7 @@ static FnCallResult FnCallCountLinesMatching(ARG_UNUSED EvalContext *ctx, ARG_UN FILE *fin = safe_fopen(filename, "rt"); if (!fin) { - Log(LOG_LEVEL_VERBOSE, "File '%s' could not be read in countlinesmatching(). (fopen: %s)", filename, GetErrorStr()); + Log(LOG_LEVEL_ERR, "File '%s' could not be read in countlinesmatching(). (fopen: %s)", filename, GetErrorStr()); pcre_free(rx); return FnReturn("0"); } diff --git a/tests/acceptance/01_vars/02_functions/countlinesmatching_errors_on_absent_file.cf b/tests/acceptance/01_vars/02_functions/countlinesmatching_errors_on_absent_file.cf new file mode 100644 index 0000000000..5fc505da90 --- /dev/null +++ b/tests/acceptance/01_vars/02_functions/countlinesmatching_errors_on_absent_file.cf @@ -0,0 +1,51 @@ +####################################################### +# +# Test countlinesmatching() errors when file is absent +# +####################################################### + +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +####################################################### + +####################################################### + +bundle agent test +{ + meta: + "description" -> { "CFE-3234" } + string => "Test that the agent emits error output when countlinesmatching() is used on a file that is inaccessible."; + + "test_skip_needs_work" + string => "windows", + meta => { "CFE-3234" }; + + vars: + + "subout" string => execresult("$(sys.cf_agent) -Kv --define AUTO -f $(this.promise_dirname)/countlinesmatching_returns_0_on_inaccessable_file.cf 2>&1 | $(G.grep) countlinesmatching", "useshell"); + +} + +####################################################### + +bundle agent check +{ + + vars: + "expression" string => ".* error: File '/asd/fgh/qwertyio0p' could not be read in countlinesmatching.*"; + + classes: + "__pass" expression => regcmp( $(expression), $(test.subout) ); + + methods: + + __pass:: + "Pass" usebundle => dcs_pass( $(this.promise_filename) ); + !__pass:: + "FAIL" usebundle => dcs_pass( $(this.promise_filename) ); +} diff --git a/tests/acceptance/01_vars/02_functions/101.cf b/tests/acceptance/01_vars/02_functions/countlinesmatching_returns_0_on_inaccessable_file.cf similarity index 87% rename from tests/acceptance/01_vars/02_functions/101.cf rename to tests/acceptance/01_vars/02_functions/countlinesmatching_returns_0_on_inaccessable_file.cf index 4970b186c9..ed57633891 100644 --- a/tests/acceptance/01_vars/02_functions/101.cf +++ b/tests/acceptance/01_vars/02_functions/countlinesmatching_returns_0_on_inaccessable_file.cf @@ -23,6 +23,10 @@ bundle agent init bundle agent test { + meta: + "description" -> { "Mantis 320", "CFE-3234" } + string => "Test that countlinesmatching() returns 0 when the file is inaccessible."; + vars: "zero_regex" string => "impossible line"; From 988b561b03b7b5c2feb8da5ee574d0021bfc265a Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Fri, 20 Dec 2019 17:06:15 +0100 Subject: [PATCH 008/333] Travis: Run on 3.15.x branch and PRs Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit b27cee904694eb9f56391035fe4d81a212ceb2ba) --- .travis.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.travis.yml b/.travis.yml index 82d87b8e7f..f076c714c1 100644 --- a/.travis.yml +++ b/.travis.yml @@ -12,6 +12,7 @@ branches: - master - 3.10.x - 3.12.x + - 3.15.x env: global: From 8a2d2351dfcce45601723da32e957586a2fa0d16 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Fri, 20 Dec 2019 16:07:04 +0100 Subject: [PATCH 009/333] Travis OSX: Fix missing OpenSSL Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit c8625d9a68bf7ce1d6da3d7bb0beeb198f46163b) --- travis-scripts/before_install.sh | 2 +- travis-scripts/script.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/travis-scripts/before_install.sh b/travis-scripts/before_install.sh index 64db288dd3..2230a0d0c2 100755 --- a/travis-scripts/before_install.sh +++ b/travis-scripts/before_install.sh @@ -9,11 +9,11 @@ then brew install make brew install autoconf brew install automake + brew install openssl # brew install gcc@7 || brew link --overwrite gcc@7 set -e # gcc-7 --version #brew install python - #brew install openssl #brew install libxml2 #brew install fakeroot else diff --git a/travis-scripts/script.sh b/travis-scripts/script.sh index 1bd2b9f0c2..9426c61999 100755 --- a/travis-scripts/script.sh +++ b/travis-scripts/script.sh @@ -28,7 +28,7 @@ git remote add upstream https://github.com/cfengine/core.git \ if [ "$TRAVIS_OS_NAME" = osx ] then - ./autogen.sh --enable-debug --prefix=$INSTDIR --bindir=$INSTDIR/var/cfengine/bin + ./autogen.sh --enable-debug --with-openssl="$(brew --prefix openssl)" --prefix=$INSTDIR --bindir=$INSTDIR/var/cfengine/bin gmake --version gmake CFLAGS="-Werror -Wall -Wno-pointer-sign" gmake --debug -C tests/unit check From ee5574585661f7796afea07c7bd3fd6c6eb26ce7 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 8 Jan 2020 14:24:59 +0100 Subject: [PATCH 010/333] Find the matching parenthesis in variable reference If a variable reference contains parentheses, we need to make sure we find the matching parenthesis for the opening one, not the first opposite parenthesis. For example '$(my_array[key(1)])' needs to match the second closing parenthesis not the first one. Ticket: CFE-3242 Changelog: Variable references with nested parentheses no longer cause errors (cherry picked from commit d3a144e829dd3e8fc3c9ae36b221e9efde969062) --- libpromises/iteration.c | 47 +++++++++++++++++-- .../01_vars/01_basic/nested_parens_var_ref.cf | 41 ++++++++++++++++ 2 files changed, 84 insertions(+), 4 deletions(-) create mode 100644 tests/acceptance/01_vars/01_basic/nested_parens_var_ref.cf diff --git a/libpromises/iteration.c b/libpromises/iteration.c index df7ce07e14..a9ef013947 100644 --- a/libpromises/iteration.c +++ b/libpromises/iteration.c @@ -253,6 +253,34 @@ static char opposite(char c) return 0; } +/** + * Find the closing parenthesis for #c in #s. #c is considered to *not* be part + * of #s (IOW, #s is considered to be a string after #c). + * + * @return A closing parenthesis for #c in #s or %NULL if not found + */ +static char *FindClosingParen(char *s, char c) +{ + char closing = opposite(c); + int counter = 0; + for (char *cur=s; *cur != '\0'; cur++) + { + if (*cur == closing) + { + if (counter == 0) + { + return cur; + } + counter--; + } + if (*cur == c) + { + counter++; + } + } + return NULL; +} + /** * Check if variable reference is mangled, while avoiding going into the inner * variables that are being expanded, or into array indexes. @@ -490,9 +518,14 @@ static char *ProcessVar(PromiseIterator *iterctx, const EvalContext *evalctx, assert(s != NULL); assert(c == '(' || c == '{'); - char closing_paren = opposite(c); - char *s_end = strchrnul(s, closing_paren); + char *s_end = FindClosingParen(s, c); const size_t s_max = strlen(s); + if (s_end == NULL) + { + /* Set s_end to the point to the NUL byte if no closing parenthesis was + * found. It's used for comparisons and other things below. */ + s_end = s + s_max; + } char *next_var = s + FindDollarParen(s, s_max); size_t deps = 0; @@ -523,7 +556,13 @@ static char *ProcessVar(PromiseIterator *iterctx, const EvalContext *evalctx, /* We are sure (subvar_end+1) is not out of bounds. */ char *s_next = subvar_end + 1; const size_t s_next_len = strlen(s_next); - s_end = strchrnul(s_next, closing_paren); + s_end = FindClosingParen(s_next, c); + if (s_end == NULL) + { + /* Set s_end to the point to the NUL byte if no closing parenthesis was + * found. It's used for comparisons and other things below. */ + s_end = s_next + s_next_len; + } next_var = s_next + FindDollarParen(s_next, s_next_len); } } @@ -576,7 +615,7 @@ static char *ProcessVar(PromiseIterator *iterctx, const EvalContext *evalctx, } assert(s_end != NULL); - assert(*s_end == closing_paren); + assert(*s_end == opposite(c)); return s_end; } diff --git a/tests/acceptance/01_vars/01_basic/nested_parens_var_ref.cf b/tests/acceptance/01_vars/01_basic/nested_parens_var_ref.cf new file mode 100644 index 0000000000..1dc941a439 --- /dev/null +++ b/tests/acceptance/01_vars/01_basic/nested_parens_var_ref.cf @@ -0,0 +1,41 @@ +########################################################### +# +# Test that a variable reference with nested parentheses works +# +########################################################### + +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default($(this.promise_filename)) }; + version => "1.0"; +} + +########################################################### + +bundle agent test +{ + meta: + "description" -> { "CFE-3242" } + string => "Test that a variable reference with nested parentheses works"; + + vars: + "my_array[key(1)]" string => "value"; + "value" string => "$(my_array[key(1)])"; +} + +########################################################### + +bundle agent check +{ + classes: + "ok" and => { isvariable( "test.value" ), + strcmp( "$(test.value)", "value") + }; + + reports: + ok:: + "$(this.promise_filename) Pass"; + !ok:: + "$(this.promise_filename) FAIL"; +} From 9726ce7f0d5865810c651671acfc448dcf160544 Mon Sep 17 00:00:00 2001 From: Karl Hole Totland Date: Thu, 12 Dec 2019 13:07:44 +0100 Subject: [PATCH 011/333] Refactor readdata to use new json-utils from libntech Now uses an enum defined in `json-utils.h` instead of trying to find the same data several times by comparing the function name. Also split readdata and e.g. readcsv into different functions, since their functionality is a bit different. Created two generic functions, one for the readcsv/yaml/envfile/json functions, and one that is used by both readdata and the former generic function that calls `JsonReadDataFile`. Also changed occurence of `JsonReadDataFile` in `libenv/sysinfo.c` to use `DATAFILETYPE_ENV` instead of `"env"`. This was minor enough to keep in this commit. Ticket: None Changelog: None (cherry picked from commit 1428ba5e0a007d1ee270629eb7efef18a616672e) --- libenv/sysinfo.c | 2 +- libpromises/evalfunction.c | 153 ++++++++++++++++++++----------------- 2 files changed, 83 insertions(+), 72 deletions(-) diff --git a/libenv/sysinfo.c b/libenv/sysinfo.c index d7c08c5e27..6fa60a5d5a 100644 --- a/libenv/sysinfo.c +++ b/libenv/sysinfo.c @@ -1122,7 +1122,7 @@ static void DefineVersionedHardClasses( static void OSReleaseParse(EvalContext *ctx, const char *file_path) { JsonElement *os_release_json = JsonReadDataFile("system info discovery", - file_path, "ENV", + file_path, DATAFILETYPE_ENV, 100 * 1024); if (os_release_json != NULL) { diff --git a/libpromises/evalfunction.c b/libpromises/evalfunction.c index 39e33858f4..aeae150f00 100644 --- a/libpromises/evalfunction.c +++ b/libpromises/evalfunction.c @@ -100,7 +100,6 @@ static bool CheckIDChar(const char ch); static bool CheckID(const char *id); static const Rlist *GetListReferenceArgument(const EvalContext *ctx, const FnCall *fp, const char *lval_str, DataType *datatype_out); static char *CfReadFile(const char *filename, int maxsize); -static const char *FileType(const char *filename); /*******************************************************************/ @@ -6634,11 +6633,29 @@ static FnCallResult FnCallReadRealList(EvalContext *ctx, ARG_UNUSED const Policy return ReadList(ctx, fp, args, CF_DATA_TYPE_REAL); } +static FnCallResult ReadDataGeneric(const char *const fname, + const char *const input_path, + const size_t size_max, + const DataFileType requested_mode) +{ + assert(fname != NULL); + assert(input_path != NULL); + + JsonElement *json = JsonReadDataFile(fname, input_path, requested_mode, size_max); + if (json == NULL) + { + return FnFailure(); + } + + return (FnCallResult) { FNCALL_SUCCESS, (Rval) { json, RVAL_TYPE_CONTAINER } }; +} + static FnCallResult FnCallReadData(ARG_UNUSED EvalContext *ctx, ARG_UNUSED const Policy *policy, const FnCall *fp, const Rlist *args) { + assert(fp != NULL); if (args == NULL) { Log(LOG_LEVEL_ERR, "Function '%s' requires at least one argument", fp->name); @@ -6646,56 +6663,73 @@ static FnCallResult FnCallReadData(ARG_UNUSED EvalContext *ctx, } const char *input_path = RlistScalarValue(args); - size_t size_max; - - // JsonReadDataFile reads key-value pairs from a json, yaml, csv or - // env file and puts them in a JSON element - // requested_mode is used to specify the input file type - // requested_mode is set based on file extension or function name - // example: readyaml -> "YAML" , ".csv" -> "CSV" etc. - // defaults to "JSON" - const char *requested_mode = NULL; - if (strcmp(fp->name, "readdata") == 0) + const char *const mode_string = RlistScalarValue(args->next); + DataFileType requested_mode = DATAFILETYPE_UNKNOWN; + if (StringSafeEqual("auto", mode_string)) { - // readdata gets rid of the size, well almost - // csv still has 50MB restriction applied later - size_max = IntFromString("inf"); - requested_mode = RlistScalarValue(args->next); - if (strcmp("auto", requested_mode) == 0) - { - requested_mode = FileType(input_path); - Log(LOG_LEVEL_VERBOSE, "%s: automatically selected data type %s from filename %s", fp->name, requested_mode, input_path); - } + requested_mode = GetDataFileTypeFromSuffix(input_path); + Log(LOG_LEVEL_VERBOSE, + "%s: automatically selected data type %s from filename %s", + fp->name, DataFileTypeToString(requested_mode), input_path); } else { - size_max = args->next ? IntFromString(RlistScalarValue(args->next)) : IntFromString("inf"); - - if (strcmp(fp->name, "readyaml") == 0) - { - requested_mode = "YAML"; - } - else if (strcmp(fp->name, "readcsv") == 0) - { - requested_mode = "CSV"; - } - else if (strcmp(fp->name, "readenvfile") == 0) - { - requested_mode = "ENV"; - } - else // always default to JSON - { - requested_mode = "JSON"; - } + requested_mode = GetDataFileTypeFromString(mode_string); } - JsonElement *json = JsonReadDataFile(fp->name, input_path, requested_mode, size_max); - if (json == NULL) + return ReadDataGeneric(fp->name, input_path, CF_INFINITY, requested_mode); +} + +static FnCallResult ReadGenericDataType(const FnCall *fp, + const Rlist *args, + const DataFileType requested_mode) +{ + assert(fp != NULL); + if (args == NULL) { + Log(LOG_LEVEL_ERR, + "Function '%s' requires at least one argument", + fp->name); return FnFailure(); } - return (FnCallResult) { FNCALL_SUCCESS, (Rval) { json, RVAL_TYPE_CONTAINER } }; + const char *const input_path = RlistScalarValue(args); + size_t size_max = args->next ? + IntFromString(RlistScalarValue(args->next)) : + CF_INFINITY; + return ReadDataGeneric(fp->name, input_path, size_max, requested_mode); +} + +static FnCallResult FnCallReadCsv(ARG_UNUSED EvalContext *ctx, + ARG_UNUSED const Policy *policy, + const FnCall *fp, + const Rlist *args) +{ + return ReadGenericDataType(fp, args, DATAFILETYPE_CSV); +} + +static FnCallResult FnCallReadEnvFile(ARG_UNUSED EvalContext *ctx, + ARG_UNUSED const Policy *policy, + const FnCall *fp, + const Rlist *args) +{ + return ReadGenericDataType(fp, args, DATAFILETYPE_ENV); +} + +static FnCallResult FnCallReadYaml(ARG_UNUSED EvalContext *ctx, + ARG_UNUSED const Policy *policy, + const FnCall *fp, + const Rlist *args) +{ + return ReadGenericDataType(fp, args, DATAFILETYPE_YAML); +} + +static FnCallResult FnCallReadJson(ARG_UNUSED EvalContext *ctx, + ARG_UNUSED const Policy *policy, + const FnCall *fp, + const Rlist *args) +{ + return ReadGenericDataType(fp, args, DATAFILETYPE_JSON); } static FnCallResult FnCallReadModuleProtocol( @@ -8059,30 +8093,6 @@ static bool ExecModule(EvalContext *ctx, char *command) return true; } -static const char *FileType(const char *filename) -{ - if (StringEndsWithCase(filename, ".csv", true)) - { - return "CSV"; - } - else if (StringEndsWithCase(filename, ".yaml", true)) - { - return "YAML"; - } - else if (StringEndsWithCase(filename, ".yml", true)) - { - return "YAML"; - } - else if (StringEndsWithCase(filename, ".env", true)) - { - return "ENV"; - } - else // always default to JSON - { - return "JSON"; - } -} - void ModuleProtocol(EvalContext *ctx, const char *command, const char *line, int print, char* context, size_t context_size, StringSet *tags, long *persistence) { assert(tags); @@ -8266,9 +8276,10 @@ void ModuleProtocol(EvalContext *ctx, const char *command, const char *line, int if (FileCanOpen(content, "r")) { const int size_max = IntFromString("inf"); - const char *requested_mode = FileType(content); + const DataFileType requested_mode = GetDataFileTypeFromSuffix(content); - Log(LOG_LEVEL_DEBUG, "Module protocol parsing %s file '%s'", requested_mode, content); + Log(LOG_LEVEL_DEBUG, "Module protocol parsing %s file '%s'", + DataFileTypeToString(requested_mode), content); JsonElement *json = JsonReadDataFile("module file protocol", content, requested_mode, size_max); if (json != NULL) @@ -9608,13 +9619,13 @@ const FnCallType CF_FNCALL_TYPES[] = FNCALL_OPTION_NONE, FNCALL_CATEGORY_IO, SYNTAX_STATUS_NORMAL), FnCallTypeNew("readfile", CF_DATA_TYPE_STRING, READFILE_ARGS, &FnCallReadFile, "Read max number of bytes from named file and assign to variable", FNCALL_OPTION_VARARG, FNCALL_CATEGORY_IO, SYNTAX_STATUS_NORMAL), - FnCallTypeNew("readcsv", CF_DATA_TYPE_CONTAINER, READFILE_ARGS, &FnCallReadData, "Parse a CSV file and return a JSON data container with the contents", + FnCallTypeNew("readcsv", CF_DATA_TYPE_CONTAINER, READFILE_ARGS, &FnCallReadCsv, "Parse a CSV file and return a JSON data container with the contents", FNCALL_OPTION_VARARG, FNCALL_CATEGORY_IO, SYNTAX_STATUS_NORMAL), - FnCallTypeNew("readenvfile", CF_DATA_TYPE_CONTAINER, READFILE_ARGS, &FnCallReadData, "Parse a ENV-style file and return a JSON data container with the contents", + FnCallTypeNew("readenvfile", CF_DATA_TYPE_CONTAINER, READFILE_ARGS, &FnCallReadEnvFile, "Parse a ENV-style file and return a JSON data container with the contents", FNCALL_OPTION_VARARG, FNCALL_CATEGORY_IO, SYNTAX_STATUS_NORMAL), - FnCallTypeNew("readjson", CF_DATA_TYPE_CONTAINER, READFILE_ARGS, &FnCallReadData, "Read a JSON data container from a file", + FnCallTypeNew("readjson", CF_DATA_TYPE_CONTAINER, READFILE_ARGS, &FnCallReadJson, "Read a JSON data container from a file", FNCALL_OPTION_VARARG, FNCALL_CATEGORY_IO, SYNTAX_STATUS_NORMAL), - FnCallTypeNew("readyaml", CF_DATA_TYPE_CONTAINER, READFILE_ARGS, &FnCallReadData, "Read a data container from a YAML file", + FnCallTypeNew("readyaml", CF_DATA_TYPE_CONTAINER, READFILE_ARGS, &FnCallReadYaml, "Read a data container from a YAML file", FNCALL_OPTION_VARARG, FNCALL_CATEGORY_IO, SYNTAX_STATUS_NORMAL), FnCallTypeNew("read_module_protocol", CF_DATA_TYPE_CONTEXT, READMODULE_ARGS, &FnCallReadModuleProtocol, "Parse a file containing module protocol output (for cached modules)", FNCALL_OPTION_NONE, FNCALL_CATEGORY_IO, SYNTAX_STATUS_NORMAL), From fa283ce2ce6ef78621fcbb0dc2f676cb34d642e0 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Tue, 14 Jan 2020 16:11:27 +0100 Subject: [PATCH 012/333] libntech: Added ReadFileStreamToBuffer from Mender Just an update to latest master. Changelog: None Ticket: None Signed-off-by: Ole Herman Schumacher Elgesem --- libntech | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libntech b/libntech index 60a2cdd015..683121898e 160000 --- a/libntech +++ b/libntech @@ -1 +1 @@ -Subproject commit 60a2cdd0158ef236c3b27de989f0be834c1a8b36 +Subproject commit 683121898e4056d821ffad372132ab8eed481352 From 421af9b3aa696aa4269509c8a392c9c46a93c225 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Thu, 16 Jan 2020 09:06:12 +0100 Subject: [PATCH 013/333] Removed (USE AT YOUR OWN RISK) from cf-key help menu for -x It is kind of implied that --force-removal can be destructive. We don't have any specific reasoning for this phrasing, so it is not so useful, just confusing / concerning for a user. Ticket: ENT-5090 Changelog: Title Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 8a3d32de48ccae14830e4c937ac61de08b728ae8) --- cf-key/cf-key.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cf-key/cf-key.c b/cf-key/cf-key.c index 3ffd31433d..04e37282ff 100644 --- a/cf-key/cf-key.c +++ b/cf-key/cf-key.c @@ -104,7 +104,7 @@ static const char *const HINTS[] = "Show lastseen hostnames and IP addresses", "Don't truncate -s / --show-hosts output", "Remove keys for specified hostname/IP/MD5/SHA (cf-key -r SHA=12345, cf-key -r MD5=12345, cf-key -r host001, cf-key -r 203.0.113.1)", - "Force removal of keys (USE AT YOUR OWN RISK)", + "Force removal of keys", "Install license file on Enterprise server (CFEngine Enterprise Only)", "Print digest of the specified public key", "Make cf-serverd/cf-agent trust the specified public key. Argument value is of the form [[USER@]IPADDR:]FILENAME where FILENAME is the local path of the public key for client at IPADDR address.", From 17ab8e2c6fdab3ff033bc5a9b40e2bce775e06ce Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Thu, 9 Jan 2020 16:21:18 +0100 Subject: [PATCH 014/333] cf-check: Added a more user friendly message when trying to print unknown binary data Changelog: Title Ticket: ENT-5234 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 5386c560df4b43497419965ecfe68948d8d68cd3) --- cf-check/dump.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/cf-check/dump.c b/cf-check/dump.c index 2fc387967d..50d6abb74d 100644 --- a/cf-check/dump.c +++ b/cf-check/dump.c @@ -49,9 +49,14 @@ static void print_json_string( { const size_t len = strnlen(data, size); - // Unfortunately, the packages_installed_apt_get.lmdb database - // has unterminated strings: - assert((size == 1) || (len == (size - 1)) || (data[size - 1] == '\n')); + // We expect the data to either be single byte ("0" or "1"), + // or a C-string + bool known_data = ((size == 1) || (len == (size - 1)) || (data[size - 1] == '\n')); + if (!known_data) + { + printf("\nError: This database contains unknown binary data - use --simple to print anyway\n"); + exit(1); + } // Most of what we store are C strings except 1 byte '0' or '1', // and some structs. So in nice mode, we try to default to printing From fd81133929f17284ba8c85d73cbade646034fb04 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Thu, 9 Jan 2020 17:04:29 +0100 Subject: [PATCH 015/333] cf-check: Added nice printing for nova_agent_executions.lmdb Changelog: Title Ticket: ENT-5234 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 689f2ad4c09c7aeb6d8405ea16550eff44e5fd0d) --- cf-check/dump.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/cf-check/dump.c b/cf-check/dump.c index 50d6abb74d..611ef22626 100644 --- a/cf-check/dump.c +++ b/cf-check/dump.c @@ -288,6 +288,25 @@ static void print_struct_or_string( { print_struct_persistent_class(value, strip_strings); } + else if (StringEndsWith(file, "nova_agent_execution.lmdb")) + { + if (StringSafeEqual(key.mv_data, "delta_gavr")) + { + assert(sizeof(double) == value.mv_size); + const double *const average = value.mv_data; + printf("%f", *average); + } + else if (StringSafeEqual(key.mv_data, "last_exec")) + { + assert(sizeof(time_t) == value.mv_size); + const time_t *const last_exec = value.mv_data; + printf("%ju", (uintmax_t) (*last_exec)); + } + else + { + debug_abort_if_reached(); + } + } else { print_json_string(value.mv_data, value.mv_size, strip_strings); From 0f998cc63cf808170a8b828fad090ca1c43f3fa4 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Fri, 17 Jan 2020 14:14:37 +0100 Subject: [PATCH 016/333] Replaced determine-version.py with a manually updated file Added determine-version.sh to replace it. A much simpler solution which trusts the version number from the commited file, but just adds a commit SHA if there is no tag. This should get rid of all our problems with version mismatch. Ticket: ENT-5304 Changelog: None Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 36f2290412894ac6c601c6fabf67870176cc49d1) --- .CFVERSION | 1 + autogen.sh | 4 +- misc/determine-version.py | 268 -------------------------------------- misc/determine-version.sh | 49 +++++++ travis-scripts/script.sh | 4 +- 5 files changed, 53 insertions(+), 273 deletions(-) create mode 100644 .CFVERSION delete mode 100755 misc/determine-version.py create mode 100755 misc/determine-version.sh diff --git a/.CFVERSION b/.CFVERSION new file mode 100644 index 0000000000..c3df54c9b8 --- /dev/null +++ b/.CFVERSION @@ -0,0 +1 @@ +3.15.1 diff --git a/autogen.sh b/autogen.sh index 61b933f079..1f38538533 100755 --- a/autogen.sh +++ b/autogen.sh @@ -61,9 +61,9 @@ test -z "$srcdir" && srcdir=. cd "$srcdir" -echo "$0: Running determine-version.py ..." +echo "$0: Running determine-version.sh ..." rm -f CFVERSION -{ misc/determine-version.py > CFVERSION && cp CFVERSION libntech/CFVERSION ; } \ +{ misc/determine-version.sh .CFVERSION > CFVERSION && cp CFVERSION libntech/CFVERSION ; } \ || echo "$0: Unable to auto-detect CFEngine version, continuing" echo "$0: Running autoreconf ..." diff --git a/misc/determine-version.py b/misc/determine-version.py deleted file mode 100755 index 33b475ee0b..0000000000 --- a/misc/determine-version.py +++ /dev/null @@ -1,268 +0,0 @@ -#!/usr/bin/env python3 - -from __future__ import print_function -import re -import subprocess -import sys -import os -from os.path import basename -try: - from functools import cmp_to_key -except ImportError: - cmp_to_key = None - -try: - DEVNULL = subprocess.DEVNULL -except AttributeError: - DEVNULL = open(os.devnull, 'wb') - - -# Determines which version we should call this build. -# -# Consider the following Git history graph with tag names: -# -# X Y Z M -# | | | | -# * 3.7.2 * 3.8.1 * 3.9.0b1 * master -# | | | | -# * 3.7.1 * 3.8.0 | | -# | | \------------+ -# * 3.7.0 * 3.8.0b1 | -# | | | -# | \-----------------------+ -# | | -# \----------------------------------+ -# | -# * 3.7.0b1 -# | -# -# At each of the points, X, Y, Z and M, the script needs to find the correct -# next version based on existing tag names. This is the algorithm to achieve -# that. -# -# 1. Find the most recent version tag reachable from the point, let's call it T. -# 2a. Fuzzy-detect whether we are on master branch. -# 2b. Find all tags that contain T in their history. If we are not on -# the master branch, exclude tags that point to the same commit as T -# (for example T itself and T-build1). -# 3. If no tags were found, then increase *patch number* of T by one and -# return that (beta becomes 0). Because having no subsequent tags -# indicates that we are on the same branch as T. -# 4. If there are tags found in step two, find the latest one (as sorted -# by version) which does not point at the same commit as T, increase -# the minor number by one, and return that. Because this indicates -# that we have branched-off after T was issued, and the new tags that -# contain T were issued from branches not reachable to us now. In the -# above graph, we are on a new ZZ branch with no tags, and the most -# recent reachable tag T is 3.7.0b1, so we search for the most recent -# tag containing 3.7.0b1, which is 3.9.0b1, and we increase the -# *minor* number, i.e. we set the version to 3.10.0. Magic! -# -# Some notes: -# - Does not take into account changes in major version. -# - If you branch off from a patch series from a point earlier than the last -# patch release, the script will think you are naming the next minor version. -# (So in the graph above, finding the next version string at point 3.7.1, -# would find 3.8.0). This scenario has no well defined version, so it's a -# compromise. - - -# Print debug info to stderr -VERBOSE=True - - -def usage(): - print("Usage: %s [revision]" % sys.argv[0]) - print() - print("Prints the predicted next version after the given revision, based on") - print("current tags.") - sys.exit(1) - -def verbose_print(*args, **kwargs): - if VERBOSE: - print("%s:" % basename(sys.argv[0]), - *args, file=sys.stderr, **kwargs) - -def final_print(s): - verbose_print("SUCCESS, version detected is: %s" % s) - print(s) - -def extract_version_components(version): - match = re.match("^([0-9]+)\\.([0-9]+)\\.([0-9]+)(.*)", version) - if match is None: - return None - return (match.group(1), match.group(2), match.group(3), match.group(4)) - -def version_cmp(a, b): - a_major = int(a[0]) - b_major = int(b[0]) - if a_major > b_major: - return 1 - elif a_major < b_major: - return -1 - - a_minor = int(a[1]) - b_minor = int(b[1]) - if a_minor > b_minor: - return 1 - elif a_minor < b_minor: - return -1 - - # A "pure" version with no extra tag info is considered higher than - # an "impure" one, IOW "3.8.0" > "3.8.0b1". - if a[3] == "": - if b[3] != "": - return 1 - else: - if b[3] == "": - return -1 - - if a[2] > b[2]: - return 1 - elif a[2] < b[2]: - return -1 - - # a[2] == b[2] - if a[3] > b[3]: - return 1 - elif a[3] < b[3]: - return -1 - - return 0 - - - - -if len(sys.argv) < 2: - REV = "HEAD" -else: - if sys.argv[1] == "-h" or sys.argv[1] == "--help": - usage() - REV = sys.argv[1] - -# Find the full and abbreviated SHAs of the commit -git = subprocess.Popen(["git", "rev-list", "-1", REV], - stdout=subprocess.PIPE) -full_rev = git.stdout.readlines()[0].decode().strip() -git = subprocess.Popen(["git", "rev-list", "-1", "--abbrev-commit", REV], - stdout=subprocess.PIPE) -abbrev_rev = git.stdout.readlines()[0].decode().strip() -verbose_print("REV = %s" % REV) -verbose_print("full_rev = %s" % full_rev) -verbose_print("abbrev_rev = %s" % abbrev_rev) - - -# If the commit is referenced exactly in a tag, then use that tag as is -git = subprocess.Popen(["git", "describe", "--tags", "--exact-match", "--abbrev=0", REV], - stdout=subprocess.PIPE, stderr=DEVNULL) -try: - exact_tag = git.stdout.readlines()[0].decode().strip() - verbose_print("exact_tag = %s" % exact_tag) - (version_major, version_minor, version_patch, version_extra) = extract_version_components(exact_tag) - final_print("%s.%s.%s%s" % (version_major, version_minor, version_patch, version_extra)) - sys.exit(0) -except IndexError: # command returned no output - verbose_print("exact_tag = not found") - -# Find the most recent tag reachable from this commit. -git = subprocess.Popen(["git", "describe", "--tags", "--abbrev=0", REV], - stdout=subprocess.PIPE) -recent_tag_random = git.stdout.readlines()[0].decode().strip() - -# Find the revision corresponding to the tag -git = subprocess.Popen(["git", "rev-parse", recent_tag_random + "^{}"], - stdout=subprocess.PIPE) -recent_rev = git.stdout.readlines()[0].decode().strip() -verbose_print("recent_rev = %s" % recent_rev) - -# Find the shortest (nicest) tag for the given revision -git = subprocess.Popen(["git", "tag", "--points-at", recent_rev], - stdout=subprocess.PIPE) -recent_tag = git.stdout.readlines()[0].decode().strip() -verbose_print("recent_tag = %s" % recent_tag) - -# Find its version, if any. -recent_version = extract_version_components(recent_tag) -verbose_print("recent_version = ", recent_version) - - -in_master_branch = True -git_all_merged_branches = subprocess.Popen(["git", "branch", "-r", "--merged"], - stdout=subprocess.PIPE) -for line in git_all_merged_branches.stdout: - line = line.decode().strip() - match = re.match('(origin|upstream)/\\d+\\.\\d+\\.x$', line) - if match: - in_master_branch = False - verbose_print("Detected that we are NOT in master branch, but in [%s]" % match.group(0)) - break - - -# List all tags that contain the most recent tag found earlier, unless -# no such tag was found in which case we list all tags. -if recent_version is None: - (recent_major, recent_minor, recent_patch, recent_extra) = ("0", "0", "0", "") - tag_finder = ["git", "tag"] -else: - (recent_major, recent_minor, recent_patch, recent_extra) = recent_version - tag_finder = ["git", "tag", "--contains", recent_tag] - -git_tag_list = subprocess.Popen(tag_finder, - stdout=subprocess.PIPE) -all_tags = [] -for tag in git_tag_list.stdout.readlines(): - tag = tag.decode().strip() - git_rev = subprocess.Popen(["git", "rev-parse", tag + "^{}"], - stdout=subprocess.PIPE) - rev = git_rev.stdout.readlines()[0].decode().strip() - if not in_master_branch and rev == recent_rev: - # Ignore tags that point at the same commit as the most recent - # tag, i.e. ignore the tag itself and same-release tags (like - # 3.7.5-build1 which points to the same commit as 3.7.5), - # because we want the len(all_tags) comparison later to - # work. But if we are on master branch, keep it, in order to - # increase the minor version of the most recent tag then. - continue - match = extract_version_components(tag) - if match is None: - # Ignore non-version tags. - continue - all_tags.append(match) - -if cmp_to_key is None: - all_tags = sorted(all_tags, cmp=version_cmp, reverse=True) -else: - all_tags = sorted(all_tags, key=cmp_to_key(version_cmp), reverse=True) -verbose_print("all_tags =", all_tags) - - -if len(all_tags) == 0 or (not in_master_branch): - # No tags besides the most recent one were found, so this is a new - # patch version. So "increase" the patch version: - - # A "pure" version with no extra tag info is considered higher than - # an "impure" one, IOW "3.8.0" > "3.8.0b1". - if recent_extra != "": - patch_version = int(recent_patch) - else: - patch_version = int(recent_patch) + 1 - final_print("%s.%s.%da.%s" % (recent_major, recent_minor, patch_version, abbrev_rev)) -else: - # The most recent tag reachable from here is contained in many more - # tags. This means that branches were created *after* that tag, and - # then further tags were issued, and they all represent new minor - # (3.x) versions. But since those are not reachable from here, we - # have branched off again, i.e. we need a new *minor* (3.x) - # version. So find the newest minor version and increase it. - - # Unless the newest minor version is an alpha or beta release which - # should not trigger minor version incrementing (we are yet to - # release the final version of that minor release). - # A tag in all_tags is like: ('3', '15', '0', 'b1-build2') - if all_tags[0][3].startswith(("a", "b")): - final_print("%s.%d.0a.%s" % (all_tags[0][0], int(all_tags[0][1]), abbrev_rev)) - else: - final_print("%s.%d.0a.%s" % (all_tags[0][0], int(all_tags[0][1]) + 1, abbrev_rev)) - - -sys.exit(0) diff --git a/misc/determine-version.sh b/misc/determine-version.sh new file mode 100755 index 0000000000..52c9d8e75f --- /dev/null +++ b/misc/determine-version.sh @@ -0,0 +1,49 @@ +# This script takes an input file (.CFVERSION) containing a version number: +# 3.16.0 +# And outputs either 3.16.0 or 3.16.0a.c1f3175c5 +# It replaces determine-version.py, but is much simpler: +# If you are on the correct tag, just use the version number as is +# otherwise +# The output is usually redirected to the CFVERSION file. +# +# NOTE: 3.16.0 or 3.16.0a.c1f3175c5 are the same version, and should not +# cause a version mismatch. The SHA is mostly a convenience, to be +# able to see non-final version numbers on nightly / PR builds, etc. +# +# NOTE: This script does not care about EXPLICIT_VERSION env var. +# This is by design, since it can be changed at configure time. +# The configure script will check for EXPLICIT_VERSION variable, and +# prefer that one, if present. + +if [ "$#" -ne 1 ] +then + echo "Usage: determine-version.sh path/to/.CFVERSION" + exit 1 +fi + +# Get the desired version from file (manually commited .CFVERSION): +desired_version=$(cat $1) + +# Get the current commit SHA (HEAD): +current_sha=$(git rev-parse HEAD) + +print_version_with_sha () { + sha=$(echo $current_sha | cut -b 1-9) + echo $desired_version"a."$sha + exit 0 +} + +print_version_without_sha () { + echo $desired_version + exit 0 +} + +tag_sha=$(git rev-list -n 1 $desired_version 2>/dev/null || echo "") +if [ "x$current_sha" = "x$tag_sha" ] +then + # Tag exists and matches our current commit (HEAD): + print_version_without_sha +else + # Tag exists, but is on another commit: + print_version_with_sha +fi diff --git a/travis-scripts/script.sh b/travis-scripts/script.sh index 9426c61999..fe3d9e92dd 100755 --- a/travis-scripts/script.sh +++ b/travis-scripts/script.sh @@ -19,9 +19,7 @@ INSTDIR=$HOME/cf_install # fi # Unshallow the clone. Fetch the tags from upstream even if we are on a -# foreign clone. Needed for determine-version.py to work, specifically -# `git describe --tags HEAD` was failing once the last tagged commit -# became too old. +# foreign clone. Needed for determine-version.sh to work. git fetch --unshallow git remote add upstream https://github.com/cfengine/core.git \ && git fetch upstream 'refs/tags/*:refs/tags/*' From 12e8b63805ec31746a453ac6b2d72905e8bfe733 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 15 Jan 2020 14:04:43 +0100 Subject: [PATCH 017/333] Whitespace / clang-format Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 7f203f38750facf300d903cf8516ec3f10ba8261) --- cf-check/diagnose.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/cf-check/diagnose.c b/cf-check/diagnose.c index 9b943cf555..b3b9611826 100644 --- a/cf-check/diagnose.c +++ b/cf-check/diagnose.c @@ -4,7 +4,8 @@ #if defined(__MINGW32__) || !defined(LMDB) -int diagnose_main(ARG_UNUSED int argc, ARG_UNUSED const char *const *const argv) +int diagnose_main( + ARG_UNUSED int argc, ARG_UNUSED const char *const *const argv) { Log(LOG_LEVEL_ERR, "cf-check diagnose not available on this platform/build"); @@ -287,8 +288,9 @@ static int fork_and_diagnose(const char *path) /** * @param[in] filenames DB files to diagnose/check - * @param[out] corrupt place to store the resulting sequence of corrupted files - * or %NULL (to only get the number of corrupted files) + * @param[out] corrupt place to store the resulting sequence of corrupted + * files or %NULL (to only get the number of corrupted + * files) * @param[in] foreground whether to run in foreground or fork (safer) * @return the number of the corrupted files */ From 1d2da5e9276dfb05298427ff12b861d6d259c564 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 15 Jan 2020 01:20:09 +0100 Subject: [PATCH 018/333] cf-check: Added data validation for cf_lastseen.lmdb Ticket: CFE-2988 Changelog: Title Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit cf0c636409f5163dcaf011177622b2fd718eb474) --- cf-check/Makefile.am | 3 +- cf-check/diagnose.c | 55 ++++- cf-check/diagnose.h | 12 +- cf-check/repair.c | 2 +- cf-check/validate.c | 524 ++++++++++++++++++++++++++++++++++++++++ cf-check/validate.h | 8 + libpromises/Makefile.am | 3 +- tests/unit/Makefile.am | 1 + 8 files changed, 591 insertions(+), 17 deletions(-) create mode 100644 cf-check/validate.c create mode 100644 cf-check/validate.h diff --git a/cf-check/Makefile.am b/cf-check/Makefile.am index 9d117fe8d4..58796a1d22 100644 --- a/cf-check/Makefile.am +++ b/cf-check/Makefile.am @@ -55,7 +55,8 @@ libcf_check_la_SOURCES = \ dump.c dump.h \ utilities.c utilities.h \ repair.c repair.h \ - replicate_lmdb.c replicate_lmdb.h + replicate_lmdb.c replicate_lmdb.h \ + validate.c validate.h if !BUILTIN_EXTENSIONS bin_PROGRAMS = cf-check diff --git a/cf-check/diagnose.c b/cf-check/diagnose.c index b3b9611826..ee19cf0f42 100644 --- a/cf-check/diagnose.c +++ b/cf-check/diagnose.c @@ -13,7 +13,10 @@ int diagnose_main( } size_t diagnose_files( - ARG_UNUSED const Seq *filenames, ARG_UNUSED Seq **corrupt, bool foreground) + ARG_UNUSED const Seq *filenames, + ARG_UNUSED Seq **corrupt, + ARG_UNUSED bool foreground, + ARG_UNUSED bool validate) { Log(LOG_LEVEL_INFO, "database diagnosis not available on this platform/build"); @@ -32,6 +35,7 @@ size_t diagnose_files( #include #include #include +#include #define CF_CHECK_CREATE_STRING(name) \ #name, @@ -224,10 +228,19 @@ int lmdb_errno_to_cf_check_code(int r) return s; } -static int diagnose(const char *path, bool temporary_redirect) +static int diagnose(const char *path, bool temporary_redirect, bool validate) { + // At this point we are already forked, we just need to decide 2 things: + // * Should output be redirected (to prevent spam)? + // * Which inner diagnose / validation function should be called? int ret; - if (temporary_redirect) + if (validate) + { + // --validate has meaningful output, so we don't want to redirect + // regardless of whether it's foreground or forked. + ret = CFCheck_Validate(path); + } + else if (temporary_redirect) { // --no-fork mode: temporarily redirect output to /dev/null & restore // Only done when necessary as it might not be so portable (/dev/fd) @@ -257,13 +270,13 @@ static int diagnose(const char *path, bool temporary_redirect) return ret; } -static int fork_and_diagnose(const char *path) +static int fork_and_diagnose(const char *path, bool validate) { const pid_t child_pid = fork(); if (child_pid == 0) { // Child - exit(diagnose(path, false)); + exit(diagnose(path, false, validate)); } else { @@ -294,7 +307,8 @@ static int fork_and_diagnose(const char *path) * @param[in] foreground whether to run in foreground or fork (safer) * @return the number of the corrupted files */ -size_t diagnose_files(const Seq *filenames, Seq **corrupt, bool foreground) +size_t diagnose_files( + const Seq *filenames, Seq **corrupt, bool foreground, bool validate) { size_t corruptions = 0; const size_t length = SeqLength(filenames); @@ -310,11 +324,12 @@ size_t diagnose_files(const Seq *filenames, Seq **corrupt, bool foreground) int r; if (foreground) { - r = lmdb_errno_to_cf_check_code(diagnose(filename, true)); + r = lmdb_errno_to_cf_check_code( + diagnose(filename, true, validate)); } else { - r = fork_and_diagnose(filename); + r = fork_and_diagnose(filename, validate); } Log(LOG_LEVEL_INFO, "Status of '%s': %s\n", @@ -348,11 +363,25 @@ int diagnose_main(int argc, const char *const *const argv) { size_t offset = 1; bool foreground = false; - if (StringSafeEqual(argv[1], "--no-fork") - || StringSafeEqual(argv[1], "-F")) + bool validate = false; + for (int i = offset; i < argc && argv[i][0] == '-'; ++i) { - foreground = true; - offset += 1; + if (StringMatchesOption(argv[i], "--no-fork", "-F")) + { + foreground = true; + offset += 1; + } + else if (StringMatchesOption(argv[i], "--validate", "-v")) + { + validate = true; + offset += 1; + } + else + { + assert(argv[i][0] == '-'); // For-loop condition + Log(LOG_LEVEL_ERR, "Unrecognized option: '%s'", argv[i]); + return 2; + } } Seq *files = argv_to_lmdb_files(argc, argv, offset); if (files == NULL || SeqLength(files) == 0) @@ -360,7 +389,7 @@ int diagnose_main(int argc, const char *const *const argv) Log(LOG_LEVEL_ERR, "No database files to diagnose"); return 1; } - const int ret = diagnose_files(files, NULL, foreground); + const int ret = diagnose_files(files, NULL, foreground, validate); SeqDestroy(files); return ret; } diff --git a/cf-check/diagnose.h b/cf-check/diagnose.h index 7c743e3ccf..e3cfe121af 100644 --- a/cf-check/diagnose.h +++ b/cf-check/diagnose.h @@ -3,6 +3,13 @@ #include +// Extensions of the errno range, for mixing with lmdb +// and operating system error codes +#define CF_CHECK_ERRNO_VALIDATE_FAILED -1 + +// cf-check has one canonical list of error codes +// which combines signals, system errno, lmdb errnos and +// cf-check specific errors: // clang-format off #define CF_CHECK_RUN_CODES(macro) \ macro(OK) \ @@ -60,6 +67,7 @@ macro(PID_ERROR) \ macro(PERMISSION_ERROR) \ macro(DOES_NOT_EXIST) \ + macro(VALIDATE_FAILED) \ macro(UNKNOWN) #define CF_CHECK_MAX CF_CHECK_UNKNOWN @@ -74,7 +82,9 @@ typedef enum { int lmdb_errno_to_cf_check_code(int r); int signal_to_cf_check_code(int sig); -size_t diagnose_files(const Seq *filenames, Seq **corrupt, bool foreground); + +size_t diagnose_files( + const Seq *filenames, Seq **corrupt, bool foreground, bool validate); int diagnose_main(int argc, const char *const *argv); #endif diff --git a/cf-check/repair.c b/cf-check/repair.c index a8c5c76fd0..adde34a87b 100644 --- a/cf-check/repair.c +++ b/cf-check/repair.c @@ -229,7 +229,7 @@ int repair_lmdb_files(Seq *files, bool force) } else { - const int corruptions = diagnose_files(files, &corrupt, false); + const int corruptions = diagnose_files(files, &corrupt, false, false); if (corruptions != 0) { assert(corrupt != NULL); diff --git a/cf-check/validate.c b/cf-check/validate.c new file mode 100644 index 0000000000..d18ba227c6 --- /dev/null +++ b/cf-check/validate.c @@ -0,0 +1,524 @@ +#include +#include +#include + +#if defined(__MINGW32__) || !defined(LMDB) + +int CFCheck_Validate(const char *path) +{ + Log(LOG_LEVEL_ERR, + "cf-check diagnose --validate not available on this platform/build"); + // Cannot include utilities.h on Windows: + return -1; // CF_CHECK_ERRNO_VALIDATE_FAILED +} + +#else + +#include // mdb_open(), mdb_close(), mdb_strerror() +#include // xcalloc(), xmemdup(), xstrdup() +#include // StringEndsWith() +#include // StringMap +#include // StringSet +#include // CF_CHECK_ERRNO_VALIDATE_FAILED + +static int lmdb_report_error(int rc) +{ + printf("err(%d): %s\n", rc, mdb_strerror(rc)); + return rc; +} + +typedef enum ValidatorMode +{ + CF_CHECK_VALIDATE_UNKNOWN, + CF_CHECK_VALIDATE_LASTSEEN, +} ValidatorMode; + +typedef struct LastSeenState +{ + StringMap *hostkey_to_address; + StringMap *address_to_hostkey; + StringSet *quality_outgoing_hostkeys; + StringSet *quality_incoming_hostkeys; +} LastSeenState; + +typedef struct ValidatorState +{ + const char *path; + ValidatorMode mode; + size_t errors; + StringSet *keys; + Seq *values; + union + { + LastSeenState lastseen; + }; +} ValidatorState; + +static Slice *NewLMDBSlice(void *data, size_t size) +{ + assert(data != NULL); + assert(size > 0); + + Slice *r = xcalloc(1, sizeof(Slice)); + r->size = size; + r->data = xmemdup(data, size); + + return r; +} + +static void DestroyLMDBSlice(Slice *value) +{ + if (value != NULL) + { + free(value->data); + free(value); + } +} + +static void NewValidator(const char *path, ValidatorState *state) +{ + assert(state != NULL); + + state->path = path; + state->errors = 0; + + state->keys = StringSetNew(); + state->values = SeqNew(0, &DestroyLMDBSlice); + + if (StringEndsWith(path, "cf_lastseen.lmdb")) + { + state->mode = CF_CHECK_VALIDATE_LASTSEEN; + state->lastseen.hostkey_to_address = StringMapNew(); + state->lastseen.address_to_hostkey = StringMapNew(); + state->lastseen.quality_outgoing_hostkeys = StringSetNew(); + state->lastseen.quality_incoming_hostkeys = StringSetNew(); + } + else + { + state->mode = CF_CHECK_VALIDATE_UNKNOWN; + } +} + +static void DestroyValidator(ValidatorState *state) +{ + // Since state is expected to be stack allocated, we don't allow NULL + // pointer. (We normally do for heap allocated data structures). + assert(state != NULL); + + state->path = NULL; + + StringSetDestroy(state->keys); + SeqDestroy(state->values); + + switch (state->mode) + { + case CF_CHECK_VALIDATE_LASTSEEN: + StringMapDestroy(state->lastseen.hostkey_to_address); + StringMapDestroy(state->lastseen.address_to_hostkey); + break; + case CF_CHECK_VALIDATE_UNKNOWN: + break; + default: + debug_abort_if_reached(); + break; + } +} + +static void va_ValidationError( + ValidatorState *state, const char *fmt, va_list ap) +{ + assert(state != NULL && state->path != NULL); + assert(fmt != NULL); + + printf("Error in %s: ", state->path); + vprintf(fmt, ap); + printf("\n"); + state->errors += 1; +} + +static void ValidationError(ValidatorState *state, const char *fmt, ...) + FUNC_ATTR_PRINTF(2, 3); + +static void ValidationError(ValidatorState *state, const char *fmt, ...) +{ + va_list ap; + va_start(ap, fmt); + va_ValidationError(state, fmt, ap); + va_end(ap); +} + +static bool ValidateString(ValidatorState *state, MDB_val string) +{ + assert(state != NULL); + + const char *const str = string.mv_data; + + if (strnlen(str, string.mv_size) != string.mv_size - 1) + { + ValidationError(state, "Invalid string - '%s'", str); + return false; + } + + if (string.mv_size == 1) + { + ValidationError(state, "Invalid string - empty"); + return false; + } + + return true; +} + +// Should be used after validating that hostkey is a NUL-terminated string +static bool ValidateHostkey(ValidatorState *state, const char *hostkey) +{ + if (EmptyString(hostkey)) + { + // For example a key of "k", missing the "kSHA=..." parts + ValidationError(state, "Empty hostkey - '%s'", hostkey); + return false; + } + + // Example hostkeys: + // MD5=14f11b956431e401ba60861402be3b9c + // SHA=e7f1fd5ea18f9d593641f4168d6140ab362f6c02cb1275e41faa17716db457ea + + if (StringStartsWith(hostkey, "SHA=")) + { + if (strlen(hostkey + 4) != 64) + { + ValidationError(state, "Bad length for hostkey - '%s'", hostkey); + return false; + } + } + else if (StringStartsWith(hostkey, "MD5=")) + { + if (strlen(hostkey + 4) != 32) + { + ValidationError(state, "Bad length for hostkey - '%s'", hostkey); + return false; + } + } + else + { + ValidationError(state, "Unknown format of hostkey - '%s'", hostkey); + return false; + } + + return true; +} + +static bool ValidateAddress(ValidatorState *state, const char *address) +{ + if (EmptyString(address)) + { + // For example a key of "a", missing the "a1.2.3.4" parts + ValidationError(state, "Empty IP address - '%s'", address); + return false; + } + + return true; +} + +static void UpdateValidatorLastseen( + ValidatorState *state, MDB_val key, MDB_val value) +{ + assert(state != NULL); + assert(key.mv_size > 0 && key.mv_data != NULL); + assert(value.mv_size > 0 && value.mv_data != NULL); + + const LastSeenState ls = state->lastseen; + StringMap *const hostkey_to_address = ls.hostkey_to_address; + StringMap *const address_to_hostkey = ls.address_to_hostkey; + StringSet *const quality_outgoing_hostkeys = ls.quality_outgoing_hostkeys; + StringSet *const quality_incoming_hostkeys = ls.quality_incoming_hostkeys; + + const char *key_string = key.mv_data; + + if (StringStartsWith(key_string, "k")) + { + if (!ValidateString(state, value)) + { + return; + } + + char *hostkey = xstrdup(key_string + 1); + char *address = xstrdup(value.mv_data); + + // This is an assert, because: + // * Only this branch adds to this data structure + // * UpdateValidator() has already checked for duplicate keys + // (Same applies to the other branches below.) + assert(StringMapGet(hostkey_to_address, hostkey) == NULL); + + StringMapInsert(hostkey_to_address, hostkey, address); + } + else if (StringStartsWith(key_string, "a")) + { + if (!ValidateString(state, value)) + { + return; + } + + char *address = xstrdup(key_string + 1); + char *hostkey = xstrdup(value.mv_data); + assert(StringMapGet(address_to_hostkey, address) == NULL); + StringMapInsert(address_to_hostkey, address, hostkey); + } + else if (StringStartsWith(key_string, "qo")) + { + const char *hostkey = key_string + 2; + assert(!StringSetContains(quality_outgoing_hostkeys, hostkey)); + StringSetAdd(quality_outgoing_hostkeys, xstrdup(hostkey)); + } + else if (StringStartsWith(key_string, "qi")) + { + const char *hostkey = key_string + 2; + assert(!StringSetContains(quality_incoming_hostkeys, hostkey)); + StringSetAdd(quality_incoming_hostkeys, xstrdup(hostkey)); + } +} + +static bool ValidateMDBValue( + ValidatorState *state, MDB_val value, const char *name) +{ + if (value.mv_size <= 0) + { + ValidationError(state, "0 size %s", name); + return false; + } + if (value.mv_data == NULL) + { + ValidationError(state, "NULL %s", name); + return false; + } + return true; +} + +static void UpdateValidator(ValidatorState *state, MDB_val key, MDB_val value) +{ + assert(state != NULL); + + if (!ValidateMDBValue(state, key, "key") || !ValidateString(state, key) + || !ValidateMDBValue(state, value, "value")) + { + return; + } + + const char *const key_string = key.mv_data; + + if (StringSetContains(state->keys, key_string)) + { + ValidationError(state, "Duplicate key - '%s'", key_string); + return; + } + + Log(LOG_LEVEL_DEBUG, + "LMDB validation: Adding key '%s'", + (const char *) key.mv_data); + + StringSetAdd(state->keys, xstrdup(key.mv_data)); + + SeqAppend(state->values, NewLMDBSlice(value.mv_data, value.mv_size)); + + switch (state->mode) + { + case CF_CHECK_VALIDATE_LASTSEEN: + UpdateValidatorLastseen(state, key, value); + break; + case CF_CHECK_VALIDATE_UNKNOWN: + break; + default: + debug_abort_if_reached(); + break; + } + + return; +} + +static void ValidateStateLastseen(ValidatorState *state) +{ + const LastSeenState ls = state->lastseen; + StringMap *const hostkey_to_address = ls.hostkey_to_address; + StringMap *const address_to_hostkey = ls.address_to_hostkey; + StringSet *const quality_outgoing_hostkeys = ls.quality_outgoing_hostkeys; + StringSet *const quality_incoming_hostkeys = ls.quality_incoming_hostkeys; + + assert(state != NULL); + + { + MapIterator iter = MapIteratorInit(hostkey_to_address->impl); + MapKeyValue *current_item; + while ((current_item = MapIteratorNext(&iter)) != NULL) + { + const char *hostkey = current_item->key; + if (!ValidateHostkey(state, hostkey)) + { + continue; + } + const char *address = current_item->value; + if (!ValidateAddress(state, address)) + { + continue; + } + const char *lookup = StringMapGet(address_to_hostkey, address); + if (lookup == NULL) + { + ValidationError( + state, "Missing address entry for '%s'", address); + } + else if (!StringSafeEqual(hostkey, lookup)) + { + ValidationError( + state, + "Bad hostkey->address->hostkey reverse lookup '%s' != '%s'", + hostkey, + lookup); + } + } + } + { + MapIterator iter = MapIteratorInit(address_to_hostkey->impl); + MapKeyValue *current_item; + while ((current_item = MapIteratorNext(&iter)) != NULL) + { + const char *address = current_item->key; + if (!ValidateAddress(state, address)) + { + continue; + } + const char *hostkey = current_item->value; + if (!ValidateHostkey(state, hostkey)) + { + continue; + } + const char *lookup = StringMapGet(hostkey_to_address, hostkey); + if (lookup == NULL) + { + ValidationError( + state, "Missing hostkey entry for '%s'", hostkey); + } + else if (!StringSafeEqual(address, lookup)) + { + ValidationError( + state, + "Bad address->hostkey->address reverse lookup '%s' != '%s'", + address, + lookup); + } + } + } + { + StringSetIterator iter = + StringSetIteratorInit(quality_incoming_hostkeys); + const char *hostkey; + while ((hostkey = StringSetIteratorNext(&iter)) != NULL) + { + if (StringMapGet(hostkey_to_address, hostkey) == NULL) + { + ValidationError( + state, + "Missing hostkey from quality-in entry '%s'", + hostkey); + } + } + } + { + StringSetIterator iter = + StringSetIteratorInit(quality_outgoing_hostkeys); + const char *hostkey; + while ((hostkey = StringSetIteratorNext(&iter)) != NULL) + { + if (StringMapGet(hostkey_to_address, hostkey) == NULL) + { + ValidationError( + state, + "Missing hostkey from quality-out entry '%s'", + hostkey); + } + } + } +} + +static void ValidateState(ValidatorState *state) +{ + assert(state != NULL); + + switch (state->mode) + { + case CF_CHECK_VALIDATE_LASTSEEN: + ValidateStateLastseen(state); + case CF_CHECK_VALIDATE_UNKNOWN: + break; + default: + debug_abort_if_reached(); + break; + } +} + +int CFCheck_Validate(const char *path) +{ + assert(path != NULL); + + MDB_env *env; + int rc = mdb_env_create(&env); + if (rc != 0) + { + return lmdb_report_error(rc); + } + + rc = mdb_env_open(env, path, MDB_NOSUBDIR | MDB_RDONLY, 0644); + if (rc != 0) + { + return lmdb_report_error(rc); + } + + MDB_txn *txn; + rc = mdb_txn_begin(env, NULL, MDB_RDONLY, &txn); + if (rc != 0) + { + return lmdb_report_error(rc); + } + + MDB_dbi dbi; + rc = mdb_open(txn, NULL, 0, &dbi); + if (rc != 0) + { + return lmdb_report_error(rc); + } + + MDB_cursor *cursor; + rc = mdb_cursor_open(txn, dbi, &cursor); + if (rc != 0) + { + return lmdb_report_error(rc); + } + + ValidatorState state; + NewValidator(path, &state); + MDB_val key, data; + while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == MDB_SUCCESS) + { + UpdateValidator(&state, key, data); + } + if (rc != MDB_NOTFOUND) + { + // At this point, not found is expected, anything else is an error + DestroyValidator(&state); + return lmdb_report_error(rc); + } + mdb_cursor_close(cursor); + mdb_close(env, dbi); + + mdb_txn_abort(txn); + mdb_env_close(env); + + ValidateState(&state); + const size_t errors = state.errors; + DestroyValidator(&state); + + // TODO: Find a better return code. + // This is mapped to errno, so 1 is definitely wrong + return (errors == 0) ? 0 : CF_CHECK_ERRNO_VALIDATE_FAILED; +} + +#endif diff --git a/cf-check/validate.h b/cf-check/validate.h new file mode 100644 index 0000000000..72fc2b5433 --- /dev/null +++ b/cf-check/validate.h @@ -0,0 +1,8 @@ +#ifndef CF_CHECK_VALIDATE_H +#define CF_CHECK_VALIDATE_H + +// Performs validation on single database file +// Returns 0 for success, errno code otherwise +int CFCheck_Validate(const char *path); + +#endif diff --git a/libpromises/Makefile.am b/libpromises/Makefile.am index 74e61f10fb..6fdded5a83 100644 --- a/libpromises/Makefile.am +++ b/libpromises/Makefile.am @@ -154,7 +154,8 @@ libpromises_la_SOURCES = \ ../cf-check/lmdump.c ../cf-check/lmdump.h \ ../cf-check/repair.c ../cf-check/repair.h \ ../cf-check/replicate_lmdb.c ../cf-check/replicate_lmdb.h \ - ../cf-check/utilities.c ../cf-check/utilities.h + ../cf-check/utilities.c ../cf-check/utilities.h \ + ../cf-check/validate.c ../cf-check/validate.h if !NT diff --git a/tests/unit/Makefile.am b/tests/unit/Makefile.am index bd00b27ac6..4e4ad674d6 100644 --- a/tests/unit/Makefile.am +++ b/tests/unit/Makefile.am @@ -92,6 +92,7 @@ libdb_la_SOURCES = db_stubs.c \ ../../cf-check/repair.c \ ../../cf-check/replicate_lmdb.c \ ../../cf-check/utilities.c \ + ../../cf-check/validate.c \ ../../libntech/libutils/logging.c \ ../../libntech/libutils/mutex.c \ ../../libntech/libutils/cleanup.c From db10f7f499d5e3e9175074e5a69aef82d4d1cbfb Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Mon, 20 Jan 2020 15:23:33 +0100 Subject: [PATCH 019/333] cf-check: Fixed recently introduced memory leak in --validate Found using valgrind: ``` valgrind --leak-check=full cf-check diagnose --validate /var/cfengine/state/cf_lastseen.lmdb ``` Unreleased, so no changelog entry. Changelog: None Ticket: CFE-2988 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit f41683db18106c009e4ee493d5134c38dd7546f2) --- cf-check/validate.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/cf-check/validate.c b/cf-check/validate.c index d18ba227c6..9d5078326a 100644 --- a/cf-check/validate.c +++ b/cf-check/validate.c @@ -115,6 +115,8 @@ static void DestroyValidator(ValidatorState *state) case CF_CHECK_VALIDATE_LASTSEEN: StringMapDestroy(state->lastseen.hostkey_to_address); StringMapDestroy(state->lastseen.address_to_hostkey); + StringSetDestroy(state->lastseen.quality_outgoing_hostkeys); + StringSetDestroy(state->lastseen.quality_incoming_hostkeys); break; case CF_CHECK_VALIDATE_UNKNOWN: break; From 76c2edb4cf3ae79cbb3dff81d813f8ea5779a96b Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Mon, 20 Jan 2020 15:55:44 +0100 Subject: [PATCH 020/333] cf-check: Added validation for timestamps in lastseen.lmdb Changelog: Title Ticket: CFE-2988 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 2dc96e2f5a227f021124637f614468c9343e7102) --- cf-check/validate.c | 50 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/cf-check/validate.c b/cf-check/validate.c index 9d5078326a..b784ee0954 100644 --- a/cf-check/validate.c +++ b/cf-check/validate.c @@ -20,6 +20,9 @@ int CFCheck_Validate(const char *path) #include // StringMap #include // StringSet #include // CF_CHECK_ERRNO_VALIDATE_FAILED +#include // KeyHostSeen + +#define CF_BIRTH 725846400 // Assume timestamps before 1993-01-01 are corrupt static int lmdb_report_error(int rc) { @@ -278,6 +281,53 @@ static void UpdateValidatorLastseen( assert(!StringSetContains(quality_incoming_hostkeys, hostkey)); StringSetAdd(quality_incoming_hostkeys, xstrdup(hostkey)); } + + if (key_string[0] == 'q') + { + const char direction = key_string[1]; + if (direction == 'i' || direction == 'o') + { + const KeyHostSeen *const data = value.mv_data; + + const time_t lastseen = data->lastseen; + const time_t current = time(NULL); + + Log(LOG_LEVEL_DEBUG, + "LMDB validation: Quality-entry lastseen time is %ju, current time is %ju", + (uintmax_t) lastseen, + (uintmax_t) current); + + if (current < CF_BIRTH) + { + ValidationError( + state, + "Current time (%ju) is before 1993-01-01", + (uintmax_t) current); + } + else if (lastseen < CF_BIRTH) + { + ValidationError( + state, + "Last seen time (%ju) is before 1993-01-01 (%s)", + (uintmax_t) lastseen, + key_string); + } + else if (lastseen > current) + { + ValidationError( + state, + "Future timestamp in last seen database: %ju > %ju (%s)", + (uintmax_t) lastseen, + (uintmax_t) current, + key_string); + } + } + else + { + ValidationError( + state, "Unexpected quality-entry key: %s", key_string); + } + } } static bool ValidateMDBValue( From 05cd5b4c8e0c61c4cbd3b5bc9da5f9cde69515f0 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Tue, 21 Jan 2020 13:10:10 +0100 Subject: [PATCH 021/333] cf-check: Added minimal validation for cf_changes.lmdb This database has non-string keys and values, and has not been known to cause a lot of issues, so I added a separate mode for very simplistic validation; only read the binary data. No changelog entry since this is part of --validate, another commit with changelog entry and it was not released yet. Changelog: None Ticket: CFE-2988 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit d5447ad121a8de8ef8876e2af58ba4208222736f) --- cf-check/validate.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/cf-check/validate.c b/cf-check/validate.c index b784ee0954..1165c5d72e 100644 --- a/cf-check/validate.c +++ b/cf-check/validate.c @@ -33,6 +33,7 @@ static int lmdb_report_error(int rc) typedef enum ValidatorMode { CF_CHECK_VALIDATE_UNKNOWN, + CF_CHECK_VALIDATE_MINIMAL, CF_CHECK_VALIDATE_LASTSEEN, } ValidatorMode; @@ -96,6 +97,10 @@ static void NewValidator(const char *path, ValidatorState *state) state->lastseen.quality_outgoing_hostkeys = StringSetNew(); state->lastseen.quality_incoming_hostkeys = StringSetNew(); } + else if (StringEndsWith(path, "cf_changes.lmdb")) + { + state->mode = CF_CHECK_VALIDATE_MINIMAL; + } else { state->mode = CF_CHECK_VALIDATE_UNKNOWN; @@ -121,6 +126,7 @@ static void DestroyValidator(ValidatorState *state) StringSetDestroy(state->lastseen.quality_outgoing_hostkeys); StringSetDestroy(state->lastseen.quality_incoming_hostkeys); break; + case CF_CHECK_VALIDATE_MINIMAL: case CF_CHECK_VALIDATE_UNKNOWN: break; default: @@ -350,6 +356,17 @@ static void UpdateValidator(ValidatorState *state, MDB_val key, MDB_val value) { assert(state != NULL); + if (state->mode == CF_CHECK_VALIDATE_MINIMAL) + { + // Databases with "weird" schemas, i.e. non-string keys, + // just check that we can read out the data: + void *key_copy = xmemdup(key.mv_data, key.mv_size); + void *value_copy = xmemdup(value.mv_data, value.mv_size); + free(key_copy); + free(value_copy); + return; + } + if (!ValidateMDBValue(state, key, "key") || !ValidateString(state, key) || !ValidateMDBValue(state, value, "value")) { @@ -500,6 +517,7 @@ static void ValidateState(ValidatorState *state) case CF_CHECK_VALIDATE_LASTSEEN: ValidateStateLastseen(state); case CF_CHECK_VALIDATE_UNKNOWN: + case CF_CHECK_VALIDATE_MINIMAL: break; default: debug_abort_if_reached(); From eb940428fa91f298a9a0a5c96ebcf1df816bdc94 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Tue, 21 Jan 2020 13:43:22 +0100 Subject: [PATCH 022/333] cf-check: Fixed child process exit codes On at least some systems (my Ubuntu 18 VM) exit codes can only be in the range 0-255. Thus exiting with -1 or -32000 is bad. The most sane thing to do is to exit with our custom cf-check error codes, which range from 0 to approximately 60, and then extended by system errno codes which are typically 0 - ~100. Note that signals are handled separately, exit code is only used if the process was not terminated by signals (SIGBUS, SIGSEGV, SIGABRT, ...). Changelog: None Ticket: CFE-2988 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit e161777b900dd6514fe658381f7b4eb6f8e7eb9c) --- cf-check/diagnose.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/cf-check/diagnose.c b/cf-check/diagnose.c index ee19cf0f42..e39466c5ef 100644 --- a/cf-check/diagnose.c +++ b/cf-check/diagnose.c @@ -214,6 +214,9 @@ int lmdb_errno_to_cf_check_code(int r) return CF_CHECK_LMDB_BAD_OR_INVALID_TRANSACTION; case MDB_BAD_VALSIZE: return CF_CHECK_LMDB_WRONG_KEY_OR_VALUE_SIZE; + // cf-check specific error codes: + case CF_CHECK_ERRNO_VALIDATE_FAILED: + return CF_CHECK_VALIDATE_FAILED; // Doesn't exist in earlier versions of LMDB: // case MDB_BAD_DBI: // return CF_CHECK_LMDB_BAD_DBI; @@ -267,7 +270,7 @@ static int diagnose(const char *path, bool temporary_redirect, bool validate) assert(f_result == stdout); ret = lmdump(LMDUMP_VALUES_ASCII, path); } - return ret; + return lmdb_errno_to_cf_check_code(ret); } static int fork_and_diagnose(const char *path, bool validate) @@ -289,7 +292,7 @@ static int fork_and_diagnose(const char *path, bool validate) } if (WIFEXITED(status) && WEXITSTATUS(status) != CF_CHECK_OK) { - return lmdb_errno_to_cf_check_code(WEXITSTATUS(status)); + return WEXITSTATUS(status); } if (WIFSIGNALED(status)) { @@ -324,8 +327,7 @@ size_t diagnose_files( int r; if (foreground) { - r = lmdb_errno_to_cf_check_code( - diagnose(filename, true, validate)); + r = diagnose(filename, true, validate); } else { From 0c790c288139e81a68f1cce003fc32d9d9743bdc Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Tue, 21 Jan 2020 14:03:39 +0100 Subject: [PATCH 023/333] cf-check: Stopped printing unnecessary error messages in --validate Changelog: None Ticket: CFE-2988 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 60e7f0c7c4c25bddd15dc9ed760a4e6dadfb3ade) --- cf-check/validate.c | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/cf-check/validate.c b/cf-check/validate.c index 1165c5d72e..c5f0437b7c 100644 --- a/cf-check/validate.c +++ b/cf-check/validate.c @@ -24,12 +24,6 @@ int CFCheck_Validate(const char *path) #define CF_BIRTH 725846400 // Assume timestamps before 1993-01-01 are corrupt -static int lmdb_report_error(int rc) -{ - printf("err(%d): %s\n", rc, mdb_strerror(rc)); - return rc; -} - typedef enum ValidatorMode { CF_CHECK_VALIDATE_UNKNOWN, @@ -533,34 +527,34 @@ int CFCheck_Validate(const char *path) int rc = mdb_env_create(&env); if (rc != 0) { - return lmdb_report_error(rc); + return rc; } rc = mdb_env_open(env, path, MDB_NOSUBDIR | MDB_RDONLY, 0644); if (rc != 0) { - return lmdb_report_error(rc); + return rc; } MDB_txn *txn; rc = mdb_txn_begin(env, NULL, MDB_RDONLY, &txn); if (rc != 0) { - return lmdb_report_error(rc); + return rc; } MDB_dbi dbi; rc = mdb_open(txn, NULL, 0, &dbi); if (rc != 0) { - return lmdb_report_error(rc); + return rc; } MDB_cursor *cursor; rc = mdb_cursor_open(txn, dbi, &cursor); if (rc != 0) { - return lmdb_report_error(rc); + return rc; } ValidatorState state; @@ -574,7 +568,7 @@ int CFCheck_Validate(const char *path) { // At this point, not found is expected, anything else is an error DestroyValidator(&state); - return lmdb_report_error(rc); + return rc; } mdb_cursor_close(cursor); mdb_close(env, dbi); From 8c3a239a7f860d6e97972dbe5cc553e9bfd7f2c0 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Tue, 21 Jan 2020 14:42:00 +0100 Subject: [PATCH 024/333] validate.c: Removed outdated comment Changelog: None Ticket: None Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 3d231f1e6903818da05411e8c413010f95c25f69) --- cf-check/validate.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/cf-check/validate.c b/cf-check/validate.c index c5f0437b7c..c7a5d27d18 100644 --- a/cf-check/validate.c +++ b/cf-check/validate.c @@ -580,8 +580,6 @@ int CFCheck_Validate(const char *path) const size_t errors = state.errors; DestroyValidator(&state); - // TODO: Find a better return code. - // This is mapped to errno, so 1 is definitely wrong return (errors == 0) ? 0 : CF_CHECK_ERRNO_VALIDATE_FAILED; } From 8fb131f1bbc6b98ccf2f2107e70ff6974b748ab0 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Tue, 21 Jan 2020 22:14:31 +0100 Subject: [PATCH 025/333] cf-check: Fixed issue causing repair to target the wrong database file The new replicate logic was picking lmdb files from the original, full list of files, not the filtered list of corrupted LMDB files. Changelog: Title Ticket: ENT-5309 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 6a75937cd4742c2b5bd6767d30cb6c62b5d76180) --- cf-check/repair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cf-check/repair.c b/cf-check/repair.c index adde34a87b..5973a3f4bd 100644 --- a/cf-check/repair.c +++ b/cf-check/repair.c @@ -251,7 +251,7 @@ int repair_lmdb_files(Seq *files, bool force) backup_files_copy(corrupt); for (int i = 0; i < length; ++i) { - const char *file = SeqAt(files, i); + const char *file = SeqAt(corrupt, i); if (repair_lmdb_file(file, -1) == -1) { ret++; From 1df2ffaab8f4b8a7b4c962c807ea15fc9363c81f Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Tue, 21 Jan 2020 22:18:56 +0100 Subject: [PATCH 026/333] cf-check: Added validation for timestamps in cf_lock.lmdb Changelog: Title Ticket: CFE-2988 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 4ba581c7843a9261dbc81ea940fdbd9e67f078be) --- cf-check/validate.c | 54 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/cf-check/validate.c b/cf-check/validate.c index c7a5d27d18..9168ed3c6a 100644 --- a/cf-check/validate.c +++ b/cf-check/validate.c @@ -28,6 +28,7 @@ typedef enum ValidatorMode { CF_CHECK_VALIDATE_UNKNOWN, CF_CHECK_VALIDATE_MINIMAL, + CF_CHECK_VALIDATE_LOCK, CF_CHECK_VALIDATE_LASTSEEN, } ValidatorMode; @@ -95,6 +96,10 @@ static void NewValidator(const char *path, ValidatorState *state) { state->mode = CF_CHECK_VALIDATE_MINIMAL; } + else if (StringEndsWith(path, "cf_lock.lmdb")) + { + state->mode = CF_CHECK_VALIDATE_LOCK; + } else { state->mode = CF_CHECK_VALIDATE_UNKNOWN; @@ -121,6 +126,7 @@ static void DestroyValidator(ValidatorState *state) StringSetDestroy(state->lastseen.quality_incoming_hostkeys); break; case CF_CHECK_VALIDATE_MINIMAL: + case CF_CHECK_VALIDATE_LOCK: case CF_CHECK_VALIDATE_UNKNOWN: break; default: @@ -330,6 +336,50 @@ static void UpdateValidatorLastseen( } } +static void UpdateValidatorLock( + ValidatorState *state, MDB_val key, MDB_val value) +{ + assert(state != NULL); + assert(key.mv_size > 0 && key.mv_data != NULL); + assert(value.mv_size > 0 && value.mv_data != NULL); + + const char *key_string = key.mv_data; + + const LockData *const lock = value.mv_data; + const time_t lock_time = lock->time; + const time_t current = time(NULL); + + Log(LOG_LEVEL_DEBUG, + "LMDB validation: Lock time is %ju, current time is %ju", + (uintmax_t) lock_time, + (uintmax_t) current); + + if (current < CF_BIRTH) + { + ValidationError( + state, + "Current time (%ju) is before 1993-01-01", + (uintmax_t) current); + } + else if (lock_time < CF_BIRTH) + { + ValidationError( + state, + "Lock time (%ju) is before 1993-01-01 (%s)", + (uintmax_t) lock_time, + key_string); + } + else if (lock_time > current) + { + ValidationError( + state, + "Future timestamp in lock database: %ju > %ju (%s)", + (uintmax_t) lock_time, + (uintmax_t) current, + key_string); + } +} + static bool ValidateMDBValue( ValidatorState *state, MDB_val value, const char *name) { @@ -388,6 +438,9 @@ static void UpdateValidator(ValidatorState *state, MDB_val key, MDB_val value) case CF_CHECK_VALIDATE_LASTSEEN: UpdateValidatorLastseen(state, key, value); break; + case CF_CHECK_VALIDATE_LOCK: + UpdateValidatorLock(state, key, value); + break; case CF_CHECK_VALIDATE_UNKNOWN: break; default: @@ -511,6 +564,7 @@ static void ValidateState(ValidatorState *state) case CF_CHECK_VALIDATE_LASTSEEN: ValidateStateLastseen(state); case CF_CHECK_VALIDATE_UNKNOWN: + case CF_CHECK_VALIDATE_LOCK: case CF_CHECK_VALIDATE_MINIMAL: break; default: From 48876b2e801dcdbd53249eb9b150601c1a2385f7 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Sat, 1 Feb 2020 15:31:53 +0100 Subject: [PATCH 027/333] Fixed memory leak in handling of inline JSON in policy evaluation RlistAppendScalar already calls `xstrdup`. Found by running binaries (bootstrap) built with AddressSanitizer (ASAN) enabled. Changelog: Title Ticket: None Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 84d4fa5008f84e43eb03f0174d9ac904cf420176) --- libpromises/cf3parse.y | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libpromises/cf3parse.y b/libpromises/cf3parse.y index 6c4ccbe560..1c2475e823 100644 --- a/libpromises/cf3parse.y +++ b/libpromises/cf3parse.y @@ -658,7 +658,7 @@ constraint: constraint_id /* BUNDLE ONLY */ if (json == NULL) { Rlist *synthetic_args = NULL; - RlistAppendScalar(&synthetic_args, xstrdup(P.rval.item)); + RlistAppendScalar(&synthetic_args, P.rval.item); RvalDestroy(P.rval); P.rval = (Rval) { FnCallNew(fname, synthetic_args), RVAL_TYPE_FNCALL }; From 955da7b2a2212a5dd034ea82c3939105b70ba728 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Sun, 2 Feb 2020 14:10:56 +0100 Subject: [PATCH 028/333] libntech: Added SeqStringReadFile() and SeqStringWriteFile() Signed-off-by: Ole Herman Schumacher Elgesem --- libntech | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libntech b/libntech index 683121898e..f197da682e 160000 --- a/libntech +++ b/libntech @@ -1 +1 @@ -Subproject commit 683121898e4056d821ffad372132ab8eed481352 +Subproject commit f197da682e2f4e903f5db82396ffb9b2f8253efa From 289efc177a3eeb97c5735ba2f64a67472fd872af Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 29 Jan 2020 08:27:54 +0100 Subject: [PATCH 029/333] Make the report_mdb_error() function reusable in cf-check It's useful in multiple commands/files in cf-check. Ticket: ENT-4484 Changelog: None (cherry picked from commit ce891ed565a5a93e831b98a8fd44ef127bcbee1e) --- cf-check/diagnose.c | 6 ++++++ cf-check/diagnose.h | 1 + cf-check/replicate_lmdb.c | 6 ------ 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/cf-check/diagnose.c b/cf-check/diagnose.c index e39466c5ef..91b6193275 100644 --- a/cf-check/diagnose.c +++ b/cf-check/diagnose.c @@ -231,6 +231,12 @@ int lmdb_errno_to_cf_check_code(int r) return s; } +void report_mdb_error(const char *db_file, const char *op, int rc) +{ + Log(LOG_LEVEL_ERR, "%s: %s error(%d): %s\n", + db_file, op, rc, mdb_strerror(rc)); +} + static int diagnose(const char *path, bool temporary_redirect, bool validate) { // At this point we are already forked, we just need to decide 2 things: diff --git a/cf-check/diagnose.h b/cf-check/diagnose.h index e3cfe121af..ce8132e895 100644 --- a/cf-check/diagnose.h +++ b/cf-check/diagnose.h @@ -82,6 +82,7 @@ typedef enum { int lmdb_errno_to_cf_check_code(int r); int signal_to_cf_check_code(int sig); +void report_mdb_error(const char *db_file, const char *op, int rc); size_t diagnose_files( const Seq *filenames, Seq **corrupt, bool foreground, bool validate); diff --git a/cf-check/replicate_lmdb.c b/cf-check/replicate_lmdb.c index b86ab71d24..c418bdc79b 100644 --- a/cf-check/replicate_lmdb.c +++ b/cf-check/replicate_lmdb.c @@ -21,12 +21,6 @@ typedef struct { MDB_txn *d_txn; } LMDBReplicationInfo; -static void report_mdb_error(const char *db_file, const char *op, int rc) -{ - Log(LOG_LEVEL_ERR, "%s: %s error(%d): %s\n", - db_file, op, rc, mdb_strerror(rc)); -} - static void HandleSrcLMDBCorruption(MDB_env *env, const char *msg) { LMDBReplicationInfo *info = mdb_env_get_userctx(env); From a0c0dae6de95847ab94328f6365db4b8957d94fd Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 29 Jan 2020 08:33:49 +0100 Subject: [PATCH 030/333] Add '--test-write' to the 'cf-check diagnose' command LMDB files can be corrupted in a way that they allow reading, but writing is impossible. 'cf-check diagnose' should thus support trying writing some data into the diagnosed files. Ticket: ENT-4484 Changelog: 'cf-check diagnose --test-write' can now be used to test writing into LMDB files (cherry picked from commit 6d340ee929937b3096b97a12fd6bdf86f2c25ac9) --- cf-check/Makefile.am | 6 +- cf-check/diagnose.c | 226 +++++++++++++++++++++++++++++++++++++++++-- cf-check/diagnose.h | 2 +- cf-check/repair.c | 2 +- 4 files changed, 226 insertions(+), 10 deletions(-) diff --git a/cf-check/Makefile.am b/cf-check/Makefile.am index 58796a1d22..e86269e09b 100644 --- a/cf-check/Makefile.am +++ b/cf-check/Makefile.am @@ -27,7 +27,8 @@ AM_CPPFLAGS = -I$(srcdir)/../libntech/libutils \ -I$(srcdir)/../libntech/libcompat \ $(PCRE_CPPFLAGS) \ $(LIBYAML_CPPFLAGS) \ - $(LMDB_CPPFLAGS) + $(LMDB_CPPFLAGS) \ + $(OPENSSL_CPPFLAGS) AM_CFLAGS = \ $(LMDB_CFLAGS) \ @@ -44,7 +45,8 @@ libcf_check_la_LIBADD = $(srcdir)/../libntech/libutils/libutils.la \ $(LMDB_LIBS) \ $(PCRE_LIBS) \ $(LIBYAML_LIBS) \ - $(PTHREAD_LIBS) + $(PTHREAD_LIBS) \ + $(OPENSSL_LIBS) libcf_check_la_SOURCES = \ backup.c backup.h \ diff --git a/cf-check/diagnose.c b/cf-check/diagnose.c index 91b6193275..cbf80f257f 100644 --- a/cf-check/diagnose.c +++ b/cf-check/diagnose.c @@ -16,7 +16,8 @@ size_t diagnose_files( ARG_UNUSED const Seq *filenames, ARG_UNUSED Seq **corrupt, ARG_UNUSED bool foreground, - ARG_UNUSED bool validate) + ARG_UNUSED bool validate, + ARG_UNUSED bool test_write) { Log(LOG_LEVEL_INFO, "database diagnosis not available on this platform/build"); @@ -36,6 +37,7 @@ size_t diagnose_files( #include #include #include +#include #define CF_CHECK_CREATE_STRING(name) \ #name, @@ -279,13 +281,210 @@ static int diagnose(const char *path, bool temporary_redirect, bool validate) return lmdb_errno_to_cf_check_code(ret); } -static int fork_and_diagnose(const char *path, bool validate) +static int diagnose_write(const char *path) +{ + MDB_env *env = NULL; + MDB_txn *txn = NULL; + MDB_dbi dbi; + bool close_dbi = false; + MDB_cursor *cursor = NULL; + + /* We need to initialize these to NULL here so that we can safely call + * free() on them in cleanup. */ + MDB_val new_key, new_data; + new_key.mv_data = NULL; + new_data.mv_data = NULL; + + int ret = 0; + int rc; + + Log(LOG_LEVEL_INFO, "Trying to write data into '%s'", path); + + rc = mdb_env_create(&env); + if (rc != MDB_SUCCESS) + { + ret = rc; + report_mdb_error(path, "mdb_env_create", rc); + goto cleanup; + } + + rc = mdb_env_open(env, path, MDB_NOSUBDIR, 0600); + if (rc != MDB_SUCCESS) + { + ret = rc; + report_mdb_error(path, "mdb_env_open", rc); + goto cleanup; + } + + rc = mdb_txn_begin(env, NULL, 0, &txn); + if (rc != MDB_SUCCESS) + { + ret = rc; + report_mdb_error(path, "mdb_txn_begin", rc); + goto cleanup; + } + + rc = mdb_dbi_open(txn, NULL, 0, &dbi); + if (rc != MDB_SUCCESS) + { + ret = rc; + report_mdb_error(path, "mdb_dbi_open", rc); + goto cleanup; + } + else + { + close_dbi = true; + } + + rc = mdb_cursor_open(txn, dbi, &cursor); + if (rc != MDB_SUCCESS) + { + ret = rc; + report_mdb_error(path, "mdb_cursor_open", rc); + goto cleanup; + } + + MDB_val key, data; + rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT); + /* MDB_NOTFOUND => no more data */ + if (rc == MDB_NOTFOUND) + { + Log(LOG_LEVEL_INFO, + "'%s' is empty, no data to use as a template, cannot test writing", + path); + ret = 0; + goto cleanup; + } + else if (rc != MDB_SUCCESS) + { + report_mdb_error(path, "mdb_cursor_get", rc); + ret = rc; + goto cleanup; + } + /* else */ + new_key.mv_size = key.mv_size; + new_data.mv_size = data.mv_size; + new_key.mv_data = xmalloc(new_key.mv_size); + new_data.mv_data = xmalloc(new_data.mv_size); + + rc = RAND_bytes((char *) new_key.mv_data, new_key.mv_size); + if (rc != 1) + { + Log(LOG_LEVEL_ERR, "Failed to generate random key data"); + ret = -1; + goto cleanup; + } + rc = RAND_bytes((char *) new_data.mv_data, new_data.mv_size); + if (rc != 1) + { + Log(LOG_LEVEL_ERR, "Failed to generate random value data"); + ret = -1; + goto cleanup; + } + rc = mdb_put(txn, dbi, &new_key, &new_data, 0); + if (rc != MDB_SUCCESS) + { + report_mdb_error(path, "mdb_put", rc); + Log(LOG_LEVEL_ERR, "Failed to write new data into '%s'", path); + ret = rc; + goto cleanup; + } + + rc = mdb_txn_commit(txn); + if (rc != MDB_SUCCESS) + { + report_mdb_error(path, "mdb_txn_commit", rc); + Log(LOG_LEVEL_ERR, "Failed to commit new data into '%s'", path); + ret = rc; + goto cleanup; + } + txn = NULL; + + mdb_close(env, dbi); + close_dbi = false; + + rc = mdb_txn_begin(env, NULL, 0, &txn); + if (rc != MDB_SUCCESS) + { + ret = rc; + report_mdb_error(path, "mdb_txn_begin", rc); + goto cleanup; + } + + rc = mdb_dbi_open(txn, NULL, 0, &dbi); + if (rc != MDB_SUCCESS) + { + ret = rc; + report_mdb_error(path, "mdb_dbi_open", rc); + goto cleanup; + } + else + { + close_dbi = true; + } + + rc = mdb_del(txn, dbi, &new_key, NULL); + if (rc != MDB_SUCCESS) + { + report_mdb_error(path, "mdb_del", rc); + Log(LOG_LEVEL_ERR, "Failed to delete new data from '%s'", path); + ret = rc; + goto cleanup; + } + + rc = mdb_txn_commit(txn); + if (rc != MDB_SUCCESS) + { + report_mdb_error(path, "mdb_txn_commit", rc); + Log(LOG_LEVEL_ERR, "Failed to commit removal of new data from '%s'", path); + ret = rc; + goto cleanup; + } + txn = NULL; + + mdb_close(env, dbi); + close_dbi = false; + + cleanup: + free(new_key.mv_data); + free(new_data.mv_data); + + if (cursor != NULL) + { + mdb_cursor_close(cursor); + } + if (close_dbi) + { + mdb_close(env, dbi); + } + if (txn != NULL) + { + mdb_txn_abort(txn); + } + if (env != NULL) + { + mdb_env_close(env); + } + + ret = lmdb_errno_to_cf_check_code(ret); + return ret; +} + +static int fork_and_diagnose(const char *path, bool validate, bool test_write) { const pid_t child_pid = fork(); if (child_pid == 0) { // Child - exit(diagnose(path, false, validate)); + /* The second argument is 'temporary_redirect' and we require a + * temporary redirect if we want to test writability because that + * produces output. */ + int r = diagnose(path, test_write, validate); + if ((r == CF_CHECK_OK) && test_write) + { + r = diagnose_write(path); + } + exit(r); } else { @@ -314,10 +513,15 @@ static int fork_and_diagnose(const char *path, bool validate) * files or %NULL (to only get the number of corrupted * files) * @param[in] foreground whether to run in foreground or fork (safer) + * @param[in] test_write whether to test writing into the DB * @return the number of the corrupted files */ size_t diagnose_files( - const Seq *filenames, Seq **corrupt, bool foreground, bool validate) + const Seq *filenames, + Seq **corrupt, + bool foreground, + bool validate, + bool test_write) { size_t corruptions = 0; const size_t length = SeqLength(filenames); @@ -334,10 +538,14 @@ size_t diagnose_files( if (foreground) { r = diagnose(filename, true, validate); + if ((r == CF_CHECK_OK) && test_write) + { + r = diagnose_write(filename); + } } else { - r = fork_and_diagnose(filename, validate); + r = fork_and_diagnose(filename, validate, test_write); } Log(LOG_LEVEL_INFO, "Status of '%s': %s\n", @@ -372,6 +580,7 @@ int diagnose_main(int argc, const char *const *const argv) size_t offset = 1; bool foreground = false; bool validate = false; + bool test_write = false; for (int i = offset; i < argc && argv[i][0] == '-'; ++i) { if (StringMatchesOption(argv[i], "--no-fork", "-F")) @@ -384,6 +593,11 @@ int diagnose_main(int argc, const char *const *const argv) validate = true; offset += 1; } + else if (StringMatchesOption(argv[i], "--test-write", "-w")) + { + test_write = true; + offset += 1; + } else { assert(argv[i][0] == '-'); // For-loop condition @@ -397,7 +611,7 @@ int diagnose_main(int argc, const char *const *const argv) Log(LOG_LEVEL_ERR, "No database files to diagnose"); return 1; } - const int ret = diagnose_files(files, NULL, foreground, validate); + const int ret = diagnose_files(files, NULL, foreground, validate, test_write); SeqDestroy(files); return ret; } diff --git a/cf-check/diagnose.h b/cf-check/diagnose.h index ce8132e895..923e2431ce 100644 --- a/cf-check/diagnose.h +++ b/cf-check/diagnose.h @@ -85,7 +85,7 @@ int signal_to_cf_check_code(int sig); void report_mdb_error(const char *db_file, const char *op, int rc); size_t diagnose_files( - const Seq *filenames, Seq **corrupt, bool foreground, bool validate); + const Seq *filenames, Seq **corrupt, bool foreground, bool validate, bool test_write); int diagnose_main(int argc, const char *const *argv); #endif diff --git a/cf-check/repair.c b/cf-check/repair.c index 5973a3f4bd..c51f3145af 100644 --- a/cf-check/repair.c +++ b/cf-check/repair.c @@ -229,7 +229,7 @@ int repair_lmdb_files(Seq *files, bool force) } else { - const int corruptions = diagnose_files(files, &corrupt, false, false); + const int corruptions = diagnose_files(files, &corrupt, false, false, false); if (corruptions != 0) { assert(corrupt != NULL); From c02f0be090f2ec3b387b2bf86ccca0347412ad22 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Thu, 23 Jan 2020 20:11:33 +0100 Subject: [PATCH 031/333] cf-agent: files: report purged dirs and files as repaired Given a recursive file promise with purging enabled, the promise will not be propagated as repaired if the target directory contains some directory hierarchy that is removed by the promise. While the function `PurgeLocalFiles` correctly keeps track of any failed attempts to remove files from the target directory, it does not report the repaired promise in case it removes files or directories from the target directory successfully. Fix the issue by properly updating the promise to PROMISE_RESULT_CHANGE if puring either a directory or file successfully. Add an acceptance test to verify behaviour. Changelog: Title Ticket: CFE-3260 Signed-off-by: Patrick Steinhardt (cherry picked from commit 24fbbca3921d34e205342a303e74234870496ca5) --- cf-agent/verify_files_utils.c | 12 ++++- .../10_files/purge_reports_as_repaired.cf | 48 +++++++++++++++++++ 2 files changed, 58 insertions(+), 2 deletions(-) create mode 100644 tests/acceptance/10_files/purge_reports_as_repaired.cf diff --git a/cf-agent/verify_files_utils.c b/cf-agent/verify_files_utils.c index 08f3043c79..4ed5284e4e 100644 --- a/cf-agent/verify_files_utils.c +++ b/cf-agent/verify_files_utils.c @@ -687,8 +687,6 @@ static PromiseResult PurgeLocalFiles(EvalContext *ctx, Item *filelist, const cha } else { - Log(LOG_LEVEL_INFO, "Purging '%s' in copy dest directory", filename); - if (lstat(filename, &sb) == -1) { cfPS(ctx, LOG_LEVEL_VERBOSE, PROMISE_RESULT_INTERRUPTED, pp, attr, "Couldn't stat '%s' while purging. (lstat: %s)", @@ -710,12 +708,22 @@ static PromiseResult PurgeLocalFiles(EvalContext *ctx, Item *filelist, const cha result = PromiseResultUpdate(result, PROMISE_RESULT_FAIL); } } + else + { + cfPS(ctx, LOG_LEVEL_INFO, PROMISE_RESULT_CHANGE, pp, attr, "Purged directory '%s' in copy dest directory", filename); + result = PromiseResultUpdate(result, PROMISE_RESULT_CHANGE); + } } else if (unlink(filename) == -1) { cfPS(ctx, LOG_LEVEL_ERR, PROMISE_RESULT_FAIL, pp, attr, "Couldn't delete '%s' while purging", filename); result = PromiseResultUpdate(result, PROMISE_RESULT_FAIL); } + else + { + cfPS(ctx, LOG_LEVEL_INFO, PROMISE_RESULT_CHANGE, pp, attr, "Purged file '%s' in copy dest directory", filename); + result = PromiseResultUpdate(result, PROMISE_RESULT_CHANGE); + } } } } diff --git a/tests/acceptance/10_files/purge_reports_as_repaired.cf b/tests/acceptance/10_files/purge_reports_as_repaired.cf new file mode 100644 index 0000000000..8483caae6b --- /dev/null +++ b/tests/acceptance/10_files/purge_reports_as_repaired.cf @@ -0,0 +1,48 @@ +bundle common test_meta +{ + vars: + "description" string => "Test that purging a directory in a target propagates promise as repaired"; +} + +####################################################### + +body common control +{ + inputs => { "../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +####################################################### + +bundle agent init +{ + vars: + "dirs" slist => { + "$(G.testdir)/source/.", + "$(G.testdir)/target/.", + "$(G.testdir)/target/subdir/." + }; + + files: + "$(dirs)" create => "true"; +} + +bundle agent test +{ + files: + "$(G.testdir)/target/" copy_from => copyfrom_sync("$(G.testdir)/source/."), + depth_search => recurse("inf"), + classes => if_repaired("purge_propagated"); +} + +####################################################### + +bundle agent check +{ + reports: + purge_propagated:: + "$(this.promise_filename) Pass"; + !purge_propagated:: + "$(this.promise_filename) FAIL"; +} From be454d21fa1fd4818580ee6b7302531609ba762a Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 5 Feb 2020 10:21:02 +0100 Subject: [PATCH 032/333] Autotools: Pick up flags from environment when building cf-check Needed for compiling with ASAN, using env vars. Changelog: None Ticket: None Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 4209ed6b243edb9437f0655e64945a868e3cc881) --- cf-check/Makefile.am | 3 +++ 1 file changed, 3 insertions(+) diff --git a/cf-check/Makefile.am b/cf-check/Makefile.am index e86269e09b..a7ebe4a2c0 100644 --- a/cf-check/Makefile.am +++ b/cf-check/Makefile.am @@ -25,18 +25,21 @@ noinst_LTLIBRARIES = libcf-check.la AM_CPPFLAGS = -I$(srcdir)/../libntech/libutils \ -I$(srcdir)/../libntech/libcompat \ + @CPPFLAGS@ \ $(PCRE_CPPFLAGS) \ $(LIBYAML_CPPFLAGS) \ $(LMDB_CPPFLAGS) \ $(OPENSSL_CPPFLAGS) AM_CFLAGS = \ + @CFLAGS@ \ $(LMDB_CFLAGS) \ $(PCRE_CFLAGS) \ $(LIBYAML_CFLAGS) \ $(PTHREAD_CFLAGS) AM_LDFLAGS = \ + @LDFLAGS@ \ $(PCRE_LDFLAGS) \ $(LIBYAML_LDFLAGS) \ $(LMDB_LDFLAGS) From 18f08dd733eaf07fad1640352e8fe2ff4a0e808c Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 5 Feb 2020 10:54:19 +0100 Subject: [PATCH 033/333] libntech: Added stdbool.h to condition_macros.h Changelog: None Ticket: None Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 2698a9853dfd94e16b685df6cddaf727ede5047c) --- libntech | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libntech b/libntech index f197da682e..79ce1efc5c 160000 --- a/libntech +++ b/libntech @@ -1 +1 @@ -Subproject commit f197da682e2f4e903f5db82396ffb9b2f8253efa +Subproject commit 79ce1efc5cb2b434a1d1e42f02d773de4210dfda From 40d9b69ef8387025307da6a9a1f91cded9161d5b Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 22 Jan 2020 16:05:33 +0100 Subject: [PATCH 034/333] Travis: Allow mac builds to fail until issue is resolved Mac builds started failing without any changes in our code. Most likely because of changes in one of the dependencies, or the travis environment. Let's delay a while and see if the issue is resolved automatically. Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 828a73f52b580ef4b7f25b91f26df12b46c3b0ef) --- .travis.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.travis.yml b/.travis.yml index f076c714c1..1d21b70cb4 100644 --- a/.travis.yml +++ b/.travis.yml @@ -52,6 +52,10 @@ matrix: - os: osx env: JOB_TYPE=compile_and_unit_test COVERAGE=no sudo: false + allow_failures: + - os: osx + env: JOB_TYPE=compile_and_unit_test COVERAGE=no + sudo: false before_install: - chmod ug+x ./travis-scripts/* From a53c4418dbb987a1fa3a9af0fa182d14008afdca Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 5 Feb 2020 11:55:39 +0100 Subject: [PATCH 035/333] Renamed macro name in lexer Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit aa5921545c8852899332e2b262aa5b44285e99b8) --- libpromises/cf3lex.l | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/libpromises/cf3lex.l b/libpromises/cf3lex.l index 85e4f04509..32780c9e31 100644 --- a/libpromises/cf3lex.l +++ b/libpromises/cf3lex.l @@ -87,8 +87,8 @@ eat_line ([\n]|[\xd][\xa]).* comment #[^\n]* -macro_if_version @if\ minimum_version\([0-9]{1,10}\.[0-9]{1,10}(\.[0-9]{1,10})?\) -macro_if_feature @if\ feature\([a-zA-Z0-9_]+\) +macro_if_minimum_version @if\ minimum_version\([0-9]{1,10}\.[0-9]{1,10}(\.[0-9]{1,10})?\) +macro_if_feature @if\ feature\([a-zA-Z0-9_]+\) macro_endif @endif macro_line [^@\n\xd\xa].* @@ -146,7 +146,8 @@ promise_type [a-zA-Z_]+: yyless(1); } -{macro_if_version} { +{macro_if_minimum_version} + { if ( P.line_pos != 1 ) { yyerror("fatal: macro @if must be at beginning of the line."); From 5e648ee27affb0c87b2d1c5f2230a9e79cf17090 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 5 Feb 2020 11:55:55 +0100 Subject: [PATCH 036/333] Whitespace Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 113ff74739d4496aafa26a60619e0b309460ff7c) --- libpromises/cf3lex.l | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libpromises/cf3lex.l b/libpromises/cf3lex.l index 32780c9e31..e183960b44 100644 --- a/libpromises/cf3lex.l +++ b/libpromises/cf3lex.l @@ -367,7 +367,7 @@ promise_type [a-zA-Z_]+: {varclass} { char *tmp = NULL; - + P.line_pos += yyleng; ParserDebug("\tL:varclass %s %d\n", yytext, P.line_pos); @@ -394,7 +394,7 @@ promise_type [a-zA-Z_]+: {class} { char *tmp = NULL; - + P.line_pos += yyleng; ParserDebug("\tL:class %s %d\n", yytext, P.line_pos); if (context_expression_whitespace_rx == NULL) From fdcaaa0ac8c672c85108ff0967a11519b3e81bac Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 5 Feb 2020 13:34:54 +0100 Subject: [PATCH 037/333] Refactored minimum_version macro Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 727212b4de7541dc7726e114056c8704dcac52b7) --- libntech | 2 +- libpromises/cf3lex.l | 78 +++++++++++++------------------------------- 2 files changed, 24 insertions(+), 56 deletions(-) diff --git a/libntech b/libntech index 79ce1efc5c..6babc76088 160000 --- a/libntech +++ b/libntech @@ -1 +1 @@ -Subproject commit 79ce1efc5cb2b434a1d1e42f02d773de4210dfda +Subproject commit 6babc7608811c43f99354696062c4c831aa9bfba diff --git a/libpromises/cf3lex.l b/libpromises/cf3lex.l index e183960b44..569740e713 100644 --- a/libpromises/cf3lex.l +++ b/libpromises/cf3lex.l @@ -36,6 +36,7 @@ #include #include #include +#include } %{ @@ -146,70 +147,37 @@ promise_type [a-zA-Z_]+: yyless(1); } -{macro_if_minimum_version} - { +{macro_if_minimum_version} { if ( P.line_pos != 1 ) { yyerror("fatal: macro @if must be at beginning of the line."); return 0; } - - const char* version_text = yytext+20; - ParserDebug("\tL:macro @if %d:version=%s\n", P.line_pos, version_text); + if (P.if_depth > 0) { - int request_major = 0; - int request_minor = 0; - int request_patch = 0; - int request_level = 0; - int major = 0; - int minor = 0; - int patch = 0; + yyerror("fatal: nested @if macros are not allowed"); + return 0; + } - if (P.if_depth > 0) - { - yyerror("fatal: nested @if macros are not allowed"); - return 0; - } + P.if_depth++; - P.if_depth++; + const char* minimum = yytext+20; + ParserDebug("\tL:macro @if %d:version=%s\n", P.line_pos, minimum); - if (sscanf(Version(), "%d.%d.%d", &major, &minor, &patch) == 3) - { - request_level = sscanf(version_text, "%d.%d.%d", &request_major, &request_minor, &request_patch); - if (request_level >= 1) - { - if (request_major > major) { - ParserDebug("\tL:macro @if %d:ignoring to next @endif or EOF\n", P.line_pos); - BEGIN(if_ignore_state); - } else if (request_major < major || request_level == 1) { - ParserDebug("\tL:macro @if %d:accepted to next @endif\n", P.line_pos); - } else { // request_major==major && request_level > 1 - if (request_minor > minor) { - ParserDebug("\tL:macro @if %d:ignoring to next @endif or EOF\n", P.line_pos); - BEGIN(if_ignore_state); - } else if (request_minor < minor || request_level == 2) { - ParserDebug("\tL:macro @if %d:accepted to next @endif\n", P.line_pos); - } else { // request_minor==minor && request_level > 2 - if (request_patch > patch) { - ParserDebug("\tL:macro @if %d:ignoring to next @endif or EOF\n", P.line_pos); - BEGIN(if_ignore_state); - } else { // request_patch <= patch - ParserDebug("\tL:macro @if %d:accepted to next @endif\n", P.line_pos); - } - } - } - } - else - { - yyerror("fatal: macro @if requested an unparseable version"); - return 0; - } - } - else - { - yyerror("fatal: Version() was unparseable"); - return 0; - } + VersionComparison result = CompareVersion(Version(), minimum); + if (result == VERSION_GREATER || result == VERSION_EQUAL) + { + ParserDebug("\tL:macro @if %d:accepted to next @endif\n", P.line_pos); + } + else if (result == VERSION_SMALLER) + { + ParserDebug("\tL:macro @if %d:ignoring to next @endif or EOF\n", P.line_pos); + BEGIN(if_ignore_state); + } + else + { + assert(result == VERSION_ERROR); + yyerror("fatal: macro @if requested an unparseable version"); } } From 18bdccaf37e07a82c730c9fca8c1f5521447a700 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 5 Feb 2020 15:17:59 +0100 Subject: [PATCH 038/333] Added maximum_version macro Changelog: Title Ticket: None Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 6368a0750f5c8a05b48cf9e34b6ea46262c0dd57) --- libpromises/cf3lex.l | 35 +++++++++++++++++ tests/acceptance/00_basics/macros/if.cf | 50 +++++++++++++++++++++---- 2 files changed, 78 insertions(+), 7 deletions(-) diff --git a/libpromises/cf3lex.l b/libpromises/cf3lex.l index 569740e713..6607f7bc9f 100644 --- a/libpromises/cf3lex.l +++ b/libpromises/cf3lex.l @@ -89,6 +89,7 @@ eat_line ([\n]|[\xd][\xa]).* comment #[^\n]* macro_if_minimum_version @if\ minimum_version\([0-9]{1,10}\.[0-9]{1,10}(\.[0-9]{1,10})?\) +macro_if_maximum_version @if\ maximum_version\([0-9]{1,10}\.[0-9]{1,10}(\.[0-9]{1,10})?\) macro_if_feature @if\ feature\([a-zA-Z0-9_]+\) macro_endif @endif macro_line [^@\n\xd\xa].* @@ -181,6 +182,40 @@ promise_type [a-zA-Z_]+: } } +{macro_if_maximum_version} { + if ( P.line_pos != 1 ) + { + yyerror("fatal: macro @if must be at beginning of the line."); + return 0; + } + if (P.if_depth > 0) + { + yyerror("fatal: nested @if macros are not allowed"); + return 0; + } + + P.if_depth++; + + const char* maximum = yytext+20; + ParserDebug("\tL:macro @if %d:version=%s\n", P.line_pos, maximum); + + VersionComparison result = CompareVersion(Version(), maximum); + if (result == VERSION_SMALLER || result == VERSION_EQUAL) + { + ParserDebug("\tL:macro @if %d:accepted to next @endif\n", P.line_pos); + } + else if (result == VERSION_GREATER) + { + ParserDebug("\tL:macro @if %d:ignoring to next @endif or EOF\n", P.line_pos); + BEGIN(if_ignore_state); + } + else + { + assert(result == VERSION_ERROR); + yyerror("fatal: macro @if requested an unparseable version"); + } + } + {macro_if_feature} { if ( P.line_pos != 1 ) { diff --git a/tests/acceptance/00_basics/macros/if.cf b/tests/acceptance/00_basics/macros/if.cf index 8114e33ec2..931f3c9070 100644 --- a/tests/acceptance/00_basics/macros/if.cf +++ b/tests/acceptance/00_basics/macros/if.cf @@ -15,31 +15,63 @@ bundle common test { @if minimum_version(3.7) classes: - "expected" expression => "any"; + "expected_3_7"; @endif @if minimum_version(3.6) classes: - "expected2" expression => "any"; + "expected_3_6"; @endif @if minimum_version(2.100) classes: - "expected_2_100" expression => "any"; + "expected_2_100"; @endif @if minimum_version(300.700) classes: - "not_expected" expression => "any"; + "not_expected"; +@endif + +@if maximum_version(3.0) + classes: + "not_expected_2"; +@endif + +@if maximum_version(4.0) + classes: + "expected_4_0"; +@endif + +@if maximum_version(4.0.0) + classes: + "expected_4_0_0"; @endif } bundle agent check { + vars: + "expected_classes" slist => classesmatching("expected.*"); + "not_expected_classes" slist => classesmatching("not_expected.*"); + "expected_length" int => length("expected_classes"); + "not_expected_length" int => length("not_expected_classes"); + classes: + "pass_expected" if => strcmp("$(expected_length)", "5"); + "pass_not_expected" if => strcmp("$(not_expected_length)", "0"); + "ok" and => { "pass_expected", "pass_not_expected" }; methods: - "" usebundle => dcs_passif_expected("expected,expected2,expected_2_100", - "not_expected", - $(this.promise_filename)); + ok:: + "" usebundle => dcs_pass($(this.promise_filename)); + reports: + "Expected classes: $(expected_classes)"; + "Not expected classes: $(not_expected_classes)"; + "Expected length: $(expected_length)"; + "Not expected length: $(not_expected_length)"; + pass_expected:: + "pass_expected"; + pass_not_expected:: + "pass_not_expected"; } @if minimum_version(300.600) @@ -67,3 +99,7 @@ Nothing should be seen here really Who knows, perhaps this text doesn't exist..? @endif + +@if maximum_version(3.6.0) +body files control { {} } +@endif From d6e5c70fcbf6864b78c255810efc7a0dc77348bf Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 5 Feb 2020 18:27:04 +0100 Subject: [PATCH 039/333] Added else macro in policy parser Changelog: Title Ticket: None Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit ec8ef917b331eec4e705b401c4ca905aadf999fa) --- libpromises/cf3lex.l | 15 ++++++++ tests/acceptance/00_basics/macros/if.cf | 46 ++++++++++++++++++++----- 2 files changed, 52 insertions(+), 9 deletions(-) diff --git a/libpromises/cf3lex.l b/libpromises/cf3lex.l index 6607f7bc9f..49fa2352ec 100644 --- a/libpromises/cf3lex.l +++ b/libpromises/cf3lex.l @@ -91,6 +91,7 @@ comment #[^\n]* macro_if_minimum_version @if\ minimum_version\([0-9]{1,10}\.[0-9]{1,10}(\.[0-9]{1,10})?\) macro_if_maximum_version @if\ maximum_version\([0-9]{1,10}\.[0-9]{1,10}(\.[0-9]{1,10})?\) macro_if_feature @if\ feature\([a-zA-Z0-9_]+\) +macro_else @else macro_endif @endif macro_line [^@\n\xd\xa].* @@ -271,6 +272,20 @@ promise_type [a-zA-Z_]+: ParserDebug("\tL:inside macro @if, ignoring line text:\"%s\"\n", yytext); } +{macro_else} { + ParserDebug("\tL:macro @else, will no longer ignore lines\n", P.line_pos); + BEGIN(INITIAL); + } +{macro_else} { + if (P.if_depth <= 0) + { + yyerror("fatal: @else macro without a matching @if are not allowed"); + return 0; + } + ParserDebug("\tL:macro @else, will now ignore lines\n", P.line_pos); + BEGIN(if_ignore_state); + } + {macro_endif} { ParserDebug("\tL:macro @endif %d\n", P.line_pos); BEGIN(INITIAL); diff --git a/tests/acceptance/00_basics/macros/if.cf b/tests/acceptance/00_basics/macros/if.cf index 931f3c9070..5595dbc097 100644 --- a/tests/acceptance/00_basics/macros/if.cf +++ b/tests/acceptance/00_basics/macros/if.cf @@ -16,36 +16,57 @@ bundle common test @if minimum_version(3.7) classes: "expected_3_7"; +@else + classes: + "not_expected_3_7"; @endif @if minimum_version(3.6) classes: "expected_3_6"; +@else + classes: + "not_expected_3_6"; @endif @if minimum_version(2.100) classes: "expected_2_100"; +@else + classes: + "not_xpected_2_100"; @endif @if minimum_version(300.700) classes: - "not_expected"; + "not_expected_300_700"; +@else + classes: + "expected_300_700"; @endif @if maximum_version(3.0) classes: - "not_expected_2"; + "not_expected_3_0"; +@else + classes: + "expected_3_0"; @endif @if maximum_version(4.0) classes: "expected_4_0"; +@else + classes: + "not_expected_4_0"; @endif @if maximum_version(4.0.0) classes: "expected_4_0_0"; +@else + classes: + "not_expected_4_0_0"; @endif } @@ -57,23 +78,30 @@ bundle agent check "expected_length" int => length("expected_classes"); "not_expected_length" int => length("not_expected_classes"); classes: - "pass_expected" if => strcmp("$(expected_length)", "5"); + "pass_expected" if => strcmp("$(expected_length)", "7"); "pass_not_expected" if => strcmp("$(not_expected_length)", "0"); "ok" and => { "pass_expected", "pass_not_expected" }; methods: ok:: "" usebundle => dcs_pass($(this.promise_filename)); reports: - "Expected classes: $(expected_classes)"; - "Not expected classes: $(not_expected_classes)"; - "Expected length: $(expected_length)"; - "Not expected length: $(not_expected_length)"; - pass_expected:: + DEBUG:: + "Expected classes: $(expected_classes)"; + "Not expected classes: $(not_expected_classes)"; + "Expected length: $(expected_length)"; + "Not expected length: $(not_expected_length)"; + DEBUG.pass_expected:: "pass_expected"; - pass_not_expected:: + DEBUG.pass_not_expected:: "pass_not_expected"; } +@if minimum_version(3.12) +@else +some invalid syntax here +body {} {}{}{{}} +@endif + @if minimum_version(300.600) This text should never be seen, it's completely ignored From 50f59c2f156a64d142b702fc0694c29258ca8893 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 5 Feb 2020 23:05:23 +0100 Subject: [PATCH 040/333] Added between_versions macro to parser Changelog: Title Ticket: CFE-3198 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 25a24e481a369f4b44232266c715c65ad67e07a1) --- libpromises/cf3lex.l | 46 +++++++++++++++++++++++-- tests/acceptance/00_basics/macros/if.cf | 23 ++++++++++++- 2 files changed, 65 insertions(+), 4 deletions(-) diff --git a/libpromises/cf3lex.l b/libpromises/cf3lex.l index 49fa2352ec..6c691d6f50 100644 --- a/libpromises/cf3lex.l +++ b/libpromises/cf3lex.l @@ -88,9 +88,10 @@ eat_line ([\n]|[\xd][\xa]).* comment #[^\n]* -macro_if_minimum_version @if\ minimum_version\([0-9]{1,10}\.[0-9]{1,10}(\.[0-9]{1,10})?\) -macro_if_maximum_version @if\ maximum_version\([0-9]{1,10}\.[0-9]{1,10}(\.[0-9]{1,10})?\) -macro_if_feature @if\ feature\([a-zA-Z0-9_]+\) +macro_if_minimum_version @if\ minimum_version\([0-9]{1,10}\.[0-9]{1,10}(\.[0-9]{1,10})?\) +macro_if_maximum_version @if\ maximum_version\([0-9]{1,10}\.[0-9]{1,10}(\.[0-9]{1,10})?\) +macro_if_between_versions @if\ between_versions\([0-9]{1,10}\.[0-9]{1,10}(\.[0-9]{1,10})?\ *,\ *[0-9]{1,10}\.[0-9]{1,10}(\.[0-9]{1,10})?\) +macro_if_feature @if\ feature\([a-zA-Z0-9_]+\) macro_else @else macro_endif @endif macro_line [^@\n\xd\xa].* @@ -217,6 +218,45 @@ promise_type [a-zA-Z_]+: } } +{macro_if_between_versions} { + if ( P.line_pos != 1 ) + { + yyerror("fatal: macro @if must be at beginning of the line."); + return 0; + } + if (P.if_depth > 0) + { + yyerror("fatal: nested @if macros are not allowed"); + return 0; + } + + P.if_depth++; + + const char* from = yytext + strlen("@if between_versions("); + + const char *to = strchr(from, ','); + to += 1; + while (*to == ' ') + { + to += 1; + } + + ParserDebug("\tL:macro @if %d:between_versions(%s\n", P.line_pos, from); + + VersionComparison a = CompareVersion(Version(), from); + VersionComparison b = CompareVersion(Version(), to); + if ((a == VERSION_EQUAL || a == VERSION_GREATER) + && (b == VERSION_EQUAL || b == VERSION_SMALLER)) + { + ParserDebug("\tL:macro @if %d:accepted to next @endif\n", P.line_pos); + } + else + { + ParserDebug("\tL:macro @if %d:ignoring to next @endif or EOF\n", P.line_pos); + BEGIN(if_ignore_state); + } + } + {macro_if_feature} { if ( P.line_pos != 1 ) { diff --git a/tests/acceptance/00_basics/macros/if.cf b/tests/acceptance/00_basics/macros/if.cf index 5595dbc097..70e09a4c01 100644 --- a/tests/acceptance/00_basics/macros/if.cf +++ b/tests/acceptance/00_basics/macros/if.cf @@ -68,6 +68,22 @@ bundle common test classes: "not_expected_4_0_0"; @endif + +@if between_versions(3.15.0, 4.0.0) + classes: + "expected_3_15_0_4_0_0"; +@else + classes: + "not_expected_3_15_0_4_0_0"; +@endif + +@if between_versions(3.11, 3.12) + classes: + "not_expected_3_11_3_12"; +@else + classes: + "expected_3_11_3_12"; +@endif } bundle agent check @@ -78,7 +94,7 @@ bundle agent check "expected_length" int => length("expected_classes"); "not_expected_length" int => length("not_expected_classes"); classes: - "pass_expected" if => strcmp("$(expected_length)", "7"); + "pass_expected" if => strcmp("$(expected_length)", "9"); "pass_not_expected" if => strcmp("$(not_expected_length)", "0"); "ok" and => { "pass_expected", "pass_not_expected" }; methods: @@ -131,3 +147,8 @@ Who knows, perhaps this text doesn't exist..? @if maximum_version(3.6.0) body files control { {} } @endif + +@if between_versions(2.0, 3.0) +more invalid syntax +body body body body {{}};;:: +@endif From b5ce981d6979040b47f77e607a37b655a68960a5 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 5 Feb 2020 23:26:20 +0100 Subject: [PATCH 041/333] Version macros now accept single digits Primarily so you can skip the .0 in 3.0, for example: ``` @if between_versions(3, 3.9.9) ``` Changelog: Title Ticket: CFE-3198 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 1b8271e2780d486c61a2c7f5d57e644398f90f02) --- libpromises/cf3lex.l | 6 +++--- tests/acceptance/00_basics/macros/if.cf | 27 +++++++++++++++++++++---- 2 files changed, 26 insertions(+), 7 deletions(-) diff --git a/libpromises/cf3lex.l b/libpromises/cf3lex.l index 6c691d6f50..2e7a96cd43 100644 --- a/libpromises/cf3lex.l +++ b/libpromises/cf3lex.l @@ -88,9 +88,9 @@ eat_line ([\n]|[\xd][\xa]).* comment #[^\n]* -macro_if_minimum_version @if\ minimum_version\([0-9]{1,10}\.[0-9]{1,10}(\.[0-9]{1,10})?\) -macro_if_maximum_version @if\ maximum_version\([0-9]{1,10}\.[0-9]{1,10}(\.[0-9]{1,10})?\) -macro_if_between_versions @if\ between_versions\([0-9]{1,10}\.[0-9]{1,10}(\.[0-9]{1,10})?\ *,\ *[0-9]{1,10}\.[0-9]{1,10}(\.[0-9]{1,10})?\) +macro_if_minimum_version @if\ minimum_version\([0-9]{1,5}(\.[0-9]{1,5}){0,2}\) +macro_if_maximum_version @if\ maximum_version\([0-9]{1,5}(\.[0-9]{1,5}){0,2}\) +macro_if_between_versions @if\ between_versions\([0-9]{1,5}(\.[0-9]{1,5}){0,2}\ *,\ *[0-9]{1,5}(\.[0-9]{1,5}){0,2}\) macro_if_feature @if\ feature\([a-zA-Z0-9_]+\) macro_else @else macro_endif @endif diff --git a/tests/acceptance/00_basics/macros/if.cf b/tests/acceptance/00_basics/macros/if.cf index 70e09a4c01..e4fe7f1991 100644 --- a/tests/acceptance/00_basics/macros/if.cf +++ b/tests/acceptance/00_basics/macros/if.cf @@ -13,6 +13,14 @@ body common control bundle common test { +@if minimum_version(3) + classes: + "expected_3"; +@else + classes: + "not_expected_3"; +@endif + @if minimum_version(3.7) classes: "expected_3_7"; @@ -53,12 +61,12 @@ bundle common test "expected_3_0"; @endif -@if maximum_version(4.0) +@if maximum_version(4) classes: - "expected_4_0"; + "expected_4"; @else classes: - "not_expected_4_0"; + "not_expected_4"; @endif @if maximum_version(4.0.0) @@ -94,7 +102,7 @@ bundle agent check "expected_length" int => length("expected_classes"); "not_expected_length" int => length("not_expected_classes"); classes: - "pass_expected" if => strcmp("$(expected_length)", "9"); + "pass_expected" if => strcmp("$(expected_length)", "10"); "pass_not_expected" if => strcmp("$(not_expected_length)", "0"); "ok" and => { "pass_expected", "pass_not_expected" }; methods: @@ -118,6 +126,12 @@ some invalid syntax here body {} {}{}{{}} @endif +@if minimum_version(3) +@else +some invalid syntax here +body {} {}{}{{}} +@endif + @if minimum_version(300.600) This text should never be seen, it's completely ignored @@ -152,3 +166,8 @@ body files control { {} } more invalid syntax body body body body {{}};;:: @endif + +@if between_versions(1, 3.6) +more invalid syntax +body body body body {{}};;:: +@endif From 2bab83dca93abd81899cc9abaa3a991ab77f4423 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Mon, 10 Feb 2020 11:12:38 +0100 Subject: [PATCH 042/333] Added macros before_version, at_version and after_version Changelog: Title Ticket: CFE-3198 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 8a21d8f55e188977d94f5fbe3a42da59c3ddf508) --- libpromises/cf3lex.l | 105 ++++++++++++++++++++++++ tests/acceptance/00_basics/macros/if.cf | 65 ++++++++++++++- 2 files changed, 169 insertions(+), 1 deletion(-) diff --git a/libpromises/cf3lex.l b/libpromises/cf3lex.l index 2e7a96cd43..986a12731c 100644 --- a/libpromises/cf3lex.l +++ b/libpromises/cf3lex.l @@ -90,6 +90,9 @@ comment #[^\n]* macro_if_minimum_version @if\ minimum_version\([0-9]{1,5}(\.[0-9]{1,5}){0,2}\) macro_if_maximum_version @if\ maximum_version\([0-9]{1,5}(\.[0-9]{1,5}){0,2}\) +macro_if_before_version @if\ before_version\([0-9]{1,5}(\.[0-9]{1,5}){0,2}\) +macro_if_at_version @if\ at_version\([0-9]{1,5}(\.[0-9]{1,5}){0,2}\) +macro_if_after_version @if\ after_version\([0-9]{1,5}(\.[0-9]{1,5}){0,2}\) macro_if_between_versions @if\ between_versions\([0-9]{1,5}(\.[0-9]{1,5}){0,2}\ *,\ *[0-9]{1,5}(\.[0-9]{1,5}){0,2}\) macro_if_feature @if\ feature\([a-zA-Z0-9_]+\) macro_else @else @@ -218,6 +221,108 @@ promise_type [a-zA-Z_]+: } } +{macro_if_before_version} { + if ( P.line_pos != 1 ) + { + yyerror("fatal: macro @if must be at beginning of the line."); + return 0; + } + if (P.if_depth > 0) + { + yyerror("fatal: nested @if macros are not allowed"); + return 0; + } + + P.if_depth++; + + const char* target = yytext + strlen("@if before_version("); + ParserDebug("\tL:macro @if %d:version=%s\n", P.line_pos, target); + + VersionComparison result = CompareVersion(Version(), target); + if (result == VERSION_SMALLER) + { + ParserDebug("\tL:macro @if %d:accepted to next @endif\n", P.line_pos); + } + else if (result == VERSION_GREATER || result == VERSION_EQUAL) + { + ParserDebug("\tL:macro @if %d:ignoring to next @endif or EOF\n", P.line_pos); + BEGIN(if_ignore_state); + } + else + { + assert(result == VERSION_ERROR); + yyerror("fatal: macro @if requested an unparseable version"); + } + } + +{macro_if_at_version} { + if ( P.line_pos != 1 ) + { + yyerror("fatal: macro @if must be at beginning of the line."); + return 0; + } + if (P.if_depth > 0) + { + yyerror("fatal: nested @if macros are not allowed"); + return 0; + } + + P.if_depth++; + + const char* target = yytext + strlen("@if at_version("); + ParserDebug("\tL:macro @if %d:version=%s\n", P.line_pos, target); + + VersionComparison result = CompareVersion(Version(), target); + if (result == VERSION_EQUAL) + { + ParserDebug("\tL:macro @if %d:accepted to next @endif\n", P.line_pos); + } + else if (result == VERSION_GREATER || result == VERSION_SMALLER) + { + ParserDebug("\tL:macro @if %d:ignoring to next @endif or EOF\n", P.line_pos); + BEGIN(if_ignore_state); + } + else + { + assert(result == VERSION_ERROR); + yyerror("fatal: macro @if requested an unparseable version"); + } + } + +{macro_if_after_version} { + if ( P.line_pos != 1 ) + { + yyerror("fatal: macro @if must be at beginning of the line."); + return 0; + } + if (P.if_depth > 0) + { + yyerror("fatal: nested @if macros are not allowed"); + return 0; + } + + P.if_depth++; + + const char* target = yytext + strlen("@if after_version("); + ParserDebug("\tL:macro @if %d:version=%s\n", P.line_pos, target); + + VersionComparison result = CompareVersion(Version(), target); + if (result == VERSION_GREATER) + { + ParserDebug("\tL:macro @if %d:accepted to next @endif\n", P.line_pos); + } + else if (result == VERSION_EQUAL || result == VERSION_SMALLER) + { + ParserDebug("\tL:macro @if %d:ignoring to next @endif or EOF\n", P.line_pos); + BEGIN(if_ignore_state); + } + else + { + assert(result == VERSION_ERROR); + yyerror("fatal: macro @if requested an unparseable version"); + } + } + {macro_if_between_versions} { if ( P.line_pos != 1 ) { diff --git a/tests/acceptance/00_basics/macros/if.cf b/tests/acceptance/00_basics/macros/if.cf index e4fe7f1991..47aa9cb3df 100644 --- a/tests/acceptance/00_basics/macros/if.cf +++ b/tests/acceptance/00_basics/macros/if.cf @@ -92,6 +92,54 @@ bundle common test classes: "expected_3_11_3_12"; @endif + +@if before_version(4) + classes: + "expected_before_version_4"; +@else + classes: + "not_expected_before_version_4"; +@endif + +@if before_version(3) + classes: + "not_expected_before_version_3"; +@else + classes: + "expected_before_version_3"; +@endif + +@if at_version(3) + classes: + "expected_at_version_3"; +@else + classes: + "not_expected_at_version_3"; +@endif + +@if at_version(2) + classes: + "not_expected_at_version_2"; +@else + classes: + "expected_at_version_2"; +@endif + +@if after_version(2) + classes: + "expected_after_version_2"; +@else + classes: + "not_expected_after_version_2"; +@endif + +@if after_version(3) + classes: + "not_expected_after_version_3"; +@else + classes: + "expected_after_version_3"; +@endif } bundle agent check @@ -102,7 +150,7 @@ bundle agent check "expected_length" int => length("expected_classes"); "not_expected_length" int => length("not_expected_classes"); classes: - "pass_expected" if => strcmp("$(expected_length)", "10"); + "pass_expected" if => strcmp("$(expected_length)", "16"); "pass_not_expected" if => strcmp("$(not_expected_length)", "0"); "ok" and => { "pass_expected", "pass_not_expected" }; methods: @@ -171,3 +219,18 @@ body body body body {{}};;:: more invalid syntax body body body body {{}};;:: @endif + +@if at_version(2) +body invalid syntax +{{{reports:}}};;;::: +@endif + +@if before_version(3) +body invalid syntax +{{{reports:}}};;;::: +@endif + +@if after_version(3) +body invalid syntax +{{{reports:}}};;;::: +@endif From eff2e37651c16271c3c1ebf3eecd7c401cad6f03 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Mon, 10 Feb 2020 07:24:13 +0100 Subject: [PATCH 043/333] Made classfiltercsv() fail properly on invalid class expression index Reported by LGTM: https://lgtm.com/rules/2165170567/ Changelog: Title Ticket: None Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit e6ca6b33875ade3a1bf93935614bb47f587abd0a) --- libpromises/evalfunction.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/libpromises/evalfunction.c b/libpromises/evalfunction.c index aeae150f00..dd4a043979 100644 --- a/libpromises/evalfunction.c +++ b/libpromises/evalfunction.c @@ -6872,7 +6872,9 @@ static FnCallResult FnCallClassFilterCsv(EvalContext *ctx, "%s: Class expression index is out of bounds. " "Row length %zu, index %zu", fp->name, num_columns, class_index); - FnFailure(); + SeqDestroy(list); + JsonDestroy(json); + return FnFailure(); } } else if (num_columns != SeqLength(list)) From 2b176abb553d3e53c20d3a95a72db7389eb7d474 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Tue, 11 Feb 2020 10:20:25 +0100 Subject: [PATCH 044/333] cf-check: Symlinked LMDB databases are now preserved in repair Performs diagnosis and repair on symlink target instead of symlink. Repaired files / copies are placed alongside symlink target. In some cases, the symlink target is deleted to repair a corrupt database, and the symlink is left as a broken symlink. This is handled gracefully by the agent, it will be recreated. Broken symlinks are now detected as an acceptable condition in diagnose, it won't try to repair them or delete them. Changelog: Commit Ticket: ENT-5162 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 588c775a74aa168689f8e5f44ca7c89de647d1f8) --- cf-check/diagnose.c | 68 ++++++++++++++++++++++++++++++++++++++++----- cf-check/diagnose.h | 1 + 2 files changed, 62 insertions(+), 7 deletions(-) diff --git a/cf-check/diagnose.c b/cf-check/diagnose.c index cbf80f257f..0c6020511b 100644 --- a/cf-check/diagnose.c +++ b/cf-check/diagnose.c @@ -507,6 +507,23 @@ static int fork_and_diagnose(const char *path, bool validate, bool test_write) return CF_CHECK_OK; } +static char *follow_symlink(const char *path) +{ + char target_buf[4096] = { 0 }; + const ssize_t r = readlink(path, target_buf, sizeof(target_buf)); + if (r < 0) + { + return NULL; + } + if (r >= sizeof(target_buf)) + { + Log(LOG_LEVEL_ERR, "Symlink target path too long: %s", path); + return NULL; + } + target_buf[r] = '\0'; + return xstrdup(target_buf); +} + /** * @param[in] filenames DB files to diagnose/check * @param[out] corrupt place to store the resulting sequence of corrupted @@ -534,8 +551,31 @@ size_t diagnose_files( for (int i = 0; i < length; ++i) { const char *filename = SeqAt(filenames, i); - int r; - if (foreground) + const char *symlink = NULL; // Only initialized because of gcc warning + int r = 0; // Only initialized because of LGTM alert + char *symlink_target = follow_symlink(filename); + bool broken_symlink_handled = false; + if (symlink_target != NULL) + { + symlink = filename; + // If the LMDB file path is a symlink + filename = symlink_target; + if (access(symlink_target, F_OK) != 0) + { + // Symlink target file does not exist + r = CF_CHECK_OK_DOES_NOT_EXIST; + broken_symlink_handled = true; + } + // If this is not the case, continue repair as normal, + // using the symlink target instead of the symlink in diagnose + // and repair functions + } + if (broken_symlink_handled) + { + // The LMDB database was a broken symlink, + // we don't need to do anything, agent will recreate it. + } + else if (foreground) { r = diagnose(filename, true, validate); if ((r == CF_CHECK_OK) && test_write) @@ -547,12 +587,25 @@ size_t diagnose_files( { r = fork_and_diagnose(filename, validate, test_write); } - Log(LOG_LEVEL_INFO, - "Status of '%s': %s\n", - filename, - CF_CHECK_STRING(r)); - if (r != CF_CHECK_OK) + if (symlink_target != NULL) + { + Log(LOG_LEVEL_INFO, + "Status of '%s' -> '%s': %s\n", + symlink, + symlink_target, + CF_CHECK_STRING(r)); + } + else + { + Log(LOG_LEVEL_INFO, + "Status of '%s': %s\n", + filename, + CF_CHECK_STRING(r)); + } + + + if (r != CF_CHECK_OK && r != CF_CHECK_OK_DOES_NOT_EXIST) { ++corruptions; if (corrupt != NULL) @@ -560,6 +613,7 @@ size_t diagnose_files( SeqAppend(*corrupt, xstrdup(filename)); } } + free(symlink_target); } if (corruptions == 0) { diff --git a/cf-check/diagnose.h b/cf-check/diagnose.h index 923e2431ce..4bb01c7dde 100644 --- a/cf-check/diagnose.h +++ b/cf-check/diagnose.h @@ -13,6 +13,7 @@ // clang-format off #define CF_CHECK_RUN_CODES(macro) \ macro(OK) \ + macro(OK_DOES_NOT_EXIST) \ macro(SIGNAL_HANGUP) \ macro(SIGNAL_INTERRUPT) \ macro(SIGNAL_QUIT) \ From 447aa8eff883eb4a752430135fffec2d4c79f418 Mon Sep 17 00:00:00 2001 From: Ole Petter Date: Fri, 17 Jan 2020 18:14:51 +0100 Subject: [PATCH 045/333] CFE-3046: Only add unique IPv4 and MAC addresses to sys interfaces Previously, when an interface had multiple IPv4 addresses set, the parsing of these addresses would add the interface to the sys.interfaces and sys.hardware variables multiple times, resulting in the interface showing up as a duplicate in the list. This fix makes sure that an interface is only added once for every interface type, regardless of how many IPv4 addresses the interface contains. Changelog: Fixed an issue causing duplicate entries in sys.interfaces, and sys.hardware. Ticket: CFE-3046 Signed-off-by: Ole Petter (cherry picked from commit 3f1bc9d2c97d760bbb31979516fe8afbd9a7b4cc) --- libenv/unix_iface.c | 38 +++++++++++++++++++++++++++++++------- 1 file changed, 31 insertions(+), 7 deletions(-) diff --git a/libenv/unix_iface.c b/libenv/unix_iface.c index ad934f30c0..c62e1cb730 100644 --- a/libenv/unix_iface.c +++ b/libenv/unix_iface.c @@ -165,8 +165,14 @@ static void GetMacAddress(EvalContext *ctx, ARG_UNUSED int fd, struct ifreq *ifr (unsigned char) ifr->ifr_hwaddr.sa_data[5]); EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_SYS, name, hw_mac, CF_DATA_TYPE_STRING, "source=agent"); - RlistAppend(hardware, hw_mac, RVAL_TYPE_SCALAR); - RlistAppend(interfaces, ifp->ifr_name, RVAL_TYPE_SCALAR); + if (!RlistContainsString(*hardware, hw_mac)) + { + RlistAppend(hardware, hw_mac, RVAL_TYPE_SCALAR); + } + if (!RlistContainsString(*interfaces, ifp->ifr_name)) + { + RlistAppend(interfaces, ifp->ifr_name, RVAL_TYPE_SCALAR); + } snprintf(name, sizeof(name), "mac_%s", CanonifyName(hw_mac)); EvalContextClassPutHard(ctx, name, "inventory,attribute_name=none,source=agent"); @@ -204,7 +210,10 @@ static void GetMacAddress(EvalContext *ctx, ARG_UNUSED int fd, struct ifreq *ifr (unsigned char) m[5]); EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_SYS, name, hw_mac, CF_DATA_TYPE_STRING, "source=agent"); - RlistAppend(hardware, hw_mac, RVAL_TYPE_SCALAR); + if (!RlistContainsString(*hardware, hw_mac)) + { + RlistAppend(hardware, hw_mac, RVAL_TYPE_SCALAR); + } RlistAppend(interfaces, ifa->ifa_name, RVAL_TYPE_SCALAR); snprintf(name, sizeof(name), "mac_%s", CanonifyName(hw_mac)); @@ -225,8 +234,15 @@ static void GetMacAddress(EvalContext *ctx, ARG_UNUSED int fd, struct ifreq *ifr mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]); EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_SYS, name, hw_mac, CF_DATA_TYPE_STRING, "source=agent"); - RlistAppend(hardware, hw_mac, RVAL_TYPE_SCALAR); - RlistAppend(interfaces, ifp->ifr_name, RVAL_TYPE_SCALAR); + + if (!RlistContainsString(*hardware, hw_mac)) + { + RlistAppend(hardware, hw_mac, RVAL_TYPE_SCALAR); + } + if (!RlistContainsString(*interfaces, ifp->ifr_name)) + { + RlistAppend(interfaces, ifp->ifr_name, RVAL_TYPE_SCALAR); + } snprintf(name, CF_MAXVARSIZE, "mac_%s", CanonifyName(hw_mac)); EvalContextClassPutHard(ctx, name, "inventory,attribute_name=none,source=agent"); @@ -273,8 +289,16 @@ static void GetMacAddress(EvalContext *ctx, ARG_UNUSED int fd, struct ifreq *ifr EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_SYS, name, hw_mac, CF_DATA_TYPE_STRING, "source=agent"); - RlistAppend(hardware, hw_mac, RVAL_TYPE_SCALAR); - RlistAppend(interfaces, ifp->ifr_name, RVAL_TYPE_SCALAR); + + if (!RlistContainsString(*hardware, hw_mac)) + { + RlistAppend(hardware, hw_mac, RVAL_TYPE_SCALAR); + } + + if (!RlistContainsString(*interfaces, ifp->ifr_name)) + { + RlistAppend(interfaces, ifp->ifr_name, RVAL_TYPE_SCALAR); + } snprintf(name, sizeof(name), "mac_%s", CanonifyName(hw_mac)); EvalContextClassPutHard(ctx, name, From 13452f59652286011e923001f90542b127dec637 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Mon, 17 Feb 2020 13:39:53 +0100 Subject: [PATCH 046/333] Do not log an error in case of no match for a measurements promise If a measurements promise regex doesn't match any line of a given file, it should not produce an error logged to syslog. It's just noise and the default value 0.0 of the measurement indicates that nothing was measured. Running with '--verbose' then shows the log message and helps with debugging. Ticket: ENT-5171 Changelog: Measurements promises with no match no longer produce errors (cherry picked from commit 29c298e649da5f87107db3d7e61533064a961a89) --- cf-monitord/history.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cf-monitord/history.c b/cf-monitord/history.c index e6b4b8b100..334daf5af3 100644 --- a/cf-monitord/history.c +++ b/cf-monitord/history.c @@ -551,7 +551,7 @@ static PromiseResult NovaExtractValueFromStream(EvalContext *ctx, const char *ha if (!found) { - cfPS(ctx, LOG_LEVEL_ERR, PROMISE_RESULT_FAIL, pp, a, "Could not locate the line for promise '%s'", handle); + cfPS(ctx, LOG_LEVEL_VERBOSE, PROMISE_RESULT_FAIL, pp, a, "Could not locate the line for promise '%s'", handle); *value_out = 0.0; return PROMISE_RESULT_FAIL; } From d540d8ddedcd96dc658791d02a0846192abcb590 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 19 Feb 2020 13:30:07 +0100 Subject: [PATCH 047/333] Added warning if CSV parser parses nothing from non-empty file Changelog: Title Ticket: CFE-3256 Signed-off-by: Ole Herman Schumacher Elgesem --- libntech | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libntech b/libntech index 6babc76088..1f07327d67 160000 --- a/libntech +++ b/libntech @@ -1 +1 @@ -Subproject commit 6babc7608811c43f99354696062c4c831aa9bfba +Subproject commit 1f07327d672f740700df738b3a4004d8c544e2ec From fa179e118f3026854ca4240f64c3decd3c2bcf78 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Wed, 19 Feb 2020 09:59:00 -0600 Subject: [PATCH 048/333] Added cf-postgres requirement to cf-apache and cf-hub systemd units During upgrades the services are started in random order and so need a Requires= to force them to start in the proper order. Changelog: title Ticket: ENT-5125 (cherry picked from commit ce2862bfc3f8c864c5335d25dd9a6ff15cfe120a) --- misc/systemd/cf-apache.service.in | 1 + misc/systemd/cf-hub.service.in | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/misc/systemd/cf-apache.service.in b/misc/systemd/cf-apache.service.in index 4e0e7b53d0..ac41ac49f8 100644 --- a/misc/systemd/cf-apache.service.in +++ b/misc/systemd/cf-apache.service.in @@ -2,6 +2,7 @@ Description=CFEngine Enterprise Webserver After=syslog.target After=cf-postgres.service +Requires=cf-postgres.service ConditionPathExists=@workdir@/httpd/bin/apachectl PartOf=cfengine3.service diff --git a/misc/systemd/cf-hub.service.in b/misc/systemd/cf-hub.service.in index 1c8c62aa46..1d6a879035 100644 --- a/misc/systemd/cf-hub.service.in +++ b/misc/systemd/cf-hub.service.in @@ -6,8 +6,8 @@ ConditionPathExists=@workdir@/inputs/promises.cf After=syslog.target After=network.target -Wants=cf-postgres.service After=cf-postgres.service +Requires=cf-postgres.service [Service] Type=simple From 224dbb67858c20def0bfa85a61e0659335b1badd Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Fri, 28 Feb 2020 19:59:10 +0100 Subject: [PATCH 049/333] Added .error.cf possibility to acceptance tests Purpose is explained in testall script; * `.x.cf` is for tests which are supposed exit with error, for example, invalid policy syntax. * `.error.cf` is for tests which are supposed to log errors, but exit with 0. For example failed promises. Changelog: None Ticket: CFE-3041 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit a1c644895591885fd80ead133493e712cc67535d) --- .../templating/mustache_no_end.error.cf | 8 +++ tests/acceptance/testall | 56 +++++++++++++------ 2 files changed, 47 insertions(+), 17 deletions(-) create mode 100644 tests/acceptance/10_files/templating/mustache_no_end.error.cf diff --git a/tests/acceptance/10_files/templating/mustache_no_end.error.cf b/tests/acceptance/10_files/templating/mustache_no_end.error.cf new file mode 100644 index 0000000000..96f49c6234 --- /dev/null +++ b/tests/acceptance/10_files/templating/mustache_no_end.error.cf @@ -0,0 +1,8 @@ +# This previously caused a crash (segfault), now we are testing that +# it prints error and exits with 0. +bundle agent main +{ + reports: + "With: $(with)" + with => string_mustache("{{"); +} diff --git a/tests/acceptance/testall b/tests/acceptance/testall index f40579a148..4e795c42e2 100755 --- a/tests/acceptance/testall +++ b/tests/acceptance/testall @@ -81,6 +81,7 @@ COMMON_TESTS=${COMMON_TESTS:-1} ; export COMMON_TESTS TIMED_TESTS=${TIMED_TESTS:-1} ; export TIMED_TESTS SLOW_TESTS=${SLOW_TESTS:-1} ; export SLOW_TESTS ERROREXIT_TESTS=${ERROREXIT_TESTS:-1} ; export ERROREXIT_TESTS +ERRORLOG_TESTS=${ERRORLOG_TESTS:-1} ; export ERRORLOG_TESTS SERIAL_TESTS=${SERIAL_TESTS:-1} ; export SERIAL_TESTS NETWORK_TESTS=${NETWORK_TESTS:-1} ; export NETWORK_TESTS LIBXML2_TESTS=${LIBXML2_TESTS:-1} ; export LIBXML2_TESTS @@ -250,7 +251,7 @@ usage() { echo " defaults to $DEFLIBTOOL." echo " --include Include the file or directory given in the workdir before the test" echo " starts. This option may be given several times." - echo " --tests=common,network,serial,timed,slow,errorexit,libxml2,libcurl" + echo " --tests=common,network,serial,timed,slow,errorexit,errorlog,libxml2,libcurl" echo " This is the default value, you can also add from the following:" echo " unsafe,staging" echo " NOTE: 'staging' tests are not expected to pass" @@ -309,7 +310,9 @@ runtest() { case "$TEST" in *.x.cf) TEST_TYPE=errorexit && - EXTRATEXT='(expected to exit with error)' ;; + EXTRATEXT='(should exit with non-zero, but not crash)' ;; + *.error.cf) TEST_TYPE=errorlog && + EXTRATEXT='(should log error(s), but exit with 0)' ;; */staging/*) TEST_TYPE=staging ;; */unsafe/*) TEST_TYPE=unsafe ;; */network/*) TEST_TYPE=network ;; @@ -327,6 +330,10 @@ runtest() { then SKIP=1 SKIPREASON="${COLOR_WARNING}'errorexit' tests are disabled${COLOR_NORMAL}" + elif [ $ERRORLOG_TESTS = 0 -a $TEST_TYPE = errorlog ] + then + SKIP=1 + SKIPREASON="${COLOR_WARNING}'errorlog' tests are disabled${COLOR_NORMAL}" elif [ $STAGING_TESTS = 0 -a $TEST_TYPE = staging ] then SKIP=1 @@ -509,9 +516,35 @@ export CFENGINE_TEST_OVERRIDE_WORKDIR TEMP CFENGINE_TEST_OVERRIDE_EXTENSION_LIBR TEST_COVERS="$(egrep "R: test covers" $OUTFILE | sed -e "s,.*test covers: \([A-Za-z0-9_]*\),\1,")" fi - RESULT_MSG= - if [ $TEST_TYPE != errorexit ] + if [ $TEST_TYPE = errorexit ] + then + if [ $RETVAL -gt 0 ] && [ $RETVAL -lt 128 ] + then + RESULT=Pass + RESULT_MSG="${COLOR_SUCCESS}Pass${COLOR_NORMAL}" + elif [ $RETVAL -ge 128 ] + then + RESULT=FAIL + RESULT_MSG="${COLOR_FAILURE}FAIL (CRASH!!! THIS IS A BUG!)${COLOR_NORMAL}" + else + RESULT=FAIL + RESULT_MSG="${COLOR_FAILURE}FAIL (Failed to exit with error!)${COLOR_NORMAL}" + fi + elif [ $TEST_TYPE = errorlog ] then + if [ $RETVAL != 0 ] + then + RESULT=FAIL + RESULT_MSG="${COLOR_FAILURE}FAIL (NON-ZERO EXIT CODE!)${COLOR_NORMAL}" + elif fgrep 'error: ' "$OUTFILE" > /dev/null + then + RESULT=Pass + RESULT_MSG="${COLOR_SUCCESS}Pass${COLOR_NORMAL}" + else + RESULT=FAIL + RESULT_MSG="${COLOR_FAILURE}FAIL (No errors printed)${COLOR_NORMAL}" + fi + else # TEST_TYPE is not errorlog or errorexit # We need to be careful when matching test outcomes. Because of convergence # passes, a test may output FAIL before it outputs Pass; in this case the # latter trumps the former. Also be careful when matching an [XS]FAIL; the @@ -578,19 +611,6 @@ export CFENGINE_TEST_OVERRIDE_WORKDIR TEMP CFENGINE_TEST_OVERRIDE_EXTENSION_LIBR RESULT=FAIL RESULT_MSG="${COLOR_FAILURE}FAIL${COLOR_NORMAL}" fi - else # TEST_TYPE = errorexit - if [ $RETVAL -gt 0 ] && [ $RETVAL -lt 128 ] - then - RESULT=Pass - RESULT_MSG="${COLOR_SUCCESS}Pass${COLOR_NORMAL}" - elif [ $RETVAL -ge 128 ] - then - RESULT=FAIL - RESULT_MSG="${COLOR_FAILURE}FAIL (CRASH!!! THIS IS A BUG!)${COLOR_NORMAL}" - else - RESULT=FAIL - RESULT_MSG="${COLOR_FAILURE}FAIL (Failed to exit with error!)${COLOR_NORMAL}" - fi fi if [ "x$RESULT_MSG" = "x" ] @@ -758,6 +778,7 @@ do TIMED_TESTS=0 SLOW_TESTS=0 ERROREXIT_TESTS=0 + ERRORLOG_TESTS=0 SERIAL_TESTS=0 NETWORK_TESTS=0 LIBXML2_TESTS=0 @@ -773,6 +794,7 @@ do timed) TIMED_TESTS=1;; slow) SLOW_TESTS=1;; errorexit) ERROREXIT_TESTS=1;; + errorlog) ERRORLOG_TESTS=1;; serial) SERIAL_TESTS=1;; network) NETWORK_TESTS=1;; libxml2) LIBXML2_TESTS=1;; From 643ff43233749ffdb82c739b07d71ad2dd10aa7c Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 4 Mar 2020 14:08:36 +0100 Subject: [PATCH 050/333] Travis: Run errorlog acceptance tests It is debatable whether errorexit tests should be considered safe or not, but errorlog should definitely be considered safe. Changelog: None Ticket: CFE-3041 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit b296f7b0b117b3ffa722d09be93edaeedfcc924d) --- travis-scripts/script.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/travis-scripts/script.sh b/travis-scripts/script.sh index fe3d9e92dd..7b452f4935 100755 --- a/travis-scripts/script.sh +++ b/travis-scripts/script.sh @@ -66,7 +66,7 @@ chmod -R go-w . if [ "$JOB_TYPE" = acceptance_tests_common ] then - ./testall --printlog --tests=common + ./testall --printlog --tests=common,errorlog exit fi From b1da77db00876a5c9022f77319107a8bed18bed4 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 4 Mar 2020 17:57:14 +0100 Subject: [PATCH 051/333] Fixed crashes and memory leaks in JSON and mustache code Done by updating libntech to master Changelog: Title Ticket: None Signed-off-by: Ole Herman Schumacher Elgesem --- libntech | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libntech b/libntech index 1f07327d67..c1ba310c89 160000 --- a/libntech +++ b/libntech @@ -1 +1 @@ -Subproject commit 1f07327d672f740700df738b3a4004d8c544e2ec +Subproject commit c1ba310c89b823450598648a07f717b784feeca0 From d760e0df7444dc10a57ec5f36d26562eeea0809b Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Thu, 12 Mar 2020 08:44:09 -0500 Subject: [PATCH 052/333] Test standard_services service_method is used implicitly from non-default namespace Ticket: ENT-5406 Changelog: None (cherry picked from commit e5a167616a425c61b734dd3b07ab1f8e23c5ce23) --- ...ard_services-from-non-default-namespace.cf | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 tests/acceptance/09_services/standard_services-from-non-default-namespace.cf diff --git a/tests/acceptance/09_services/standard_services-from-non-default-namespace.cf b/tests/acceptance/09_services/standard_services-from-non-default-namespace.cf new file mode 100644 index 0000000000..8c2057b96f --- /dev/null +++ b/tests/acceptance/09_services/standard_services-from-non-default-namespace.cf @@ -0,0 +1,50 @@ +body common control +{ + inputs => { "../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +####################################################### + +bundle agent test +{ + meta: + "description" -> { "ENT-5406" } + string => "Test that standard_services can be used from non-default namespace"; + + methods: + "Test" usebundle => ENT_5406:ns_test; +} + +bundle agent standard_services(service, state) { + files: + "$(G.testfile)" + create => "true"; +} + +####################################################### + +bundle agent check +{ + classes: + "ok" expression => fileexists( $(G.testfile) ); + + reports: + ok:: + "$(this.promise_filename) Pass"; + !ok:: + "$(this.promise_filename) FAIL"; +} + +body file control +{ + namespace => "ENT_5406"; +} + +bundle agent ns_test +{ + services: + "myservice" + service_policy => "start"; +} From bd08cc9782e00b749be67dc26ef626515e2fac4f Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Tue, 10 Mar 2020 10:32:13 -0500 Subject: [PATCH 053/333] Fixed selection of standard_services when used from non-default namespace When no service_method was specified the standard_services services bundle should be used. This default only worked if you were already in the default namespace. This change makes sure that the standard_services method used is from the default namespace and fixes the case where services promises are used from non default namespaces expecting to leverage the standard_services implementation. Ticket: ENT-5406 Changelog: Title (cherry picked from commit e944d6363cb42fa6c00c1e327890672dcdfee49a) --- cf-agent/verify_services.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cf-agent/verify_services.c b/cf-agent/verify_services.c index 2e97c720d9..fc1be71bae 100644 --- a/cf-agent/verify_services.c +++ b/cf-agent/verify_services.c @@ -194,7 +194,7 @@ static FnCall *DefaultServiceBundleCall(const Promise *pp, const char *service_p } else { - call = FnCallNew("standard_services", args); + call = FnCallNew("default:standard_services", args); } return call; From fa869047031cb754db2d8bcb03aef07239474c83 Mon Sep 17 00:00:00 2001 From: Lex-2008 Date: Tue, 10 Mar 2020 14:24:15 +0100 Subject: [PATCH 054/333] Added changelog for 3.15.1 --- ChangeLog | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/ChangeLog b/ChangeLog index c4a6bd2671..11bbbf6c9b 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,52 @@ +3.15.1: + - `cf-check diagnose --test-write` can now be used to test writing into LMDB files + (ENT-4484) + - Added cf-postgres requirement to cf-apache and cf-hub systemd units + (ENT-5125) + - Added new binary version macros: + - `@if maximum_version(x)` + - `@else` + - `@if between_versions(x, y)` + - `@if before_version(x)` + - `@if at_version(x)` + - `@if after_version(x)` + (CFE-3198) + - Added warning if CSV parser parses nothing from a non-empty file + (CFE-3256) + - Fixed an issue causing duplicate entries in sys.interfaces, and + sys.hardware (CFE-3046) + - Fixed crashes and memory leaks in JSON and mustache code + - Fixed memory leak in handling of inline JSON in policy evaluation + - Made classfiltercsv() fail properly on invalid class expression index + - Measurements promises with no match no longer produce errors + (ENT-5171) + - Moved error reading file in countlinesmatching() from verbose to error + (CFE-3234) + - Removed (USE AT YOUR OWN RISK) from cf-key help menu for -x + (ENT-5090) + - Variable references with nested parentheses no longer + cause errors (CFE-3242) + - Version macros now accept single digits (CFE-3198) + - cf-agent: report purged dirs and files as repaired in 'files' promises (CFE-3260) + - cf-check: Added a more user friendly message when trying to print unknown binary data + (ENT-5234) + - cf-check: Added data validation for cf_lastseen.lmdb (CFE-2988) + - cf-check: Added nice printing for nova_agent_executions.lmdb + (ENT-5234) + - cf-check: Added validation for timestamps in cf_lock.lmdb (CFE-2988) + - cf-check: Added validation for timestamps in lastseen.lmdb (CFE-2988) + - cf-check: Fixed issue causing repair to target the wrong database file + (ENT-5309) + - cf-check: Symlinked LMDB databases are now preserved in repair + Performs diagnosis and repair on symlink target instead of symlink. + Repaired files / copies are placed alongside symlink target. + In some cases, the symlink target is deleted to repair a corrupt + database, and the symlink is left as a broken symlink. This is + handled gracefully by the agent, it will be recreated. Broken + symlinks are now detected as an acceptable condition in diagnose, + it won't try to repair them or delete them. + (ENT-5162) + 3.15.0: - New policy function basename() added (CFE-3196) - Added read_module_protocol() policy function From 8dc72b537e24341672e0be47a0772b5ec2dc40b0 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Sun, 15 Mar 2020 08:39:04 -0500 Subject: [PATCH 055/333] Added changelog entry for ENT-5406 --- ChangeLog | 1 + 1 file changed, 1 insertion(+) diff --git a/ChangeLog b/ChangeLog index 11bbbf6c9b..d0489b68e6 100644 --- a/ChangeLog +++ b/ChangeLog @@ -46,6 +46,7 @@ symlinks are now detected as an acceptable condition in diagnose, it won't try to repair them or delete them. (ENT-5162) + - Fixed selection of standard_services by default when used from non-default namespace (ENT-5406) 3.15.0: - New policy function basename() added (CFE-3196) From 07f59de8b05569f63d12d297f2538e4928bbfd6e Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 13 Mar 2020 14:34:14 +0100 Subject: [PATCH 056/333] Define 'RELEASE' macro based on the $RELEASE environment variable This way we can propagate the environment variable to C sources. If there is no such environment variable, or the value is empty, we default to "1". Ticket: ENT-5348 Changelog: None (cherry picked from commit 415343e00d550ea5d937025ccb3498a86359fd9a) --- configure.ac | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/configure.ac b/configure.ac index 52eda33dd2..ac10c5ef84 100644 --- a/configure.ac +++ b/configure.ac @@ -80,6 +80,10 @@ m4_undefine([cfversion]) m4_undefine([cfversion_from_file]) m4_undefine([cfversion_from_env]) +AS_IF([test -n "$RELEASE"], + [AC_DEFINE_UNQUOTED(RELEASE, ["$RELEASE"], [Release number])], + [AC_DEFINE(RELEASE, ["1"], [Release number])]) + AC_DEFINE(BUILD_YEAR, esyscmd([date +%Y | tr -d '\n']), "Software build year") AC_DEFINE_UNQUOTED(ABS_TOP_SRCDIR, From ede3ff3571755a948d7aafce7d71b40bce7038df Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 13 Mar 2020 15:00:58 +0100 Subject: [PATCH 057/333] Always set $(sys.libdir) and $(sys.local_libdir) These no longer depend on version so they should be set in all cases not just when the CFEngine version is known and parsed. The code should not be in DiscoverVersion(), but that's out of scope of the changes done here. (cherry picked from commit a8aa6274c98fa2b3de99c666a1b96edb50cc9a57) --- libenv/sysinfo.c | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/libenv/sysinfo.c b/libenv/sysinfo.c index 6fa60a5d5a..4cfd35d727 100644 --- a/libenv/sysinfo.c +++ b/libenv/sysinfo.c @@ -388,33 +388,27 @@ void DiscoverVersion(EvalContext *ctx) int major = 0; int minor = 0; int patch = 0; - const char* const workdir = GetWorkDir(); - + char workbuf[CF_BUFSIZE]; if (sscanf(Version(), "%d.%d.%d", &major, &minor, &patch) == 3) { - char workbuf[CF_BUFSIZE]; - snprintf(workbuf, CF_MAXVARSIZE, "%d", major); EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_SYS, "cf_version_major", workbuf, CF_DATA_TYPE_STRING, "source=agent"); snprintf(workbuf, CF_MAXVARSIZE, "%d", minor); EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_SYS, "cf_version_minor", workbuf, CF_DATA_TYPE_STRING, "source=agent"); snprintf(workbuf, CF_MAXVARSIZE, "%d", patch); EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_SYS, "cf_version_patch", workbuf, CF_DATA_TYPE_STRING, "source=agent"); - - snprintf(workbuf, CF_BUFSIZE, "%s%cinputs%clib", - workdir, FILE_SEPARATOR, FILE_SEPARATOR); - - EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_SYS, "libdir", workbuf, CF_DATA_TYPE_STRING, "source=agent"); - - EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_SYS, "local_libdir", "lib", CF_DATA_TYPE_STRING, "source=agent"); } else { EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_SYS, "cf_version_major", "BAD VERSION " VERSION, CF_DATA_TYPE_STRING, "source=agent"); EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_SYS, "cf_version_minor", "BAD VERSION " VERSION, CF_DATA_TYPE_STRING, "source=agent"); EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_SYS, "cf_version_patch", "BAD VERSION " VERSION, CF_DATA_TYPE_STRING, "source=agent"); - EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_SYS, "libdir", workdir, CF_DATA_TYPE_STRING, "source=agent"); } + + EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_SYS, "local_libdir", "lib", CF_DATA_TYPE_STRING, "source=agent"); + + snprintf(workbuf, CF_BUFSIZE, "%s%cinputs%clib", GetWorkDir(), FILE_SEPARATOR, FILE_SEPARATOR); + EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_SYS, "libdir", workbuf, CF_DATA_TYPE_STRING, "source=agent"); } static void GetNameInfo3(EvalContext *ctx) From 8c3a79c809ce37bba78ff5a3ff9fa8d1dc489857 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 13 Mar 2020 15:05:32 +0100 Subject: [PATCH 058/333] Add a new $(sys.cf_version_release) variable Making the release number available in the policy. Ticket: ENT-5348 Changelog: Added $(sys.cf_version_release) variable (cherry picked from commit 70055112504505ec25408c2ab3eb23e8323dfb87) --- examples/variablesmatching.cf | 1 + libenv/sysinfo.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/examples/variablesmatching.cf b/examples/variablesmatching.cf index cdd01b42be..c82305143b 100644 --- a/examples/variablesmatching.cf +++ b/examples/variablesmatching.cf @@ -44,5 +44,6 @@ bundle agent run #@ R: Variables matching 'default:sys.cf_version.*' = default:sys.cf_version_major #@ R: Variables matching 'default:sys.cf_version.*' = default:sys.cf_version_minor #@ R: Variables matching 'default:sys.cf_version.*' = default:sys.cf_version_patch +#@ R: Variables matching 'default:sys.cf_version.*' = default:sys.cf_version_release #@ ``` #+end_src diff --git a/libenv/sysinfo.c b/libenv/sysinfo.c index 4cfd35d727..d2446aa92a 100644 --- a/libenv/sysinfo.c +++ b/libenv/sysinfo.c @@ -405,6 +405,8 @@ void DiscoverVersion(EvalContext *ctx) EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_SYS, "cf_version_patch", "BAD VERSION " VERSION, CF_DATA_TYPE_STRING, "source=agent"); } + EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_SYS, "cf_version_release", RELEASE, CF_DATA_TYPE_STRING, "source=agent"); + EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_SYS, "local_libdir", "lib", CF_DATA_TYPE_STRING, "source=agent"); snprintf(workbuf, CF_BUFSIZE, "%s%cinputs%clib", GetWorkDir(), FILE_SEPARATOR, FILE_SEPARATOR); From 7523178747606718019ae1a4db7960c54901c3af Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 20 Mar 2020 16:38:56 +0100 Subject: [PATCH 059/333] Allow cf-serverd to connect to hub for call-collect When call-collect (client initiated reporting) is used, the cf-serverd process needs to connect to the hub. This is not allowed by default for constrained SELinux domains so we need to add a rule for that. Ticket: ENT-5415 Changelog: Client initiated reporting was fixed on RHEL 8.1 (cherry picked from commit a00b9f8b7b5c0284fc2db56fd595e9cb0294c87a) --- misc/selinux/cfengine-enterprise.te | 3 +++ 1 file changed, 3 insertions(+) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 574f95252c..62e3e0a3f6 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -235,6 +235,9 @@ allow cfengine_serverd_t cfengine_var_lib_t:file execute; # allow cf-serverd to execute cf-promises allow cfengine_serverd_t cfengine_var_lib_t:file execute_no_trans; +# allow cf-serverd to connect in case of call-collect +allow cfengine_serverd_t unreserved_port_t:tcp_socket name_connect; + allow cfengine_serverd_t cfengine_execd_exec_t:file getattr; allow cfengine_serverd_t cfengine_monitord_exec_t:file getattr; From e31a9939a322240240aecc55498431e29086b1f8 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 20 Mar 2020 16:46:07 +0100 Subject: [PATCH 060/333] Add a few more rules for our daemons to reduce SELinux noise Our daemons are now trying to 'stat()' /usr/sbin/semanage which was added to 'paths.cf' in MPF. And they need some more capabilities. Rules based on the output of the following command on a CentOS 8.1 host running CFEngine: ausearch -m AVC -se cfengine | audit2allow (cherry picked from commit ad37783619c85724d073825db2ee8570c0d6bd05) --- misc/selinux/cfengine-enterprise.te | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 62e3e0a3f6..cb53019a07 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -47,6 +47,7 @@ require { type crontab_exec_t; type hostname_exec_t; type groupadd_exec_t; + type semanage_exec_t; role system_r; class tcp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown name_connect accept listen name_bind }; class udp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; @@ -116,7 +117,7 @@ require { class filesystem getattr; class lnk_file { create getattr read unlink }; class unix_stream_socket connectto; - class capability { dac_read_search sys_module chown dac_read_search dac_override fowner fsetid sys_admin mknod net_raw net_admin sys_nice sys_rawio sys_resource setuid setgid sys_nice }; + class capability { dac_read_search sys_module chown dac_read_search dac_override fowner fsetid sys_admin mknod net_raw net_admin sys_nice sys_rawio sys_resource setuid setgid sys_nice sys_ptrace}; class capability2 { mac_admin mac_override block_suspend syslog compromise_kernel wake_alarm }; class association { sendto recvfrom setcontext polmatch }; class security setsecparam; @@ -167,6 +168,8 @@ allow cfengine_execd_t cfengine_log_t:lnk_file { create getattr read unlink }; allow cfengine_execd_t cfengine_monitord_exec_t:file getattr; allow cfengine_execd_t cfengine_serverd_exec_t:file getattr; +allow cfengine_execd_t self:capability sys_ptrace; + allow cfengine_execd_t crontab_exec_t:file getattr; allow cfengine_execd_t dmidecode_exec_t:file getattr; allow cfengine_execd_t fs_t:filesystem getattr; @@ -192,6 +195,7 @@ allow cfengine_execd_t unreserved_port_t:tcp_socket name_connect; allow cfengine_execd_t user_cron_spool_t:dir getattr; allow cfengine_execd_t useradd_exec_t:file getattr; allow cfengine_execd_t var_t:dir read; +allow cfengine_execd_t semanage_exec_t:file getattr; #============= cfengine_monitord_t ============== @@ -206,6 +210,8 @@ allow cfengine_monitord_t cfengine_execd_exec_t:file getattr; allow cfengine_monitord_t cfengine_serverd_exec_t:file getattr; allow cfengine_monitord_t cfengine_agent_exec_t:file getattr; +allow cfengine_monitord_t self:capability { dac_override dac_read_search sys_ptrace }; + allow cfengine_monitord_t crontab_exec_t:file getattr; allow cfengine_monitord_t dmidecode_exec_t:file getattr; allow cfengine_monitord_t groupadd_exec_t:file getattr; @@ -219,6 +225,7 @@ allow cfengine_monitord_t systemd_systemctl_exec_t:file getattr; allow cfengine_monitord_t user_cron_spool_t:dir getattr; allow cfengine_monitord_t useradd_exec_t:file getattr; allow cfengine_monitord_t var_t:dir read; +allow cfengine_monitord_t semanage_exec_t:file getattr; #============= cfengine_serverd_t ============== @@ -261,3 +268,4 @@ allow cfengine_serverd_t unreserved_port_t:tcp_socket name_bind; allow cfengine_serverd_t user_cron_spool_t:dir getattr; allow cfengine_serverd_t useradd_exec_t:file getattr; allow cfengine_serverd_t var_t:dir read; +allow cfengine_serverd_t semanage_exec_t:file getattr; From dabf113b910a1fe1f269c3d6dc5ef6bcd134f92e Mon Sep 17 00:00:00 2001 From: Kristian Amlie Date: Thu, 12 Mar 2020 15:35:43 +0100 Subject: [PATCH 061/333] ENT-5329: Don't go back further than field start when parsing CMD field. This can happen if the CMD field is empty and filled with space, for example in the case of zombie processes. Presumably the reason this has passed before is because it would later result in a strndup call with a negative size, which on most platforms would just mean the same as zero. However, on AIX 7 this crashes instead, which is how the issue was detected (split_process_line_test:test_platform_extra_table test). Changelog: Fix rare crashing bug when parsing zombie entries in ps output. The problem was only ever observed on AIX, but could theoretically happen on any platform depending on exact libc behavior. Signed-off-by: Kristian Amlie (cherry picked from commit b79535b2a72e4efd7431f19e70cd0bd225f5a6f8) --- libpromises/processes_select.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libpromises/processes_select.c b/libpromises/processes_select.c index a3aa26fa59..b5b59aa669 100644 --- a/libpromises/processes_select.c +++ b/libpromises/processes_select.c @@ -882,7 +882,7 @@ Solaris 9: { // Last field, slurp up the rest, but discard trailing whitespace. last = linelen; - while (isspace(line[last - 1])) + while (last > pos && isspace(line[last - 1])) { last--; } From ded82ac9dd55a376d137664a566e8f00d0dd1211 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Fri, 27 Mar 2020 13:13:45 +0100 Subject: [PATCH 062/333] Bumped .CFVERSION number Signed-off-by: Ole Herman Schumacher Elgesem --- .CFVERSION | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.CFVERSION b/.CFVERSION index c3df54c9b8..861845e450 100644 --- a/.CFVERSION +++ b/.CFVERSION @@ -1 +1 @@ -3.15.1 +3.15.2 From a5ce3da8015037ec4e899805e9786a6dd59f1dd4 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Tue, 7 Apr 2020 11:01:56 -0500 Subject: [PATCH 063/333] Added more tests around user promise secondary groups attribute (cherry picked from commit 573d37ae51416b8dbee79ccdaecde13b10c73a10) --- .../unsafe/10_password_hash_is_not_cached.cf | 6 +- .../20_modify_user_missing_secondary_group.cf | 57 +++++++++++++++++++ .../unsafe/20_modify_user_secondary_groups.cf | 57 +++++++++++++++++++ .../17_users/unsafe/user_queries.cf.sub | 17 +++++- 4 files changed, 132 insertions(+), 5 deletions(-) create mode 100644 tests/acceptance/17_users/unsafe/20_modify_user_missing_secondary_group.cf create mode 100644 tests/acceptance/17_users/unsafe/20_modify_user_secondary_groups.cf diff --git a/tests/acceptance/17_users/unsafe/10_password_hash_is_not_cached.cf b/tests/acceptance/17_users/unsafe/10_password_hash_is_not_cached.cf index 46a2d6ba55..29aa62fd8c 100644 --- a/tests/acceptance/17_users/unsafe/10_password_hash_is_not_cached.cf +++ b/tests/acceptance/17_users/unsafe/10_password_hash_is_not_cached.cf @@ -123,12 +123,12 @@ bundle agent check { methods: "any" usebundle => user_has_password_hash("user1", "$(test.hash)", "user1_success", "user1_failure"), - classes => always("methods_run"); + classes => always("user1_methods_run"); "any" usebundle => user_has_password_hash("user2", "$(init.hash)", "user2_failure", "user2_success"), - classes => always("methods_run"); + classes => always("user2_methods_run"); classes: - "ready" expression => "methods_run"; + "ready" and => { "user1_methods_run", "user2_methods_run" }; "ok" and => { "user1_success", "!user1_failure", "user2_success", "!user2_failure" }; diff --git a/tests/acceptance/17_users/unsafe/20_modify_user_missing_secondary_group.cf b/tests/acceptance/17_users/unsafe/20_modify_user_missing_secondary_group.cf new file mode 100644 index 0000000000..e4409c14a5 --- /dev/null +++ b/tests/acceptance/17_users/unsafe/20_modify_user_missing_secondary_group.cf @@ -0,0 +1,57 @@ +body common control +{ + inputs => { "../../default.cf.sub", "user_queries.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; +} + +bundle agent init +{ + commands: + "/usr/bin/sudo $(user_tests.pw_path) userdel testu"; + "/usr/bin/sudo $(user_tests.pw_path) delgroup testu"; + "/usr/bin/sudo $(user_tests.pw_path) delgroup testg1"; + "/usr/bin/sudo $(user_tests.pw_path) delgroup testg2"; + "/usr/bin/sudo $(user_tests.pw_path) groupadd testg1"; + "/usr/bin/sudo $(user_tests.pw_path) useradd testu"; + freebsd:: + "/usr/bin/sudo $(user_tests.pw_path) usermod testu --groups testg1 --append"; + !freebsd:: + "/usr/bin/sudo $(user_tests.pw_path) usermod --groups testg1 --append testu"; +} + +bundle agent test +{ + meta: + "description" -> { "ENT-3710" } + string => "Managing a user without specifying any secondary groups attribute + should result in no changes to existing secondary groups."; + + users: + "testu" + policy => "present"; +} + +bundle agent check +{ + methods: + "any" usebundle => user_is_in_secondary_group("testu", "testg1", "testg1_success", "testg1_failure"), + classes => always("testg1_method_run"); + + "any" usebundle => user_is_in_secondary_group("testu", "testg2", "testg2_success", "testg2_failure"), + classes => always("testg2_method_run"); + + "any" usebundle => user_exists("testu", "testu_success", "testu_failure"), + classes => always("testu_method_run"); + + classes: + "ready" and => { "testg1_method_run", "testg2_method_run", "testu_method_run" }; + "ok" and => { "testg1_success", "!testg1_failure", + "!testg2_success", "testg2_failure", + "testu_success", "!testu_failure" }; + + reports: + ok.ready:: + "$(this.promise_filename) Pass"; + !ok.ready:: + "$(this.promise_filename) FAIL"; +} diff --git a/tests/acceptance/17_users/unsafe/20_modify_user_secondary_groups.cf b/tests/acceptance/17_users/unsafe/20_modify_user_secondary_groups.cf new file mode 100644 index 0000000000..088d8bea56 --- /dev/null +++ b/tests/acceptance/17_users/unsafe/20_modify_user_secondary_groups.cf @@ -0,0 +1,57 @@ +body common control +{ + inputs => { "../../default.cf.sub", "user_queries.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; +} + +bundle agent init +{ + commands: + "/usr/bin/sudo $(user_tests.pw_path) userdel testu"; + "/usr/bin/sudo $(user_tests.pw_path) delgroup testu"; + "/usr/bin/sudo $(user_tests.pw_path) delgroup testg1"; + "/usr/bin/sudo $(user_tests.pw_path) delgroup testg2"; + "/usr/bin/sudo $(user_tests.pw_path) groupadd testg1"; + "/usr/bin/sudo $(user_tests.pw_path) useradd testu"; + freebsd:: + "/usr/bin/sudo $(user_tests.pw_path) usermod testu --groups testg1 --append"; + !freebsd:: + "/usr/bin/sudo $(user_tests.pw_path) usermod --groups testg1 --append testu"; +} + +bundle agent test +{ + meta: + "description" -> { "ENT-3710" } + string => "Adding a secondary group which doesn't exist should not remove existing secondary groups"; + + users: + "testu" + groups_secondary => { "testg1", "testg2" }, + policy => "present"; +} + +bundle agent check +{ + methods: + "any" usebundle => user_is_in_secondary_group("testu", "testg1", "testg1_success", "testg1_failure"), + classes => always("testg1_method_run"); + + "any" usebundle => user_is_in_secondary_group("testu", "testg2", "testg2_success", "testg2_failure"), + classes => always("testg2_method_run"); + + "any" usebundle => user_exists("testu", "testu_success", "testu_failure"), + classes => always("testu_method_run"); + + classes: + "ready" and => { "testg1_method_run", "testg2_method_run", "testu_method_run" }; + "ok" and => { "testg1_success", "!testg1_failure", + "!testg2_success", "testg2_failure", + "testu_success", "!testu_failure" }; + + reports: + ok.ready:: + "$(this.promise_filename) Pass"; + !ok.ready:: + "$(this.promise_filename) FAIL"; +} diff --git a/tests/acceptance/17_users/unsafe/user_queries.cf.sub b/tests/acceptance/17_users/unsafe/user_queries.cf.sub index cf9bd07211..7f3824d29c 100644 --- a/tests/acceptance/17_users/unsafe/user_queries.cf.sub +++ b/tests/acceptance/17_users/unsafe/user_queries.cf.sub @@ -14,6 +14,13 @@ bundle common user_tests "group1" string => "Users"; "group2" string => "Administrators"; + freebsd:: + "pw_path" string => "/usr/sbin/pw"; + "sudo_path" string => "/usr/local/bin/sudo"; + !freebsd:: + "pw_path" string => ""; + "sudo_path" string => "/usr/bin/sudo"; + classes: # On HPUX we use a sudo hack, which doesn't support '-u'. !windows.!hpux:: @@ -22,10 +29,14 @@ bundle common user_tests hpux:: "passwords_in_shadow" expression => fileexists("/etc/shadow"); "passwords_in_passwd" not => fileexists("/etc/shadow"); + freebsd:: + "passwords_in_shadow" expression => "!any"; + "passwords_in_passwd" expression => "!any"; + "passwords_in_master_passwd" expression => "any"; windows|aix:: "passwords_in_shadow" expression => "!any"; "passwords_in_passwd" expression => "!any"; - !hpux.!windows.!aix:: + !hpux.!windows.!aix.!freebsd:: "passwords_in_shadow" expression => "any"; "passwords_in_passwd" expression => "!any"; windows|aix:: @@ -43,7 +54,7 @@ bundle agent remove_stale_groups "rmgroup johndoe" contain => in_shell; !aix:: - "groupdel johndoe" + "$(user_tests.pw_path) groupdel johndoe" contain => in_shell; } @@ -215,6 +226,8 @@ bundle agent user_has_password_hash(user, hash, true_class, false_class) "passwd_file" string => "/etc/passwd"; passwords_in_shadow:: "passwd_file" string => "/etc/shadow"; + passwords_in_master_passwd:: + "passwd_file" string => "/etc/master.passwd"; classes: aix:: From ed3daacadf3c1af2d067d204a64fc280ad41d03c Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Wed, 15 Apr 2020 09:59:32 -0500 Subject: [PATCH 064/333] Added some needed command paths to acceptance test framework (cherry picked from commit efc4d2f2a6a69795b896f921d738ec45dfb5554b) --- .../20_modify_user_missing_secondary_group.cf | 16 ++++++++-------- .../unsafe/20_modify_user_secondary_groups.cf | 16 ++++++++-------- tests/acceptance/dcs.cf.sub | 6 ++++++ 3 files changed, 22 insertions(+), 16 deletions(-) diff --git a/tests/acceptance/17_users/unsafe/20_modify_user_missing_secondary_group.cf b/tests/acceptance/17_users/unsafe/20_modify_user_missing_secondary_group.cf index e4409c14a5..56714687eb 100644 --- a/tests/acceptance/17_users/unsafe/20_modify_user_missing_secondary_group.cf +++ b/tests/acceptance/17_users/unsafe/20_modify_user_missing_secondary_group.cf @@ -7,16 +7,16 @@ body common control bundle agent init { commands: - "/usr/bin/sudo $(user_tests.pw_path) userdel testu"; - "/usr/bin/sudo $(user_tests.pw_path) delgroup testu"; - "/usr/bin/sudo $(user_tests.pw_path) delgroup testg1"; - "/usr/bin/sudo $(user_tests.pw_path) delgroup testg2"; - "/usr/bin/sudo $(user_tests.pw_path) groupadd testg1"; - "/usr/bin/sudo $(user_tests.pw_path) useradd testu"; + "$(G.sudo) $(user_tests.pw_path) $(G.userdel) testu"; + "$(G.sudo) $(user_tests.pw_path) $(G.groupdel) testu"; + "$(G.sudo) $(user_tests.pw_path) $(G.groupdel) testg1"; + "$(G.sudo) $(user_tests.pw_path) $(G.groupdel) testg2"; + "$(G.sudo) $(user_tests.pw_path) $(G.groupadd) testg1"; + "$(G.sudo) $(user_tests.pw_path) $(G.useradd) testu"; freebsd:: - "/usr/bin/sudo $(user_tests.pw_path) usermod testu --groups testg1 --append"; + "$(G.sudo) $(user_tests.pw_path) $(G.usermod) testu -G testg1"; !freebsd:: - "/usr/bin/sudo $(user_tests.pw_path) usermod --groups testg1 --append testu"; + "$(G.sudo) $(user_tests.pw_path) $(G.usermod) -G testg1 testu"; } bundle agent test diff --git a/tests/acceptance/17_users/unsafe/20_modify_user_secondary_groups.cf b/tests/acceptance/17_users/unsafe/20_modify_user_secondary_groups.cf index 088d8bea56..d84054ded6 100644 --- a/tests/acceptance/17_users/unsafe/20_modify_user_secondary_groups.cf +++ b/tests/acceptance/17_users/unsafe/20_modify_user_secondary_groups.cf @@ -7,16 +7,16 @@ body common control bundle agent init { commands: - "/usr/bin/sudo $(user_tests.pw_path) userdel testu"; - "/usr/bin/sudo $(user_tests.pw_path) delgroup testu"; - "/usr/bin/sudo $(user_tests.pw_path) delgroup testg1"; - "/usr/bin/sudo $(user_tests.pw_path) delgroup testg2"; - "/usr/bin/sudo $(user_tests.pw_path) groupadd testg1"; - "/usr/bin/sudo $(user_tests.pw_path) useradd testu"; + "$(G.sudo) $(user_tests.pw_path) $(G.userdel) testu"; + "$(G.sudo) $(user_tests.pw_path) $(G.groupdel) testu"; + "$(G.sudo) $(user_tests.pw_path) $(G.groupdel) testg1"; + "$(G.sudo) $(user_tests.pw_path) $(G.groupdel) testg2"; + "$(G.sudo) $(user_tests.pw_path) $(G.groupadd) testg1"; + "$(G.sudo) $(user_tests.pw_path) $(G.useradd) testu"; freebsd:: - "/usr/bin/sudo $(user_tests.pw_path) usermod testu --groups testg1 --append"; + "$(G.sudo) $(user_tests.pw_path) $(G.usermod) testu -G testg1"; !freebsd:: - "/usr/bin/sudo $(user_tests.pw_path) usermod --groups testg1 --append testu"; + "$(G.sudo) $(user_tests.pw_path) $(G.usermod) -G testg1 testu"; } bundle agent test diff --git a/tests/acceptance/dcs.cf.sub b/tests/acceptance/dcs.cf.sub index fb5a4e2d19..2c5a6f9711 100644 --- a/tests/acceptance/dcs.cf.sub +++ b/tests/acceptance/dcs.cf.sub @@ -53,6 +53,7 @@ bundle common G "sleep", "sort", "stat", + "sudo", "tee", "touch", "true", @@ -61,6 +62,7 @@ bundle common G }; # Special cases. "make_cmds" slist => { "gmake", "make" }; + "sbin_cmds" slist => { "groupadd", "groupdel", "useradd", "userdel", "usermod" }; any:: "paths[tests]" string => "$(this.promise_dirname)"; @@ -72,6 +74,7 @@ bundle common G !windows:: "paths[bin]" string => "/bin"; "paths[usr_bin]" string => "/usr/bin"; + "paths[usr_sbin]" string => "/usr/sbin"; "paths[usr_local_bin]" string => "/usr/local/bin"; "paths[usr_contrib_bin]" string => "/usr/contrib/bin"; windows:: @@ -99,6 +102,7 @@ bundle common G scope => "namespace"; # Special cases. + "$(paths_indices)_$(sbin_cmds)$(exeext)" expression => fileexists("$(paths[$(paths_indices)])$(const.dirsep)$(sbin_cmds)$(exeext)"); "$(paths_indices)_$(make_cmds)$(exeext)" expression => fileexists("$(paths[$(paths_indices)])$(const.dirsep)$(make_cmds)$(exeext)"); "has_make" expression => fileexists("$(paths[$(paths_indices)])$(const.dirsep)$(make_cmds)$(exeext)"), scope => "namespace"; @@ -112,6 +116,8 @@ bundle common G ifvarclass => canonify("$(paths_indices)_$(cmds)$(exeext)"); # Special cases. + "$(sbin_cmds)" string => "$(paths[$(paths_indices)])$(const.dirsep)$(sbin_cmds)$(exeext)", + ifvarclass => canonify("$(paths_indices)_$(sbin_cmds)$(exeext)"); # gmake has higher priority, so should come last. "make" string => "$(paths[$(paths_indices)])$(const.dirsep)make$(exeext)", ifvarclass => canonify("$(paths_indices)_make$(exeext)"); From 1a4d2d0821ead1f43bcfae72efa79a123d297454 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Wed, 8 Apr 2020 15:52:50 -0500 Subject: [PATCH 065/333] Added debug option to tests/acceptance/testall script (cherry picked from commit 6e7e3a475f251b5e7f865ff0720fdf602adad054) (cherry picked from commit c5c3c420cef260df7f341e24b0e485e6d1c86f3f) --- tests/acceptance/testall | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/tests/acceptance/testall b/tests/acceptance/testall index 4e795c42e2..ba0eacf9a4 100755 --- a/tests/acceptance/testall +++ b/tests/acceptance/testall @@ -270,6 +270,7 @@ usage() { echo " -jobs=[n] Run tests in parallel, works like make -j option. Note that some" echo " tests will always run one by one." echo " --verbose Run tests with verbose logging" + echo " --debug Run tests with debug logging" } workdir() { @@ -481,9 +482,12 @@ export CFENGINE_TEST_OVERRIDE_WORKDIR TEMP CFENGINE_TEST_OVERRIDE_EXTENSION_LIBR then printf "\"$LIBTOOL\" --mode=execute " >> "$WORKDIR/runtest" fi - printf "valgrind ${VALGRIND_OPTS} \"$AGENT\" $VERBOSE -Klf \"$TEST\" -D ${PASS_NUM:+test_pass_$PASS_NUM,}${BASECLASSES},${EXTRACLASSES} 2>&1\n" >> "$WORKDIR/runtest" + printf "valgrind ${VALGRIND_OPTS} \"$AGENT\" $VERBOSE $DEBUG -Klf \"$TEST\" -D ${PASS_NUM:+test_pass_$PASS_NUM,}${BASECLASSES},${EXTRACLASSES} 2>&1\n" >> "$WORKDIR/runtest" + elif [ x"$PRELOAD_ASAN" != x ] + then + printf "LD_PRELOAD=$PRELOAD_ASAN \"$AGENT\" $VERBOSE $DEBUG -Klf \"$TEST\" -D ${PASS_NUM:+test_pass_$PASS_NUM,}${BASECLASSES},${EXTRACLASSES}\n" >> "$WORKDIR/runtest" else - printf "\"$AGENT\" $VERBOSE -Klf \"$TEST\" -D ${PASS_NUM:+test_pass_$PASS_NUM,}${BASECLASSES},${EXTRACLASSES}\n" >> "$WORKDIR/runtest" + printf "\"$AGENT\" $VERBOSE $DEBUG -Klf \"$TEST\" -D ${PASS_NUM:+test_pass_$PASS_NUM,}${BASECLASSES},${EXTRACLASSES}\n" >> "$WORKDIR/runtest" fi chmod +x "$WORKDIR/runtest" @@ -847,6 +851,8 @@ do NO_CLEAN=1;; --verbose) VERBOSE="-v";; + --debug) + DEBUG="--debug";; --stay-in-workdir) # Internal option. Meant to keep sub invocations from interfering by # writing files only into the workdir. @@ -880,6 +886,7 @@ then # each test is horribly slow. echo "export GAINROOT=" > runtests.sh echo "export TESTALL_DO_NOT_RECURSE=1" >> runtests.sh + echo "export DEBUG='$DEBUG'" >> runtests.sh echo "export VERBOSE='$VERBOSE'" >> runtests.sh echo "$0 $ORIG_ARGS $@ &" >> runtests.sh # Note quote change. We want to keep below variables unexpanded. From 21ad747074e0f195c5797cb8ff64d5256ac02c95 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Thu, 16 Apr 2020 10:11:39 -0500 Subject: [PATCH 066/333] Adjustments to acceptance tests for AIX mkgroup and rmgroup (cherry picked from commit 7b9187839a1121a9e0fdff2cea2613f3c3fa4b49) --- tests/acceptance/dcs.cf.sub | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/tests/acceptance/dcs.cf.sub b/tests/acceptance/dcs.cf.sub index 2c5a6f9711..d2163c60c0 100644 --- a/tests/acceptance/dcs.cf.sub +++ b/tests/acceptance/dcs.cf.sub @@ -37,6 +37,7 @@ bundle common G "ls", "mkdir", "mkfifo", + "mkgroup", # AIX equivalent of groupadd "mock_package_manager", "mv", "no_fds", @@ -62,7 +63,8 @@ bundle common G }; # Special cases. "make_cmds" slist => { "gmake", "make" }; - "sbin_cmds" slist => { "groupadd", "groupdel", "useradd", "userdel", "usermod" }; + # rmgroup is AIX equivalent of groupdel + "sbin_cmds" slist => { "rmgroup", "groupadd", "groupdel", "useradd", "userdel", "usermod" }; any:: "paths[tests]" string => "$(this.promise_dirname)"; @@ -124,6 +126,11 @@ bundle common G "make" string => "$(paths[$(paths_indices)])$(const.dirsep)gmake$(exeext)", ifvarclass => canonify("$(paths_indices)_gmake$(exeext)"); + # on AIX groupdel is rmgroup and groupadd is mkgroup + aix:: + "groupdel" string => "$(G.rmgroup)"; + "groupadd" string => "$(G.mkgroup)"; + want_color.have_colordiff:: "diff_maybecolor" string => $(colordiff); !want_color|!have_colordiff:: From 51fa38bde99124f768bb43ff07aeb114eccaf422 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Tue, 21 Apr 2020 14:56:18 -0500 Subject: [PATCH 067/333] Improved management of secondary groups to avoid intermediary state failures Manual cherry pick of 2e40f70ba85349f965673b1c68b231035811f4d7 due to large number of changes between 3.15.x and master for verify_users_pam.c. Changelog: Title Ticket: ENT-3710 --- cf-agent/verify_users_pam.c | 158 ++++++++++++++++++------------------ 1 file changed, 77 insertions(+), 81 deletions(-) diff --git a/cf-agent/verify_users_pam.c b/cf-agent/verify_users_pam.c index b6ac366c6a..e29d426289 100644 --- a/cf-agent/verify_users_pam.c +++ b/cf-agent/verify_users_pam.c @@ -774,8 +774,9 @@ static bool SetAccountLocked(const char *puser, const char *hash, bool lock) return SetAccountLockExpiration(puser, lock); } +static void TransformGidsToGroups(StringSet **list); -static bool GroupGetUserMembership (const char *user, StringSet *result) +static bool GetGroupInfo (const char *user, const User *u, StringSet **groups_to_set, StringSet **groups_missing, StringSet **current_secondary_groups) { bool ret = true; struct group *group_info; @@ -787,6 +788,18 @@ static bool GroupGetUserMembership (const char *user, StringSet *result) return false; } + StringSet *wanted_groups = StringSetNew(); + if (u->groups_secondary_given) + { + for (Rlist *ptr = u->groups_secondary; ptr != NULL; ptr = ptr->next) + { + StringSetAdd(*groups_missing, xstrdup(RvalScalarValue(ptr->val))); + StringSetAdd(wanted_groups, xstrdup(RvalScalarValue(ptr->val))); + } + TransformGidsToGroups(groups_missing); + TransformGidsToGroups(&wanted_groups); + } + while (true) { errno = 0; @@ -806,20 +819,40 @@ static bool GroupGetUserMembership (const char *user, StringSet *result) } break; } + + + if (StringSetContains(wanted_groups, group_info->gr_name)) + { + StringSetRemove(*groups_missing, group_info->gr_name); + } + // At least on FreeBSD, gr_mem can be NULL: if (group_info->gr_mem != NULL) { - for (int i = 0; group_info->gr_mem[i] != NULL; i++) + bool found = false; + for (int i = 0; !found && group_info->gr_mem[i] != NULL; i++) { if (strcmp(user, group_info->gr_mem[i]) == 0) { - StringSetAdd(result, xstrdup(group_info->gr_name)); - break; + found = true; + StringSetAdd(*current_secondary_groups, xstrdup(group_info->gr_name)); + if (StringSetContains(wanted_groups, group_info->gr_name)) + { + StringSetAdd(*groups_to_set, xstrdup(group_info->gr_name)); + } } } + if (!found && StringSetContains(wanted_groups, group_info->gr_name)) + { + StringSetAdd(*groups_to_set, xstrdup(group_info->gr_name)); + } } +#ifdef __FreeBSD__ + free(group_info); +#endif } + StringSetDestroy(wanted_groups); fclose(fptr); return ret; @@ -929,7 +962,7 @@ static void TransformGidsToGroups(StringSet **list) } static bool VerifyIfUserNeedsModifs (const char *puser, const User *u, const struct passwd *passwd_info, - uint32_t *changemap) + uint32_t *changemap, StringSet *groups_to_set, StringSet *current_secondary_groups) { assert(u != NULL); if (u->description != NULL && strcmp (u->description, passwd_info->pw_gecos)) @@ -963,6 +996,10 @@ static bool VerifyIfUserNeedsModifs (const char *puser, const User *u, const str CFUSR_SETBIT (*changemap, i_password); } } + if (u->groups_secondary_given && !StringSetIsEqual(groups_to_set, current_secondary_groups)) + { + CFUSR_SETBIT (*changemap, i_groups); + } if (SafeStringLength(u->group_primary)) { @@ -1001,26 +1038,6 @@ static bool VerifyIfUserNeedsModifs (const char *puser, const User *u, const str } } - if (u->groups_secondary_given) - { - StringSet *wanted_groups = StringSetNew(); - for (Rlist *ptr = u->groups_secondary; ptr; ptr = ptr->next) - { - StringSetAdd(wanted_groups, xstrdup(RvalScalarValue(ptr->val))); - } - TransformGidsToGroups(&wanted_groups); - StringSet *current_groups = StringSetNew(); - if (!GroupGetUserMembership (puser, current_groups)) - { - CFUSR_SETBIT (*changemap, i_groups); - } - else if (!StringSetIsEqual (current_groups, wanted_groups)) - { - CFUSR_SETBIT (*changemap, i_groups); - } - StringSetDestroy(current_groups); - StringSetDestroy(wanted_groups); - } //////////////////////////////////////////// if (*changemap == 0) @@ -1253,7 +1270,7 @@ static bool DoRemoveUser (const char *puser, enum cfopaction action) return ExecuteUserCommand(puser, cmd, sizeof(cmd), "removing", "Removing"); } -static bool DoModifyUser (const char *puser, const User *u, const struct passwd *passwd_info, uint32_t changemap, enum cfopaction action) +static bool DoModifyUser (const char *puser, const User *u, const struct passwd *passwd_info, uint32_t changemap, enum cfopaction action, StringSet *groups_to_set) { assert(u != NULL); char cmd[CF_BUFSIZE]; @@ -1281,20 +1298,6 @@ static bool DoModifyUser (const char *puser, const User *u, const struct passwd StringAppend(cmd, "\"", sizeof(cmd)); } -#ifndef __hpux - // HP-UX does not support -G with empty argument - if (CFUSR_CHECKBIT (changemap, i_groups) != 0) - { - /* Work around bug on SUSE. If secondary groups contain a group that is - the same group as the primary group, the secondary group is not set. - This happens even if the primary group is changed in the same call. - Therefore, set an empty group list first, and then set it to the real - list later. - */ - StringAppend(cmd, " -G \"\"", sizeof(cmd)); - } -#endif - if (CFUSR_CHECKBIT (changemap, i_home) != 0) { StringAppend(cmd, " -d \"", sizeof(cmd)); @@ -1309,9 +1312,6 @@ static bool DoModifyUser (const char *puser, const User *u, const struct passwd StringAppend(cmd, "\"", sizeof(cmd)); } - StringAppend(cmd, " ", sizeof(cmd)); - StringAppend(cmd, puser, sizeof(cmd)); - if (CFUSR_CHECKBIT (changemap, i_password) != 0) { if (action == cfa_warn || DONTDO) @@ -1360,6 +1360,19 @@ static bool DoModifyUser (const char *puser, const User *u, const struct passwd } } + if (CFUSR_CHECKBIT (changemap, i_groups) != 0) + { + StringAppend(cmd, " -G \"", sizeof(cmd)); + Buffer *buf = BufferNew(); + buf = StringSetToBuffer(groups_to_set, ','); + StringAppend(cmd, buf->buffer, sizeof(cmd)); + BufferDestroy(buf); + StringAppend(cmd, "\" ", sizeof(cmd)); + } +#ifndef HAVE_PW + StringAppend(cmd, " ", sizeof(cmd)); + StringAppend(cmd, puser, sizeof(cmd)); +#endif // If password and locking were the only things changed, don't run the command. CFUSR_CLEARBIT(changemap, i_password); CFUSR_CLEARBIT(changemap, i_locked); @@ -1370,38 +1383,10 @@ static bool DoModifyUser (const char *puser, const User *u, const struct passwd } else if (changemap != 0) { -#ifdef __hpux - // This is to overcome the Suse hack above which does not work on HP-UX and thus we - // risk getting an empty command if change of secondary groups is the only change - // Only run for other changes than i_groups, otherwise the command will be empty - uint32_t changemap_without_groups = changemap; - CFUSR_CLEARBIT(changemap_without_groups, i_groups); - if(changemap_without_groups != 0) -#endif - { - if (!ExecuteUserCommand(puser, cmd, sizeof(cmd), "modifying", "Modifying")) - { - return false; - } - } - if (CFUSR_CHECKBIT (changemap, i_groups) != 0) + + if (!ExecuteUserCommand(puser, cmd, sizeof(cmd), "modifying", "Modifying")) { - // Set real group list (see -G comment earlier). - strcpy(cmd, USERMOD); - StringAppend(cmd, " -G \"", sizeof(cmd)); - char sep[2] = { '\0', '\0' }; - for (Rlist *i = u->groups_secondary; i; i = i->next) - { - StringAppend(cmd, sep, sizeof(cmd)); - StringAppend(cmd, RvalScalarValue(i->val), sizeof(cmd)); - sep[0] = ','; - } - StringAppend(cmd, "\" ", sizeof(cmd)); - StringAppend(cmd, puser, sizeof(cmd)); - if (!ExecuteUserCommand(puser, cmd, sizeof(cmd), "modifying", "Modifying")) - { - return false; - } + return false; } } return true; @@ -1452,6 +1437,9 @@ void VerifyOneUsersPromise (const char *puser, const User *u, PromiseResult *res bool res; struct passwd *passwd_info; + StringSet *groups_to_set = StringSetNew(); + StringSet *current_secondary_groups = StringSetNew(); + StringSet *groups_missing = StringSetNew(); passwd_info = GetPwEntry(puser); if (!passwd_info && errno != 0) { @@ -1463,22 +1451,30 @@ void VerifyOneUsersPromise (const char *puser, const User *u, PromiseResult *res { if (passwd_info) { - uint32_t cmap = 0; - if (VerifyIfUserNeedsModifs (puser, u, passwd_info, &cmap)) + res = GetGroupInfo(puser, u, &groups_to_set, &groups_missing, ¤t_secondary_groups); + if (res) { - res = DoModifyUser (puser, u, passwd_info, cmap, action); - if (res) + uint32_t cmap = 0; + if (VerifyIfUserNeedsModifs (puser, u, passwd_info, &cmap, groups_to_set, current_secondary_groups)) { - *result = PROMISE_RESULT_CHANGE; + res = DoModifyUser (puser, u, passwd_info, cmap, action, groups_to_set); + if (res) + { + *result = PROMISE_RESULT_CHANGE; + } + else + { + *result = PROMISE_RESULT_FAIL; + } } else { - *result = PROMISE_RESULT_FAIL; + *result = PROMISE_RESULT_NOOP; } } else { - *result = PROMISE_RESULT_NOOP; + *result = PROMISE_RESULT_FAIL; } } else From 52d0aa5f8e173b19203f01b8908db6ccc775bb45 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Wed, 22 Apr 2020 11:29:20 -0500 Subject: [PATCH 068/333] Added test for expected output Ticket: CFE-628 Changelog: None (cherry picked from commit 7ed51e8bfdcc55374c683d559ce523a643cef265) --- tests/acceptance/01_vars/01_basic/CFE-628.cf | 44 +++++++++++++++++++ .../01_vars/01_basic/CFE-628.cf.sub | 14 ++++++ 2 files changed, 58 insertions(+) create mode 100644 tests/acceptance/01_vars/01_basic/CFE-628.cf create mode 100644 tests/acceptance/01_vars/01_basic/CFE-628.cf.sub diff --git a/tests/acceptance/01_vars/01_basic/CFE-628.cf b/tests/acceptance/01_vars/01_basic/CFE-628.cf new file mode 100644 index 0000000000..c1db73e67a --- /dev/null +++ b/tests/acceptance/01_vars/01_basic/CFE-628.cf @@ -0,0 +1,44 @@ +########################################################### +# +# Test evaluation order expectations +# +########################################################### + +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default($(this.promise_filename)) }; + version => "1.0"; +} + +########################################################### + +bundle agent test +{ + meta: + "description" -> { "CFE-628" } + string => "Test that the output is as expected"; + + "test_skip_needs_work" -> { "CFE-628" } + string => "windows", + meta => { "CFE-628" }; + +} + +########################################################### + +bundle agent check +{ + vars: + "cmd" string => "$(sys.cf_agent) -Kf $(this.promise_filename).sub"; + + methods: + + "Pass/Fail" + usebundle => dcs_passif_output( + ".*found_the_file.*", # wanted, + ".*didnt_find_the_file.*", # unwanted, + "$(cmd)", # command, + $(this.promise_filename) ); + +} diff --git a/tests/acceptance/01_vars/01_basic/CFE-628.cf.sub b/tests/acceptance/01_vars/01_basic/CFE-628.cf.sub new file mode 100644 index 0000000000..af39da095b --- /dev/null +++ b/tests/acceptance/01_vars/01_basic/CFE-628.cf.sub @@ -0,0 +1,14 @@ +bundle agent main +{ + classes: + "found_the_file" expression => fileexists( $(this.promise_filename) ); + + vars: + found_the_file:: + "myvar" string => execresult("/bin/echo found_the_file", "noshell"); + !found_the_file:: + "myvar" string => execresult("/bin/echo didnt_find_the_file", "noshell"); + + reports: + "$(myvar)"; +} From cbafdde2a1eef74f493a06d591d011778f19ccdf Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Wed, 22 Apr 2020 10:35:45 -0500 Subject: [PATCH 069/333] Added test for CFE-2191 Ticket: CFE-2191 Changelog: None (cherry picked from commit 14798359882789b4d0b80348ccf0a5d8da90b51b) --- tests/acceptance/01_vars/01_basic/CFE-2191.cf | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 tests/acceptance/01_vars/01_basic/CFE-2191.cf diff --git a/tests/acceptance/01_vars/01_basic/CFE-2191.cf b/tests/acceptance/01_vars/01_basic/CFE-2191.cf new file mode 100644 index 0000000000..3ed5591cbc --- /dev/null +++ b/tests/acceptance/01_vars/01_basic/CFE-2191.cf @@ -0,0 +1,41 @@ +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default($(this.promise_filename)) }; + version => "1.0"; +} + +########################################################### + +bundle agent init +{ + vars: + "suffix" string => "devel"; +} +bundle agent test +{ + meta: + "description" -> { "CFE-2191" } + string => "Test that expansion of a nested variable in another bundle expands to the expected value"; +} + +########################################################### + +bundle agent check +{ + + vars: + "expected" string => "___.___"; + "var_devel" string => "___"; + + methods: + "Pass/Fail" + # First, init.suffix is expanded (to devel) + # Then, $(var_devel) is expanded (to ___) + usebundle => dcs_check_strcmp( "$(expected)", "$(var_$(init.suffix)).___", $(this.promise_filename), "no"); + + reports: + DEBUG|EXTRA:: + "Expect '$(expected)', got '$(var_$(test.suffix)).___'"; + +} From 39497096b088659b73b1f78198e6b082574baa40 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Thu, 16 Apr 2020 14:17:31 -0500 Subject: [PATCH 070/333] Fixed service status cfengine3 on systemd managed hosts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ENT-5528/master: Fixed service status cfengine3 on systemd managed hosts Before change: [root@host001 ~]# service cfengine3 status Warning: PID file '/var/cfengine/cf-execd.pid' does not contain the right PID ('' != '5991'). Should restart CFEngine. cf-execd is not running Warning: PID file '/var/cfengine/cf-serverd.pid' does not contain the right PID ('' != '5992'). Should restart CFEngine. cf-serverd is not running Warning: PID file '/var/cfengine/cf-monitord.pid' does not contain the right PID ('' != '5993'). Should restart CFEngine. cf-monitord is not running [root@host001 ~]# ps aux | grep [c]f- root 5991 0.0 3.1 137460 7648 ? Ss 17:33 0:01 /var/cfengine/bin/cf-execd --no-fork root 5992 0.0 6.0 239056 14468 ? Ss 17:33 0:04 /var/cfengine/bin/cf-serverd --no-fork root 5993 0.1 3.0 70264 7392 ? Ss 17:33 0:07 /var/cfengine/bin/cf-monitord --no-fork [root@host001 ~]# cat /var/cfengine/cf-execd.pid /var/cfengine/cf-serverd.pid /var/cfengine/cf-monitord.pid 5991 5992 5993 After change: [root@host001 ~]# service cfengine3 status â— cfengine3.service - CFEngine 3 umbrella service Loaded: loaded (/usr/lib/systemd/system/cfengine3.service; enabled; vendor preset: disabled) Active: active (exited) since Thu 2020-04-16 17:33:10 UTC; 1h 52min ago Docs: https://docs.cfengine.com/ https://tracker.mender.io Process: 5987 ExecStart=/bin/true (code=exited, status=0/SUCCESS) Main PID: 5987 (code=exited, status=0/SUCCESS) CGroup: /system.slice/cfengine3.service Apr 16 17:33:09 host001.example.com systemd[1]: Starting CFEngine 3 umbrella service... Apr 16 17:33:10 host001.example.com systemd[1]: Started CFEngine 3 umbrella service. â— cfengine3.service - CFEngine 3 umbrella service Loaded: loaded (/usr/lib/systemd/system/cfengine3.service; enabled; vendor preset: disabled) Active: active (exited) since Thu 2020-04-16 17:33:10 UTC; 1h 52min ago Docs: https://docs.cfengine.com/ https://tracker.mender.io Process: 5987 ExecStart=/bin/true (code=exited, status=0/SUCCESS) Main PID: 5987 (code=exited, status=0/SUCCESS) CGroup: /system.slice/cfengine3.service Apr 16 17:33:09 host001.example.com systemd[1]: Starting CFEngine 3 umbrella service... Apr 16 17:33:10 host001.example.com systemd[1]: Started CFEngine 3 umbrella service. â— cfengine3.service - CFEngine 3 umbrella service Loaded: loaded (/usr/lib/systemd/system/cfengine3.service; enabled; vendor preset: disabled) Active: active (exited) since Thu 2020-04-16 17:33:10 UTC; 1h 52min ago Docs: https://docs.cfengine.com/ https://tracker.mender.io Process: 5987 ExecStart=/bin/true (code=exited, status=0/SUCCESS) Main PID: 5987 (code=exited, status=0/SUCCESS) CGroup: /system.slice/cfengine3.service Apr 16 17:33:09 host001.example.com systemd[1]: Starting CFEngine 3 umbrella service... Apr 16 17:33:10 host001.example.com systemd[1]: Started CFEngine 3 umbrella service. Ticket: ENT-5528 Changelog: Title (cherry picked from commit 380a60c2c9571544e284c381d3b22bb6d60d7055) --- misc/init.d/cfengine3.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/misc/init.d/cfengine3.in b/misc/init.d/cfengine3.in index 11793d1ff0..577a85b3c5 100644 --- a/misc/init.d/cfengine3.in +++ b/misc/init.d/cfengine3.in @@ -226,7 +226,7 @@ cf_get_matching_pids() for pid in $pids; do # If the last character of the cgroup is not the root directory, # then this is a container process. - if [ -f /proc/$pid/cgroup ] && sed 's_/\(-\|system\|user\).slice__;s_/user-[0-9]*.slice__;s_/session-[0-9]*.scope__;s_/cfengine3.service__' /proc/$pid/cgroup | egrep '[0-9]+:devices:/[^:]' > /dev/null; then + if [ -f /proc/$pid/cgroup ] && sed 's_/\(-\|system\|user\).slice__;s_/user-[0-9]*.slice__;s_/session-[0-9]*.scope__;s_/cfengine3.service__;s_/cf-execd.service__;s_/cf-serverd.service__;s_/cf-monitord.service__' /proc/$pid/cgroup | egrep '[0-9]+:devices:/[^:]' > /dev/null; then if [ $INSIDE_CONTAINER = 1 ]; then echo $pid fi From beead7ece63c9e160e224fef4dcba1e94feded69 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Thu, 23 Apr 2020 13:55:09 -0500 Subject: [PATCH 071/333] Removed unneeded cfengine3-web.service.in The cfengine3-web service was removed from packaging in the following buildscripts commit and should not be needed here in core anymore. https://github.com/cfengine/buildscripts/commit/fdbe42f4b54e02230602a4d76b0a50aad4efe23c Changelog: None (cherry picked from commit 755f3cec3bc25f832f6403eaaa679e48e38df841) --- misc/systemd/cfengine3-web.service.in | 18 ------------------ 1 file changed, 18 deletions(-) delete mode 100644 misc/systemd/cfengine3-web.service.in diff --git a/misc/systemd/cfengine3-web.service.in b/misc/systemd/cfengine3-web.service.in deleted file mode 100644 index 42ccc551d0..0000000000 --- a/misc/systemd/cfengine3-web.service.in +++ /dev/null @@ -1,18 +0,0 @@ -[Unit] -Description=CFEngine 3 Mission Portal daemons -Documentation=https://docs.cfengine.com -After=syslog.target -After=network.target -PartOf=cfengine3.service -ConditionPathExists=@bindir@/cfengine3-nova-hub-init-d.sh - -[Service] -Type=forking -EnvironmentFile=@OS_ENVIRONMENT_PATH@/cfengine3 -Environment="CF_ONLY_WEB_SERVICES=1" -ExecStart=/etc/init.d/cfengine3 start -ExecStop=/etc/init.d/cfengine3 stop - -[Install] -WantedBy=multi-user.target -WantedBy=cfengine3.service From 4c6dcdfb769272341a097804dc9e4d031b2fb071 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Wed, 22 Apr 2020 11:14:26 -0500 Subject: [PATCH 072/333] Added acceptance test for ifelse() failure cases (cherry picked from commit d4c0721c8d3dd23a9175dfd14a41d0984d134de2) --- tests/acceptance/01_vars/02_functions/ifelse.cf | 2 ++ tests/acceptance/01_vars/02_functions/ifelse.cf.expected.json | 4 +++- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/tests/acceptance/01_vars/02_functions/ifelse.cf b/tests/acceptance/01_vars/02_functions/ifelse.cf index b933032bcb..e33851efc3 100644 --- a/tests/acceptance/01_vars/02_functions/ifelse.cf +++ b/tests/acceptance/01_vars/02_functions/ifelse.cf @@ -34,6 +34,8 @@ bundle agent test "five" string => ifelse("this is not true", "5 parameters broken", "this is also not true", "5 parameters broken 2", "5 parameters OK"); + "unresolved1" string => ifelse( isvariable( "missing" ), "$(missing)", "expected fallback 1"); + "unresolved2" string => ifelse( isvariable( "missing" ), "$($(missing))", "expected fallback 2"); any:: "mystring" string => ifelse("any","pass","!any","$(foo)","bar"); diff --git a/tests/acceptance/01_vars/02_functions/ifelse.cf.expected.json b/tests/acceptance/01_vars/02_functions/ifelse.cf.expected.json index 210a9c6b27..0689e517ef 100644 --- a/tests/acceptance/01_vars/02_functions/ifelse.cf.expected.json +++ b/tests/acceptance/01_vars/02_functions/ifelse.cf.expected.json @@ -5,5 +5,7 @@ "five": "5 parameters OK", "hardclass": "hardclass OK", "one": "1", - "single": "single string parameter" + "single": "single string parameter", + "unresolved1": "expected fallback 1", + "unresolved2": "expected fallback 2" } From 39e6821ea9417977fdd7c370f0be441d03b8eb6b Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Mon, 20 Apr 2020 13:34:56 -0500 Subject: [PATCH 073/333] Fixed ifelse() to return fallback in case of unresolved variables The ifelse() function is a special case where it should still be evaluated even if all parameters are unresolved. Changelog: Title Ticket: ENT-4653 (cherry picked from commit d6231acb53b565ce344443f104269a45ddf5c6ee) --- libpromises/expand.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/libpromises/expand.c b/libpromises/expand.c index 1dd92a2781..b1eb00600d 100644 --- a/libpromises/expand.c +++ b/libpromises/expand.c @@ -188,13 +188,21 @@ static void MapIteratorsFromRval(EvalContext *ctx, } static PromiseResult ExpandPromiseAndDo(EvalContext *ctx, PromiseIterator *iterctx, - PromiseActuator *act_on_promise, void *param) + PromiseActuator *act_on_promise, void *param, + bool actuate_ifelse) { PromiseResult result = PROMISE_RESULT_SKIPPED; + /* In the case of ifelse() we must always include an extra round of "actuation" + * in the while loop below. PromiseIteratorNext() will return false in the case + * that there are doubly-unresolved Rvals like $($(missing)). + * We can't add an empty wheel because that is skipped as well as noted in + * libpromises/iteration.c ShouldAddVariableAsIterationWheel(). */ + bool ifelse_actuated = !actuate_ifelse; + /* TODO this loop could be completely skipped for for non vars/classes if * act_on_promise is CommonEvalPromise(). */ - while (PromiseIteratorNext(iterctx, ctx)) + while (PromiseIteratorNext(iterctx, ctx) || !ifelse_actuated) { /* * ACTUAL WORK PART 1: Get a (another) copy of the promise. @@ -208,6 +216,7 @@ static PromiseResult ExpandPromiseAndDo(EvalContext *ctx, PromiseIterator *iterc if (pexp == NULL) /* is the promise excluded? */ { result = PromiseResultUpdate(result, PROMISE_RESULT_SKIPPED); + ifelse_actuated = true; continue; } @@ -239,6 +248,7 @@ static PromiseResult ExpandPromiseAndDo(EvalContext *ctx, PromiseIterator *iterc /* Why do we push/pop an iteration frame, if all iterated variables * are Put() on the previous scope? */ EvalContextStackPopFrame(ctx); + ifelse_actuated = true; } return result; @@ -272,16 +282,22 @@ PromiseResult ExpandPromise(EvalContext *ctx, const Promise *pp, MapIteratorsFromRval(ctx, iterctx, pcopy->promisee); } + bool actuate_ifelse = false; for (size_t i = 0; i < SeqLength(pcopy->conlist); i++) { Constraint *cp = SeqAt(pcopy->conlist, i); + if (cp->rval.type == RVAL_TYPE_FNCALL && + strcmp(RvalFnCallValue(cp->rval)->name, "ifelse") == 0) + { + actuate_ifelse = true; + } MapIteratorsFromRval(ctx, iterctx, cp->rval); } /* 3. GO! */ PutHandleVariable(ctx, pcopy); PromiseResult result = ExpandPromiseAndDo(ctx, iterctx, - act_on_promise, param); + act_on_promise, param, actuate_ifelse); EvalContextStackPopFrame(ctx); PromiseIteratorDestroy(iterctx); From 15d9906e3ffcb12c6be445fc5a0d8e00d0b9d5ab Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 24 Apr 2020 13:40:37 +0200 Subject: [PATCH 074/333] Evaluate 'if' even in case of class context or 'ifvarclass' If a class context is specified in the policy or if 'ifvarclass' constraint is used, the 'if' constraint should be evaluated as well. Ticket: CFE-2615 Changelog: 'if' constraint now works in combination with class contexts (cherry picked from commit 6e1dd86e56cc977b587c92ac18b4a62bee61fdd3) --- libpromises/promises.c | 33 ++++++++++++++----- .../variable_class_expressions_with_if.cf | 10 +++--- 2 files changed, 28 insertions(+), 15 deletions(-) diff --git a/libpromises/promises.c b/libpromises/promises.c index 723c10f3b7..079352862f 100644 --- a/libpromises/promises.c +++ b/libpromises/promises.c @@ -625,25 +625,40 @@ Promise *ExpandDeRefPromise(EvalContext *ctx, const Promise *pp, bool *excluded) /* Look for 'if'/'ifvarclass' exclusion. */ { + /* We need to make sure to check both 'if' and 'ifvarclass' constraints. */ + bool checked_if = false; const Constraint *ifvarclass = PromiseGetConstraint(pp, "ifvarclass"); if (!ifvarclass) { ifvarclass = PromiseGetConstraint(pp, "if"); + checked_if = true; } // if - Skip if false or error: - if (ifvarclass && (CheckVarClassExpression(ctx, ifvarclass, pcopy) != EXPRESSION_VALUE_TRUE)) + while (ifvarclass != NULL) { - if (LogGetGlobalLevel() >= LOG_LEVEL_VERBOSE) + if (CheckVarClassExpression(ctx, ifvarclass, pcopy) != EXPRESSION_VALUE_TRUE) { - char *ifvarclass_string = RvalToString(ifvarclass->rval); - Log(LOG_LEVEL_VERBOSE, "Skipping promise '%s'" - " because constraint '%s => %s' is not met", - pp->promiser, ifvarclass->lval, ifvarclass_string); - free(ifvarclass_string); + if (LogGetGlobalLevel() >= LOG_LEVEL_VERBOSE) + { + char *ifvarclass_string = RvalToString(ifvarclass->rval); + Log(LOG_LEVEL_VERBOSE, "Skipping promise '%s'" + " because constraint '%s => %s' is not met", + pp->promiser, ifvarclass->lval, ifvarclass_string); + free(ifvarclass_string); + } + *excluded = true; + return pcopy; + } + if (!checked_if) + { + ifvarclass = PromiseGetConstraint(pp, "if"); + checked_if = true; + } + else + { + ifvarclass = NULL; } - *excluded = true; - return pcopy; } } diff --git a/tests/acceptance/02_classes/01_basic/variable_class_expressions_with_if.cf b/tests/acceptance/02_classes/01_basic/variable_class_expressions_with_if.cf index 3f81806437..6015526242 100644 --- a/tests/acceptance/02_classes/01_basic/variable_class_expressions_with_if.cf +++ b/tests/acceptance/02_classes/01_basic/variable_class_expressions_with_if.cf @@ -9,10 +9,8 @@ bundle agent test { meta: - "description" string => "Test that variable class expressions and the if alias work together as expected."; - - "test_soft_fail" - string => "any", + "description" + string => "Test that variable class expressions and the if alias work together as expected.", meta => { "CFE-2615" }; vars: @@ -93,8 +91,8 @@ bundle agent test "Found class indicating failure '$(FAIL_classes)'"; FAIL_classes:: - "$(this.promise_dirname) FAIL"; + "$(this.promise_filename) FAIL"; no_FAIL_classes:: - "$(this.promise_dirname) Pass"; + "$(this.promise_filename) Pass"; } From 09aa4832a160444bf9642cb6283b1cae5d9f425b Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Tue, 5 May 2020 09:29:40 -0500 Subject: [PATCH 075/333] Added example illustrating use of cifs mount_type Ticket: CFE-3187 Changelog: None (cherry picked from commit f5ea3145f68b930e0e3e1ce15229ac7c8c0a0a63) --- examples/storage-cifs.cf | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 examples/storage-cifs.cf diff --git a/examples/storage-cifs.cf b/examples/storage-cifs.cf new file mode 100644 index 0000000000..3568f69c15 --- /dev/null +++ b/examples/storage-cifs.cf @@ -0,0 +1,30 @@ +bundle agent main +# @brief Example illustrating CIFS/Samba storage type promise mount +{ + vars: + redhat|centos:: + "cifs" data => '{ "server": "192.168.42.251", "path": "/Audio" }'; + + packages: + redhat|centos:: + "cifs-utils" policy => "present"; + "samba-client" policy => "present"; + + files: + redhat|centos:: + "/mnt/CIFS/." create => "true"; + + storage: + redhat|centos:: + "/mnt/CIFS" + mount => cifs_guest( $(cifs[server]) , $(cifs[path]) ); +} + +body mount cifs_guest(server,source) +{ + mount_type => "cifs"; + mount_source => "$(source)"; + mount_server => "$(server)"; + mount_options => { "guest" }; + edit_fstab => "false"; +} From a28e5031ddc67b93c865fd1c5a9ec752e19fb8c8 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Fri, 8 May 2020 11:13:48 -0500 Subject: [PATCH 076/333] Added example illustrating template_method => "inline_mustache" Ticket: ENT-5966 Changelog: None (cherry picked from commit c9192bd3339125ff4a4815a86117a76f51166992) --- examples/template_method-inline_mustache.cf | 72 +++++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100644 examples/template_method-inline_mustache.cf diff --git a/examples/template_method-inline_mustache.cf b/examples/template_method-inline_mustache.cf new file mode 100644 index 0000000000..7b6d3aabc6 --- /dev/null +++ b/examples/template_method-inline_mustache.cf @@ -0,0 +1,72 @@ +# Copyright 2020 Northern.tech AS + +# This file is part of Cfengine 3 - written and maintained by Northern.tech AS. + +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation; version 3. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA + +# To the extent this program is licensed as part of the Enterprise +# versions of Cfengine, the applicable Commercial Open Source License +# (COSL) may apply to this file if you as a licensee so wish it. See +# included file COSL.txt. + +############################################################################### +#+begin_src cfengine3 +bundle agent example_using_template_method_inline_mustache +{ + vars: + + # Here we construct a data container that will be passed to the mustache + # templating engine + + "d" + data => '{ "host": "docs.cfengine.com" }'; + + # Here we specify a string that will be used as an inline mustache template + "mustache_template_string" + string => "Welcome to host '{{{host}}}'"; + + files: + # Here we render the file using the data container and inline template specification + + "/tmp/example.txt" + create => "true", + template_method => "inline_mustache", + edit_template_string => "$(mustache_template_string)", + template_data => @(d); + + reports: + "/tmp/example.txt" + printfile => cat( $(this.promiser) ); +} + +# Copied from stdlib, lib/reports.cf +body printfile cat(file) +# @brief Report the contents of a file +# @param file The full path of the file to report +{ + file_to_print => "$(file)"; + number_of_lines => "inf"; +} +bundle agent __main__ +{ + methods: "example_using_template_method_inline_mustache"; +} +#+end_src +############################################################################### +#+begin_src example_output +#@ ``` +#@ R: /tmp/example.txt +#@ R: Welcome to host 'docs.cfengine.com' +#@ ``` +#+end_src From 8ed9d0857e5ce9ae2d6fb05b4ecc9d68fda6ecac Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Tue, 19 May 2020 17:19:35 +0200 Subject: [PATCH 077/333] libntech: Bumped to master Signed-off-by: Ole Herman Schumacher Elgesem --- libntech | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libntech b/libntech index c1ba310c89..0943bb572d 160000 --- a/libntech +++ b/libntech @@ -1 +1 @@ -Subproject commit c1ba310c89b823450598648a07f717b784feeca0 +Subproject commit 0943bb572d636b0fefb3378ae6f6692f990350dc From d422ec7052d30b17241aa4f26118d449bfc286f3 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 11 Mar 2020 13:40:29 +0100 Subject: [PATCH 078/333] configure.ac: Removed libntech automake files It should not list them, they are listed in libntech/configure.ac and have different AM_CONDITIONAL's Also updated libntech to latest master, which needs this. Changelog: None Ticket: None Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit a9266d386c1da0026abe602b240b9dd075df5f34) --- configure.ac | 3 --- 1 file changed, 3 deletions(-) diff --git a/configure.ac b/configure.ac index ac10c5ef84..2d89faf84e 100644 --- a/configure.ac +++ b/configure.ac @@ -1758,9 +1758,6 @@ dnl Now make the Makefiles dnl ###################################################################### AC_CONFIG_FILES([Makefile - libntech/libcompat/Makefile - libntech/libutils/Makefile - libntech/config.post.h libcfnet/Makefile libenv/Makefile libpromises/Makefile From aeb8bb1a3dae057d7b75afadbbf17ecb586e7bf1 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 9 Apr 2020 14:53:25 +0200 Subject: [PATCH 079/333] Add COMMANDS section to the cf-net and cf-keycrypt man pages Quite important piece of information that should be there. Conflicts: cf-keycrypt Ticket: CFE-3272 Changelog: None (cherry picked from commit c524680b88d594906f0f4690b17fd1e6490c2cc3) --- cf-agent/cf-agent.c | 1 + cf-check/cf-check.c | 1 + cf-execd/cf-execd.c | 1 + cf-key/cf-key.c | 1 + cf-monitord/cf-monitord.c | 1 + cf-net/cf-net.c | 1 + cf-promises/cf-promises.c | 1 + cf-runagent/cf-runagent.c | 1 + cf-serverd/cf-serverd-functions.c | 1 + 9 files changed, 9 insertions(+) diff --git a/cf-agent/cf-agent.c b/cf-agent/cf-agent.c index 09060490fd..1819881a18 100644 --- a/cf-agent/cf-agent.c +++ b/cf-agent/cf-agent.c @@ -537,6 +537,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) CF_AGENT_SHORT_DESCRIPTION, CF_AGENT_MANPAGE_LONG_DESCRIPTION, OPTIONS, HINTS, + NULL, false, true); FileWriterDetach(out); DoCleanupAndExit(EXIT_SUCCESS); diff --git a/cf-check/cf-check.c b/cf-check/cf-check.c index 23546fff31..548ba7fa8b 100644 --- a/cf-check/cf-check.c +++ b/cf-check/cf-check.c @@ -187,6 +187,7 @@ int main(int argc, const char *const *argv) CF_CHECK_SHORT_DESCRIPTION, CF_CHECK_MANPAGE_LONG_DESCRIPTION, OPTIONS, HINTS, + NULL, false, true); FileWriterDetach(out); return EXIT_SUCCESS; diff --git a/cf-execd/cf-execd.c b/cf-execd/cf-execd.c index c4ca62479d..5eecbaacbb 100644 --- a/cf-execd/cf-execd.c +++ b/cf-execd/cf-execd.c @@ -301,6 +301,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) CF_EXECD_SHORT_DESCRIPTION, CF_EXECD_MANPAGE_LONG_DESCRIPTION, OPTIONS, HINTS, + NULL, false, true); FileWriterDetach(out); DoCleanupAndExit(EXIT_SUCCESS); diff --git a/cf-key/cf-key.c b/cf-key/cf-key.c index 04e37282ff..5d679292c2 100644 --- a/cf-key/cf-key.c +++ b/cf-key/cf-key.c @@ -336,6 +336,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) CF_KEY_SHORT_DESCRIPTION, CF_KEY_MANPAGE_LONG_DESCRIPTION, OPTIONS, HINTS, + NULL, false, false); FileWriterDetach(out); DoCleanupAndExit(EXIT_SUCCESS); diff --git a/cf-monitord/cf-monitord.c b/cf-monitord/cf-monitord.c index 4557510d99..5c16ef4849 100644 --- a/cf-monitord/cf-monitord.c +++ b/cf-monitord/cf-monitord.c @@ -209,6 +209,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) CF_MONITORD_SHORT_DESCRIPTION, CF_MONITORD_MANPAGE_LONG_DESCRIPTION, OPTIONS, HINTS, + NULL, false, true); FileWriterDetach(out); DoCleanupAndExit(EXIT_SUCCESS); diff --git a/cf-net/cf-net.c b/cf-net/cf-net.c index f76eb9ea54..ca54c2d178 100644 --- a/cf-net/cf-net.c +++ b/cf-net/cf-net.c @@ -295,6 +295,7 @@ static int CFNetParse(int argc, char **argv, CF_NET_SHORT_DESCRIPTION, CF_NET_MANPAGE_LONG_DESCRIPTION, OPTIONS, HINTS, + COMMANDS, false, true); FileWriterDetach(out); DoCleanupAndExit(EXIT_SUCCESS); diff --git a/cf-promises/cf-promises.c b/cf-promises/cf-promises.c index 238a8a6e5a..f83596a959 100644 --- a/cf-promises/cf-promises.c +++ b/cf-promises/cf-promises.c @@ -427,6 +427,7 @@ GenericAgentConfig *CheckOpts(int argc, char **argv) CF_PROMISES_SHORT_DESCRIPTION, CF_PROMISES_MANPAGE_LONG_DESCRIPTION, OPTIONS, HINTS, + NULL, false, true); FileWriterDetach(out); DoCleanupAndExit(EXIT_SUCCESS); diff --git a/cf-runagent/cf-runagent.c b/cf-runagent/cf-runagent.c index 06db6b5285..02f31260dc 100644 --- a/cf-runagent/cf-runagent.c +++ b/cf-runagent/cf-runagent.c @@ -372,6 +372,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) CF_RUNAGENT_SHORT_DESCRIPTION, CF_RUNAGENT_MANPAGE_LONG_DESCRIPTION, OPTIONS, HINTS, + NULL, false, true); FileWriterDetach(out); DoCleanupAndExit(EXIT_SUCCESS); diff --git a/cf-serverd/cf-serverd-functions.c b/cf-serverd/cf-serverd-functions.c index c22d2f1642..4555b91970 100644 --- a/cf-serverd/cf-serverd-functions.c +++ b/cf-serverd/cf-serverd-functions.c @@ -250,6 +250,7 @@ GenericAgentConfig *CheckOpts(int argc, char **argv) CF_SERVERD_SHORT_DESCRIPTION, CF_SERVERD_MANPAGE_LONG_DESCRIPTION, OPTIONS, HINTS, + NULL, false, true); FileWriterDetach(out); DoCleanupAndExit(EXIT_SUCCESS); From a41c44ab56623b53bdd5f37f9e38298205c217a4 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 9 Apr 2020 12:00:50 +0200 Subject: [PATCH 080/333] Backported WriterWriteHelp changes to 3.15.x Conflicts: Lots, this was originally a commit about cf-keycrypt Ticket: CFE-3272 Changelog: None (cherry picked from commit 163ecc3b884ae3368319de897a1f91db216d6b3f) --- cf-agent/cf-agent.c | 4 ++-- cf-execd/cf-execd.c | 4 ++-- cf-key/cf-key.c | 2 +- cf-monitord/cf-monitord.c | 4 ++-- cf-net/cf-net.c | 2 +- cf-promises/cf-promises.c | 4 ++-- cf-runagent/cf-runagent.c | 4 ++-- cf-serverd/cf-serverd-functions.c | 4 ++-- cf-testd/cf-testd.c | 2 +- tests/acceptance/mock_package_manager.c | 2 +- 10 files changed, 16 insertions(+), 16 deletions(-) diff --git a/cf-agent/cf-agent.c b/cf-agent/cf-agent.c index 1819881a18..4974f08830 100644 --- a/cf-agent/cf-agent.c +++ b/cf-agent/cf-agent.c @@ -525,7 +525,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) case 'h': { Writer *w = FileWriter(stdout); - WriterWriteHelp(w, "cf-agent", OPTIONS, HINTS, true, NULL); + WriterWriteHelp(w, "cf-agent", OPTIONS, HINTS, NULL, false, true); FileWriterDetach(w); } DoCleanupAndExit(EXIT_SUCCESS); @@ -650,7 +650,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) default: { Writer *w = FileWriter(stdout); - WriterWriteHelp(w, "cf-agent", OPTIONS, HINTS, true, NULL); + WriterWriteHelp(w, "cf-agent", OPTIONS, HINTS, NULL, false, true); FileWriterDetach(w); } DoCleanupAndExit(EXIT_FAILURE); diff --git a/cf-execd/cf-execd.c b/cf-execd/cf-execd.c index 5eecbaacbb..0d58208896 100644 --- a/cf-execd/cf-execd.c +++ b/cf-execd/cf-execd.c @@ -289,7 +289,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) case 'h': { Writer *w = FileWriter(stdout); - WriterWriteHelp(w, "cf-execd", OPTIONS, HINTS, true, NULL); + WriterWriteHelp(w, "cf-execd", OPTIONS, HINTS, NULL, false, true); FileWriterDetach(w); } DoCleanupAndExit(EXIT_SUCCESS); @@ -352,7 +352,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) default: { Writer *w = FileWriter(stdout); - WriterWriteHelp(w, "cf-execd", OPTIONS, HINTS, true, NULL); + WriterWriteHelp(w, "cf-execd", OPTIONS, HINTS, NULL, false, true); FileWriterDetach(w); } DoCleanupAndExit(EXIT_FAILURE); diff --git a/cf-key/cf-key.c b/cf-key/cf-key.c index 5d679292c2..f81c241421 100644 --- a/cf-key/cf-key.c +++ b/cf-key/cf-key.c @@ -247,7 +247,7 @@ int main(int argc, char *argv[]) static void PrintHelp() { Writer *w = FileWriter(stdout); - WriterWriteHelp(w, "cf-key", OPTIONS, HINTS, false, NULL); + WriterWriteHelp(w, "cf-key", OPTIONS, HINTS, NULL, false, false); FileWriterDetach(w); } diff --git a/cf-monitord/cf-monitord.c b/cf-monitord/cf-monitord.c index 5c16ef4849..160e86905e 100644 --- a/cf-monitord/cf-monitord.c +++ b/cf-monitord/cf-monitord.c @@ -197,7 +197,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) case 'h': { Writer *w = FileWriter(stdout); - WriterWriteHelp(w, "cf-monitord", OPTIONS, HINTS, true, NULL); + WriterWriteHelp(w, "cf-monitord", OPTIONS, HINTS, NULL, false, true); FileWriterDetach(w); } DoCleanupAndExit(EXIT_SUCCESS); @@ -233,7 +233,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) default: { Writer *w = FileWriter(stdout); - WriterWriteHelp(w, "cf-monitord", OPTIONS, HINTS, true, NULL); + WriterWriteHelp(w, "cf-monitord", OPTIONS, HINTS, NULL, false, true); FileWriterDetach(w); } DoCleanupAndExit(EXIT_FAILURE); diff --git a/cf-net/cf-net.c b/cf-net/cf-net.c index ca54c2d178..c52ec243ab 100644 --- a/cf-net/cf-net.c +++ b/cf-net/cf-net.c @@ -557,7 +557,7 @@ static int CFNetHelp(const char *topic) else { Writer *w = FileWriter(stdout); - WriterWriteHelp(w, "cf-net", OPTIONS, HINTS, false, COMMANDS); + WriterWriteHelp(w, "cf-net", OPTIONS, HINTS, COMMANDS, false, false); FileWriterDetach(w); DoCleanupAndExit(EXIT_SUCCESS); } diff --git a/cf-promises/cf-promises.c b/cf-promises/cf-promises.c index f83596a959..f234738e22 100644 --- a/cf-promises/cf-promises.c +++ b/cf-promises/cf-promises.c @@ -415,7 +415,7 @@ GenericAgentConfig *CheckOpts(int argc, char **argv) case 'h': { Writer *w = FileWriter(stdout); - WriterWriteHelp(w, "cf-promises", OPTIONS, HINTS, true, NULL); + WriterWriteHelp(w, "cf-promises", OPTIONS, HINTS, NULL, false, true); FileWriterDetach(w); } DoCleanupAndExit(EXIT_SUCCESS); @@ -483,7 +483,7 @@ GenericAgentConfig *CheckOpts(int argc, char **argv) default: { Writer *w = FileWriter(stdout); - WriterWriteHelp(w, "cf-promises", OPTIONS, HINTS, true, NULL); + WriterWriteHelp(w, "cf-promises", OPTIONS, HINTS, NULL, false, true); FileWriterDetach(w); } DoCleanupAndExit(EXIT_FAILURE); diff --git a/cf-runagent/cf-runagent.c b/cf-runagent/cf-runagent.c index 02f31260dc..6a16a77f5d 100644 --- a/cf-runagent/cf-runagent.c +++ b/cf-runagent/cf-runagent.c @@ -360,7 +360,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) case 'h': { Writer *w = FileWriter(stdout); - WriterWriteHelp(w, "cf-runagent", OPTIONS, HINTS, true, NULL); + WriterWriteHelp(w, "cf-runagent", OPTIONS, HINTS, NULL, false, true); FileWriterDetach(w); } DoCleanupAndExit(EXIT_SUCCESS); @@ -420,7 +420,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) default: { Writer *w = FileWriter(stdout); - WriterWriteHelp(w, "cf-runagent", OPTIONS, HINTS, true, NULL); + WriterWriteHelp(w, "cf-runagent", OPTIONS, HINTS, NULL, false, true); FileWriterDetach(w); } DoCleanupAndExit(EXIT_FAILURE); diff --git a/cf-serverd/cf-serverd-functions.c b/cf-serverd/cf-serverd-functions.c index 4555b91970..85092b8656 100644 --- a/cf-serverd/cf-serverd-functions.c +++ b/cf-serverd/cf-serverd-functions.c @@ -238,7 +238,7 @@ GenericAgentConfig *CheckOpts(int argc, char **argv) case 'h': { Writer *w = FileWriter(stdout); - WriterWriteHelp(w, "cf-serverd", OPTIONS, HINTS, true, NULL); + WriterWriteHelp(w, "cf-serverd", OPTIONS, HINTS, NULL, false, true); FileWriterDetach(w); } DoCleanupAndExit(EXIT_SUCCESS); @@ -288,7 +288,7 @@ GenericAgentConfig *CheckOpts(int argc, char **argv) default: { Writer *w = FileWriter(stdout); - WriterWriteHelp(w, "cf-serverd", OPTIONS, HINTS, true, NULL); + WriterWriteHelp(w, "cf-serverd", OPTIONS, HINTS, NULL, false, true); FileWriterDetach(w); } DoCleanupAndExit(EXIT_FAILURE); diff --git a/cf-testd/cf-testd.c b/cf-testd/cf-testd.c index 51986e9e94..279a3c61ab 100644 --- a/cf-testd/cf-testd.c +++ b/cf-testd/cf-testd.c @@ -144,7 +144,7 @@ void CFTestD_ConfigDestroy(CFTestD_Config *config) void CFTestD_Help() { Writer *w = FileWriter(stdout); - WriterWriteHelp(w, "cf-testd", OPTIONS, HINTS, true, NULL); + WriterWriteHelp(w, "cf-testd", OPTIONS, HINTS, NULL, false, true); FileWriterDetach(w); } diff --git a/tests/acceptance/mock_package_manager.c b/tests/acceptance/mock_package_manager.c index b41b71fefa..834eb1e43b 100644 --- a/tests/acceptance/mock_package_manager.c +++ b/tests/acceptance/mock_package_manager.c @@ -444,7 +444,7 @@ int main(int argc, char *argv[]) default: { Writer *w = FileWriter(stdout); - WriterWriteHelp(w, "mock-package-manager - pretend that you are managing packages!", OPTIONS, HINTS, false, NULL); + WriterWriteHelp(w, "mock-package-manager - pretend that you are managing packages!", OPTIONS, HINTS, NULL, false, false); FileWriterDetach(w); } exit(EXIT_FAILURE); From 97cb3c46ae8dc4bf8d220c51278c57f93989ac45 Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Fri, 22 May 2020 13:04:05 +0200 Subject: [PATCH 081/333] Added changelog for 3.15.2 --- ChangeLog | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/ChangeLog b/ChangeLog index d0489b68e6..e4bdc7290e 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,16 @@ +3.15.2: + - 'if' constraint now works in combination with class contexts + (CFE-2615) + - Fixed rare crashing bug when parsing zombie entries in ps + output. The problem was only ever observed on AIX, but could + theoretically happen on any platform depending on exact libc behavior. + (ENT-5329) + - Fixed ifelse() to return fallback in case of unresolved variables + (ENT-4653) + - Fixed service status cfengine3 on systemd managed hosts (ENT-5528) + - Improved management of secondary groups to avoid intermediary state failures + (ENT-3710) + 3.15.1: - `cf-check diagnose --test-write` can now be used to test writing into LMDB files (ENT-4484) From 223f55742ead4b65ba92caecdebacfaec6c73aa7 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 3 Jun 2020 11:54:53 +0200 Subject: [PATCH 082/333] Bumped .CFVERSION number Signed-off-by: Ole Herman Schumacher Elgesem --- .CFVERSION | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.CFVERSION b/.CFVERSION index 861845e450..b7b6bc30c0 100644 --- a/.CFVERSION +++ b/.CFVERSION @@ -1 +1 @@ -3.15.2 +3.15.3 From 51079d384dcdc1cc9f73a86257b52092df027169 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jon=20Henrik=20Bj=C3=B8rnstad?= Date: Thu, 6 Jul 2017 00:23:39 +0200 Subject: [PATCH 083/333] Moved the cf-keycrypt utility to CFEngine core This was copied from https://github.com/cfengineers-net/cf-keycrypt Moved into core by Ole Herman Schumacher Elgesem Ticket: CFE-2613 Changelog: cf-keycrypt binary for managing secrets was added Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 76058ffd0addfeb907576214cb9fbe50f4e8ddbe) --- cf-keycrypt/cf-keycrypt.c | 334 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 334 insertions(+) create mode 100644 cf-keycrypt/cf-keycrypt.c diff --git a/cf-keycrypt/cf-keycrypt.c b/cf-keycrypt/cf-keycrypt.c new file mode 100644 index 0000000000..a47a0b0caa --- /dev/null +++ b/cf-keycrypt/cf-keycrypt.c @@ -0,0 +1,334 @@ +/* + cf-keycrypt.c + + Copyright (C) 2017 cfengineers.net + + Written and maintained by Jon Henrik Bjornstad + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +*/ + +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +/* +#ifdef LMDB +#include +#endif +*/ + +#define STDIN 0 +#define STDOUT 1 + +#define BUFSIZE 1024 + +void usage() { + printf("Workdir is defined as %s\n", WORKDIR); + printf( +"\n" +"Usage: cf-keycrypt [-e public-key|-d private-key|-H hostname-or-ip] -o outfile -i infile [-h]\n" +"\n" +"Use CFEngine cryptographic keys to encrypt and decrypt files, eg. files containing\n" +"passwords or certificates.\n" +"\n" +" -e Encrypt with key.\n" +" -d Decrypt with key.\n" +" -i File to encrypt/decrypt. '-' reads from stdin.\n" +" -o File to write encrypted/decrypted contents to. '-' writes to stdout.\n" +" -H Hostname/IP to encrypt for, gets data from lastseen database\n" +" -h Print help.\n" +"\n" +"Examples:\n" +" Encrypt:\n" +" cf-keycrypt -e /var/cfengine/ppkeys/localhost.pub -i myplain.dat -o mycrypt.dat\n" +"\n" +" Decrypt:\n" +" cf-keycrypt -d /var/cfengine/ppkeys/localhost.priv -i mycrypt.dat -o myplain.dat\n" +"\n" +"Written and maintained by Jon Henrik Bjornstad \n" +"\n" +"Copyright (C) cfengineers.net\n" +"\n" +); +} + +void *get_in_addr(struct sockaddr *sa) { + if (sa->sa_family == AF_INET) + return &(((struct sockaddr_in*)sa)->sin_addr); + return &(((struct sockaddr_in6*)sa)->sin6_addr); +} + +int file_exist(char *filename) { + struct stat buffer; + return (stat (filename, &buffer) == 0); +} + +char *get_host_pubkey(char *host) { + char *key; + char value[BUFSIZE]; + char *buffer = (char *) malloc(BUFSIZE *sizeof(char)); + //char *keyname = NULL; + char hash[CF_HOSTKEY_STRING_SIZE]; + //char buffer[BUFSIZE]; + char ipaddress[BUFSIZE]; + int ecode; + struct addrinfo *result; + struct addrinfo *res; + bool found; + + int error = getaddrinfo(host,NULL,NULL,&result); + + for(res = result; res != NULL && !found; res = res->ai_next) { + inet_ntop(res->ai_family, get_in_addr((struct sockaddr *)res->ai_addr), ipaddress, sizeof(ipaddress)); + if((strcmp(ipaddress, "127.0.0.1") == 0) || (strcmp(ipaddress, "::1") == 0)){ + found = true; + snprintf(buffer, BUFSIZE * sizeof(char), "%s/ppkeys/localhost.pub", WORKDIR); + return buffer; + } + found = Address2Hostkey(hash, sizeof(hash), ipaddress); + } + if(found) { + snprintf(buffer, BUFSIZE * sizeof(char), "%s/ppkeys/root-%s.pub", WORKDIR, hash); + return buffer; + }else{ + for(res = result; res != NULL; res = res->ai_next) { + inet_ntop(res->ai_family, get_in_addr((struct sockaddr *)res->ai_addr), ipaddress, sizeof(ipaddress)); + snprintf(buffer, BUFSIZE * sizeof(char), "%s/ppkeys/root-%s.pub", WORKDIR, ipaddress); + if(file_exist(buffer)){ + return buffer; + } + } + } + return NULL; +} + +void *readseckey(char *secfile) { + FILE *fp=NULL; + RSA* PRIVKEY=NULL; + static char *passphrase = "Cfengine passphrase"; + unsigned long err; + + if ((fp = fopen(secfile,"r")) == NULL) { + printf("Couldn't find a private key - use cf-key to get one"); + return (void *)NULL; + } + + if ((PRIVKEY = PEM_read_RSAPrivateKey(fp,(RSA **)NULL,NULL,passphrase)) == NULL) { + err = ERR_get_error(); + printf("PEM_readError reading Private Key = %s\n", ERR_reason_error_string(err)); + PRIVKEY = NULL; + fclose(fp); + return (void *)NULL; + } else { + return (void *)PRIVKEY; + } +} + +void *readpubkey(char *pubfile) { + FILE *fp; + RSA *key=NULL; + static char *passphrase = "Cfengine passphrase"; + + if((fp = fopen(pubfile, "r")) == NULL) { + fprintf(stderr,"Error: Cannot locate Public Key file '%s'.\n", pubfile); + return NULL; + } + if((key = PEM_read_RSAPublicKey(fp,(RSA **)NULL,NULL,passphrase)) == NULL) { + fprintf(stderr,"Error: failed reading Public Key in '%s' file.\n", pubfile); + return NULL; + } + fclose(fp); + return (void *)key; +} + +long int rsa_encrypt(char *pubfile, char *filein, char *fileout) { + int ks=0; + unsigned long size=0, len=0, ciphsz=0; + RSA* key=NULL; + FILE *infile=NULL; + FILE *outfile = NULL; + char *tmpciph=NULL, *tmpplain=NULL; + + key = (RSA *)readpubkey(pubfile); + if (!key) { + return -1; + } + + if((strcmp("-",filein) == 0)) { + infile = stdin; + }else if(!(infile = fopen(filein, "r"))) { + fprintf(stderr, "Error: Cannot locate input file.\n"); + return -1; + } + + if((strcmp("-",fileout) == 0)) { + outfile = stdout; + //printf("Write to stdout\n"); + } else if(!(outfile = fopen(fileout, "w"))){ + fprintf(stderr, "Error: Cannot create output file.\n"); + return -1; + } + + ks = RSA_size(key); + tmpplain = (unsigned char *)malloc(ks * sizeof(unsigned char)); + tmpciph = (unsigned char *)malloc(ks * sizeof(unsigned char)); + srand(time(NULL)); + + while(!feof(infile)) { + memset(tmpplain, '\0', ks); + memset(tmpciph, '\0', ks); + len = fread(tmpplain, 1, ks-RSA_PKCS1_PADDING_SIZE, infile); + if((size = RSA_public_encrypt(strlen(tmpplain), tmpplain, tmpciph, key, RSA_PKCS1_PADDING)) == -1) { + fprintf(stderr, "%s\n", ERR_error_string(ERR_get_error(), NULL)); + return -1; + } + fwrite(tmpciph,sizeof(unsigned char),ks,outfile); + } + fclose(infile); + fclose(outfile); + free(tmpciph); + free(tmpplain); + RSA_free(key); + return ciphsz; +} + +long int rsa_decrypt(char *secfile, char *cryptfile, char *plainfile) { + unsigned long int plsz=0, size=0, ks=0, len=0; + RSA* key=NULL; + char *tmpplain=NULL, *tmpciph=NULL; + + FILE *infile=NULL; + FILE *outfile = NULL; + + key = (RSA *)readseckey(secfile); + if (!key) { + return -1; + } + + if((strcmp("-",cryptfile) == 0)) { + infile = stdin; + }else if(!(infile = fopen(cryptfile, "r"))) { + fprintf(stderr, "Error: Cannot locate input file.\n"); + return -1; + } + + if((strcmp("-",plainfile) == 0)) { + outfile = stdout; + //printf("Write to stdout\n"); + }else if(!(outfile = fopen(plainfile, "w"))){ + fprintf(stderr, "Error: Cannot create output file.\n"); + return -1; + } + + ks = RSA_size(key); + tmpciph = (unsigned char *)malloc(ks * sizeof(unsigned char)); + tmpplain = (unsigned char *)malloc(ks * sizeof(unsigned char)); + //printf("Keysize: %d\n", ks); + while(!feof(infile)) { + memset(tmpciph, '\0', ks); + memset(tmpplain, '\0', ks); + len = fread(tmpciph, 1, ks, infile); + if(len > 0){ + if((size = RSA_private_decrypt(ks, tmpciph, tmpplain, key, RSA_PKCS1_PADDING)) == -1) { + fprintf(stderr, "%s\n", ERR_error_string(ERR_get_error(), NULL)); + return -1; + } + } + fwrite(tmpplain,1,strlen(tmpplain),outfile); + } + fclose(infile); + fclose(outfile); + free(tmpplain); + free(tmpciph); + RSA_free(key); + return plsz; +} + +int main(int argc, char *argv[]) { + OpenSSL_add_all_algorithms(); + ERR_load_crypto_strings(); + + opterr = 0; + char *key = NULL; + char *infile = NULL; + char *outfile = NULL; + char *host = NULL; + int encrypt = 0; + int decrypt = 0; + int c = 0; + int size = 0; + +// strcpy(WORKDIR,CFCR_WORKDIR); + + while ((c = getopt (argc, argv, "he:d:i:o:H:")) != -1) + switch (c) + { + case 'e': + encrypt = 1; + key = optarg; + break; + case 'd': + decrypt = 1; + key = optarg; + break; + case 'i': + infile = optarg; + break; + case 'o': + outfile = optarg; + break; + case 'H': + encrypt = 1; + host = optarg; + break; + case 'h': + usage(); + exit(1); + default: + printf("ERROR: Unknown option '-%c'\n", optopt); + usage(); + exit(1); + } + + if(host){ + key = get_host_pubkey(host); + if(!key){ + fprintf(stderr,"ERROR: Unable to locate public key for host %s\n",host); + exit(1); + } + } + + if(encrypt > 0 && key && infile && outfile){ + size = rsa_encrypt(key, infile, outfile); + if(size < 0) + exit(1); + }else if(decrypt > 0 && key && infile && outfile){ + size = rsa_decrypt(key, infile, outfile); + if(size < 0) + exit(1); + }else{ + usage(); + exit(1); + } + exit(0); +} From 4ee03f8b8ac9f5fec693e06bb8c326cf90498861 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Thu, 6 Jul 2017 01:21:21 +0200 Subject: [PATCH 084/333] Added sys.cf-keycrypt component variable Also did a bunch of clean up Squashed commits: cf-keycrypt: Changed to spaces for indentation cf-keycrypt: Added Makefile cf-keycrypt: Added sys.cf-keycrypt variable(component) cf-keycrypt: Added automake subfolder cf-keycrypt: Added binaries to .gitignore cf-keycrypt: Added Makefile to configure.ac cf-keycrypt: Coding style cf-keycrypt: Fixed unused and uninitialized variables Changelog: Title (cherry picked from commit 97c3d744597574da9f5b956927f28f866ce17888) --- .gitignore | 2 + Makefile.am | 1 + cf-keycrypt/Makefile.am | 51 ++++ cf-keycrypt/cf-keycrypt.c | 600 +++++++++++++++++++++----------------- configure.ac | 1 + libenv/sysinfo.c | 2 +- 6 files changed, 383 insertions(+), 274 deletions(-) create mode 100644 cf-keycrypt/Makefile.am diff --git a/.gitignore b/.gitignore index 312dadb126..6baaf6bb33 100644 --- a/.gitignore +++ b/.gitignore @@ -91,6 +91,8 @@ stamp-h1 /cf-runagent/cf-runagent.exe /cf-net/cf-net /cf-net/cf-net.exe +/cf-keycrypt/cf-keycrypt +/cf-keycrypt/cf-keycrypt.exe /cf-serverd/cf-serverd /cf-serverd/cf-serverd.exe /cf-testd/cf-testd diff --git a/Makefile.am b/Makefile.am index 988d1f1d5c..60c7e329c8 100644 --- a/Makefile.am +++ b/Makefile.am @@ -43,6 +43,7 @@ SUBDIRS = \ cf-testd \ cf-upgrade \ cf-net \ + cf-keycrypt \ misc \ ext \ examples \ diff --git a/cf-keycrypt/Makefile.am b/cf-keycrypt/Makefile.am new file mode 100644 index 0000000000..937d0f6220 --- /dev/null +++ b/cf-keycrypt/Makefile.am @@ -0,0 +1,51 @@ +# +# Copyright (C) CFEngine AS +# +# This file is part of CFEngine 3 - written and maintained by CFEngine AS. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation; version 3. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA +# +# To the extent this program is licensed as part of the Enterprise +# versions of CFEngine, the applicable Commercial Open Source License +# (COSL) may apply to this file if you as a licensee so wish it. See +# included file COSL.txt. +# +noinst_LTLIBRARIES = libcf-keycrypt.la + +AM_CPPFLAGS = -I$(srcdir)/../libpromises -I$(srcdir)/../libutils \ + -I$(srcdir)/../libcfnet \ + $(OPENSSL_CPPFLAGS) \ + $(PCRE_CPPFLAGS) \ + $(ENTERPRISE_CPPFLAGS) + +AM_CFLAGS = @CFLAGS@ \ + $(OPENSSL_CFLAGS) \ + $(ENTERPRISE_CFLAGS) + +libcf_keycrypt_la_LIBADD = ../libpromises/libpromises.la + +libcf_keycrypt_la_SOURCES = cf-keycrypt.c + +if !BUILTIN_EXTENSIONS + bin_PROGRAMS = cf-keycrypt + cf_keycrypt_LDADD = libcf-keycrypt.la + cf_keycrypt_SOURCES = +endif + +CLEANFILES = *.gcno *.gcda + +# +# Some basic clean ups +# +MOSTLYCLEANFILES = *~ *.orig *.rej diff --git a/cf-keycrypt/cf-keycrypt.c b/cf-keycrypt/cf-keycrypt.c index a47a0b0caa..4c4894a372 100644 --- a/cf-keycrypt/cf-keycrypt.c +++ b/cf-keycrypt/cf-keycrypt.c @@ -42,293 +42,347 @@ #define BUFSIZE 1024 -void usage() { - printf("Workdir is defined as %s\n", WORKDIR); - printf( -"\n" -"Usage: cf-keycrypt [-e public-key|-d private-key|-H hostname-or-ip] -o outfile -i infile [-h]\n" -"\n" -"Use CFEngine cryptographic keys to encrypt and decrypt files, eg. files containing\n" -"passwords or certificates.\n" -"\n" -" -e Encrypt with key.\n" -" -d Decrypt with key.\n" -" -i File to encrypt/decrypt. '-' reads from stdin.\n" -" -o File to write encrypted/decrypted contents to. '-' writes to stdout.\n" -" -H Hostname/IP to encrypt for, gets data from lastseen database\n" -" -h Print help.\n" -"\n" -"Examples:\n" -" Encrypt:\n" -" cf-keycrypt -e /var/cfengine/ppkeys/localhost.pub -i myplain.dat -o mycrypt.dat\n" -"\n" -" Decrypt:\n" -" cf-keycrypt -d /var/cfengine/ppkeys/localhost.priv -i mycrypt.dat -o myplain.dat\n" -"\n" -"Written and maintained by Jon Henrik Bjornstad \n" -"\n" -"Copyright (C) cfengineers.net\n" -"\n" -); +void usage() +{ + printf("Workdir is defined as %s\n", WORKDIR); + printf( + "\n" + "Usage: cf-keycrypt [-e public-key|-d private-key|-H hostname-or-ip] -o outfile -i infile [-h]\n" + "\n" + "Use CFEngine cryptographic keys to encrypt and decrypt files, eg. files containing\n" + "passwords or certificates.\n" + "\n" + " -e Encrypt with key.\n" + " -d Decrypt with key.\n" + " -i File to encrypt/decrypt. '-' reads from stdin.\n" + " -o File to write encrypted/decrypted contents to. '-' writes to stdout.\n" + " -H Hostname/IP to encrypt for, gets data from lastseen database\n" + " -h Print help.\n" + "\n" + "Examples:\n" + " Encrypt:\n" + " cf-keycrypt -e /var/cfengine/ppkeys/localhost.pub -i myplain.dat -o mycrypt.dat\n" + "\n" + " Decrypt:\n" + " cf-keycrypt -d /var/cfengine/ppkeys/localhost.priv -i mycrypt.dat -o myplain.dat\n" + "\n" + "Written and maintained by Jon Henrik Bjornstad \n" + "\n" + "Copyright (C) cfengineers.net\n" + "\n" + ); } -void *get_in_addr(struct sockaddr *sa) { - if (sa->sa_family == AF_INET) - return &(((struct sockaddr_in*)sa)->sin_addr); - return &(((struct sockaddr_in6*)sa)->sin6_addr); +void *get_in_addr(struct sockaddr *sa) +{ + if (sa->sa_family == AF_INET) + { + return &(((struct sockaddr_in*)sa)->sin_addr); + } + return &(((struct sockaddr_in6*)sa)->sin6_addr); } -int file_exist(char *filename) { - struct stat buffer; - return (stat (filename, &buffer) == 0); +int file_exist(char *filename) +{ + struct stat buffer; + return (stat (filename, &buffer) == 0); } -char *get_host_pubkey(char *host) { - char *key; - char value[BUFSIZE]; - char *buffer = (char *) malloc(BUFSIZE *sizeof(char)); - //char *keyname = NULL; - char hash[CF_HOSTKEY_STRING_SIZE]; - //char buffer[BUFSIZE]; - char ipaddress[BUFSIZE]; - int ecode; - struct addrinfo *result; - struct addrinfo *res; - bool found; - - int error = getaddrinfo(host,NULL,NULL,&result); - - for(res = result; res != NULL && !found; res = res->ai_next) { - inet_ntop(res->ai_family, get_in_addr((struct sockaddr *)res->ai_addr), ipaddress, sizeof(ipaddress)); - if((strcmp(ipaddress, "127.0.0.1") == 0) || (strcmp(ipaddress, "::1") == 0)){ - found = true; - snprintf(buffer, BUFSIZE * sizeof(char), "%s/ppkeys/localhost.pub", WORKDIR); - return buffer; - } - found = Address2Hostkey(hash, sizeof(hash), ipaddress); - } - if(found) { - snprintf(buffer, BUFSIZE * sizeof(char), "%s/ppkeys/root-%s.pub", WORKDIR, hash); - return buffer; - }else{ - for(res = result; res != NULL; res = res->ai_next) { - inet_ntop(res->ai_family, get_in_addr((struct sockaddr *)res->ai_addr), ipaddress, sizeof(ipaddress)); - snprintf(buffer, BUFSIZE * sizeof(char), "%s/ppkeys/root-%s.pub", WORKDIR, ipaddress); - if(file_exist(buffer)){ - return buffer; - } - } - } - return NULL; +char *get_host_pubkey(char *host) +{ + // char *key; // TODO: this was unused(!) + // char value[BUFSIZE]; // TODO: this was unused(!) + char *buffer = (char *) malloc(BUFSIZE *sizeof(char)); + //char *keyname = NULL; + char hash[CF_HOSTKEY_STRING_SIZE]; + //char buffer[BUFSIZE]; + char ipaddress[BUFSIZE]; + // int ecode; // TODO: this was unused(!) + struct addrinfo *result; + struct addrinfo *res; + bool found = false; + + int error = getaddrinfo(host, NULL, NULL, &result); + if (error != 0) + { + Log(LOG_LEVEL_ERR, "Failed to get IP from host (getaddrinfo: %s)", + gai_strerror(error)); + return NULL; + } + + for(res = result; res != NULL && !found; res = res->ai_next) + { + inet_ntop(res->ai_family, get_in_addr((struct sockaddr *)res->ai_addr), ipaddress, sizeof(ipaddress)); + if ((strcmp(ipaddress, "127.0.0.1") == 0) || (strcmp(ipaddress, "::1") == 0)) + { + found = true; + snprintf(buffer, BUFSIZE * sizeof(char), "%s/ppkeys/localhost.pub", WORKDIR); + return buffer; + } + found = Address2Hostkey(hash, sizeof(hash), ipaddress); + } + if (found) + { + snprintf(buffer, BUFSIZE * sizeof(char), "%s/ppkeys/root-%s.pub", WORKDIR, hash); + return buffer; + } + else + { + for(res = result; res != NULL; res = res->ai_next) + { + inet_ntop(res->ai_family, get_in_addr((struct sockaddr *)res->ai_addr), ipaddress, sizeof(ipaddress)); + snprintf(buffer, BUFSIZE * sizeof(char), "%s/ppkeys/root-%s.pub", WORKDIR, ipaddress); + if (file_exist(buffer)) + { + return buffer; + } + } + } + return NULL; } -void *readseckey(char *secfile) { - FILE *fp=NULL; - RSA* PRIVKEY=NULL; - static char *passphrase = "Cfengine passphrase"; - unsigned long err; - - if ((fp = fopen(secfile,"r")) == NULL) { - printf("Couldn't find a private key - use cf-key to get one"); - return (void *)NULL; - } - - if ((PRIVKEY = PEM_read_RSAPrivateKey(fp,(RSA **)NULL,NULL,passphrase)) == NULL) { - err = ERR_get_error(); - printf("PEM_readError reading Private Key = %s\n", ERR_reason_error_string(err)); - PRIVKEY = NULL; - fclose(fp); - return (void *)NULL; - } else { - return (void *)PRIVKEY; - } +void *readseckey(char *secfile) +{ + FILE *fp = NULL; + RSA* PRIVKEY = NULL; + static char *passphrase = "Cfengine passphrase"; + unsigned long err; + + if ((fp = fopen(secfile,"r")) == NULL) + { + printf("Couldn't find a private key - use cf-key to get one"); + return (void *)NULL; + } + + if ((PRIVKEY = PEM_read_RSAPrivateKey(fp,(RSA **)NULL,NULL,passphrase)) == NULL) + { + err = ERR_get_error(); + printf("PEM_readError reading Private Key = %s\n", ERR_reason_error_string(err)); + PRIVKEY = NULL; + fclose(fp); + return (void *)NULL; + } + else + { + return (void *)PRIVKEY; + } } -void *readpubkey(char *pubfile) { - FILE *fp; - RSA *key=NULL; - static char *passphrase = "Cfengine passphrase"; - - if((fp = fopen(pubfile, "r")) == NULL) { - fprintf(stderr,"Error: Cannot locate Public Key file '%s'.\n", pubfile); - return NULL; - } - if((key = PEM_read_RSAPublicKey(fp,(RSA **)NULL,NULL,passphrase)) == NULL) { - fprintf(stderr,"Error: failed reading Public Key in '%s' file.\n", pubfile); - return NULL; - } - fclose(fp); - return (void *)key; +void *readpubkey(char *pubfile) +{ + FILE *fp; + RSA *key = NULL; + static char *passphrase = "Cfengine passphrase"; + + if ((fp = fopen(pubfile, "r")) == NULL) + { + fprintf(stderr,"Error: Cannot locate Public Key file '%s'.\n", pubfile); + return NULL; + } + if ((key = PEM_read_RSAPublicKey(fp,(RSA **)NULL,NULL,passphrase)) == NULL) + { + fprintf(stderr,"Error: failed reading Public Key in '%s' file.\n", pubfile); + return NULL; + } + fclose(fp); + return (void *)key; } -long int rsa_encrypt(char *pubfile, char *filein, char *fileout) { - int ks=0; - unsigned long size=0, len=0, ciphsz=0; - RSA* key=NULL; - FILE *infile=NULL; - FILE *outfile = NULL; - char *tmpciph=NULL, *tmpplain=NULL; - - key = (RSA *)readpubkey(pubfile); - if (!key) { - return -1; - } - - if((strcmp("-",filein) == 0)) { - infile = stdin; - }else if(!(infile = fopen(filein, "r"))) { - fprintf(stderr, "Error: Cannot locate input file.\n"); - return -1; - } - - if((strcmp("-",fileout) == 0)) { - outfile = stdout; - //printf("Write to stdout\n"); - } else if(!(outfile = fopen(fileout, "w"))){ - fprintf(stderr, "Error: Cannot create output file.\n"); - return -1; - } - - ks = RSA_size(key); - tmpplain = (unsigned char *)malloc(ks * sizeof(unsigned char)); - tmpciph = (unsigned char *)malloc(ks * sizeof(unsigned char)); - srand(time(NULL)); - - while(!feof(infile)) { - memset(tmpplain, '\0', ks); - memset(tmpciph, '\0', ks); - len = fread(tmpplain, 1, ks-RSA_PKCS1_PADDING_SIZE, infile); - if((size = RSA_public_encrypt(strlen(tmpplain), tmpplain, tmpciph, key, RSA_PKCS1_PADDING)) == -1) { - fprintf(stderr, "%s\n", ERR_error_string(ERR_get_error(), NULL)); - return -1; - } - fwrite(tmpciph,sizeof(unsigned char),ks,outfile); - } - fclose(infile); - fclose(outfile); - free(tmpciph); - free(tmpplain); - RSA_free(key); - return ciphsz; +long int rsa_encrypt(char *pubfile, char *filein, char *fileout) +{ + int ks = 0; + unsigned long size = 0, len = 0, ciphsz = 0; + RSA* key = NULL; + FILE *infile = NULL; + FILE *outfile = NULL; + char *tmpciph = NULL, *tmpplain = NULL; + + key = (RSA *)readpubkey(pubfile); + if (!key) + { + return -1; + } + + if ((strcmp("-",filein) == 0)) + { + infile = stdin; + } + else if (!(infile = fopen(filein, "r"))) + { + fprintf(stderr, "Error: Cannot locate input file.\n"); + return -1; + } + + if ((strcmp("-",fileout) == 0)) + { + outfile = stdout; + //printf("Write to stdout\n"); + } + else if (!(outfile = fopen(fileout, "w"))) + { + fprintf(stderr, "Error: Cannot create output file.\n"); + return -1; + } + + ks = RSA_size(key); + tmpplain = (unsigned char *)malloc(ks * sizeof(unsigned char)); + tmpciph = (unsigned char *)malloc(ks * sizeof(unsigned char)); + srand(time(NULL)); + + while(!feof(infile)) + { + memset(tmpplain, '\0', ks); + memset(tmpciph, '\0', ks); + len = fread(tmpplain, 1, ks-RSA_PKCS1_PADDING_SIZE, infile); + if (len <= 0) + { + Log(LOG_LEVEL_ERR, "Could not read file '%s'(fread: %lu)", + filein, len); + } + if ((size = RSA_public_encrypt(strlen(tmpplain), tmpplain, tmpciph, key, RSA_PKCS1_PADDING)) == -1) + { + fprintf(stderr, "%s\n", ERR_error_string(ERR_get_error(), NULL)); + return -1; + } + fwrite(tmpciph,sizeof(unsigned char),ks,outfile); + } + fclose(infile); + fclose(outfile); + free(tmpciph); + free(tmpplain); + RSA_free(key); + return ciphsz; } -long int rsa_decrypt(char *secfile, char *cryptfile, char *plainfile) { - unsigned long int plsz=0, size=0, ks=0, len=0; - RSA* key=NULL; - char *tmpplain=NULL, *tmpciph=NULL; - - FILE *infile=NULL; - FILE *outfile = NULL; - - key = (RSA *)readseckey(secfile); - if (!key) { - return -1; - } - - if((strcmp("-",cryptfile) == 0)) { - infile = stdin; - }else if(!(infile = fopen(cryptfile, "r"))) { - fprintf(stderr, "Error: Cannot locate input file.\n"); - return -1; - } - - if((strcmp("-",plainfile) == 0)) { - outfile = stdout; - //printf("Write to stdout\n"); - }else if(!(outfile = fopen(plainfile, "w"))){ - fprintf(stderr, "Error: Cannot create output file.\n"); - return -1; - } - - ks = RSA_size(key); - tmpciph = (unsigned char *)malloc(ks * sizeof(unsigned char)); - tmpplain = (unsigned char *)malloc(ks * sizeof(unsigned char)); - //printf("Keysize: %d\n", ks); - while(!feof(infile)) { - memset(tmpciph, '\0', ks); - memset(tmpplain, '\0', ks); - len = fread(tmpciph, 1, ks, infile); - if(len > 0){ - if((size = RSA_private_decrypt(ks, tmpciph, tmpplain, key, RSA_PKCS1_PADDING)) == -1) { - fprintf(stderr, "%s\n", ERR_error_string(ERR_get_error(), NULL)); - return -1; - } - } - fwrite(tmpplain,1,strlen(tmpplain),outfile); - } - fclose(infile); - fclose(outfile); - free(tmpplain); - free(tmpciph); - RSA_free(key); - return plsz; +long int rsa_decrypt(char *secfile, char *cryptfile, char *plainfile) +{ + unsigned long int plsz = 0, size = 0, ks = 0, len = 0; // TODO: Unused: len = 0; + RSA *key = NULL; + char *tmpplain = NULL, *tmpciph = NULL; + + FILE *infile = NULL; + FILE *outfile = NULL; + + key = (RSA *)readseckey(secfile); + if (!key) { + return -1; + } + + if ((strcmp("-",cryptfile) == 0)) { + infile = stdin; + }else if (!(infile = fopen(cryptfile, "r"))) { + fprintf(stderr, "Error: Cannot locate input file.\n"); + return -1; + } + + if ((strcmp("-",plainfile) == 0)) { + outfile = stdout; + //printf("Write to stdout\n"); + }else if (!(outfile = fopen(plainfile, "w"))){ + fprintf(stderr, "Error: Cannot create output file.\n"); + return -1; + } + + ks = RSA_size(key); + tmpciph = (unsigned char *)malloc(ks * sizeof(unsigned char)); + tmpplain = (unsigned char *)malloc(ks * sizeof(unsigned char)); + //printf("Keysize: %d\n", ks); + while(!feof(infile)) + { + memset(tmpciph, '\0', ks); + memset(tmpplain, '\0', ks); + len = fread(tmpciph, 1, ks, infile); + if (len > 0){ + if ((size = RSA_private_decrypt(ks, tmpciph, tmpplain, key, RSA_PKCS1_PADDING)) == -1) + { + fprintf(stderr, "%s\n", ERR_error_string(ERR_get_error(), NULL)); + return -1; + } + } + fwrite(tmpplain,1,strlen(tmpplain),outfile); + } + fclose(infile); + fclose(outfile); + free(tmpplain); + free(tmpciph); + RSA_free(key); + return plsz; } -int main(int argc, char *argv[]) { - OpenSSL_add_all_algorithms(); - ERR_load_crypto_strings(); - - opterr = 0; - char *key = NULL; - char *infile = NULL; - char *outfile = NULL; - char *host = NULL; - int encrypt = 0; - int decrypt = 0; - int c = 0; - int size = 0; - -// strcpy(WORKDIR,CFCR_WORKDIR); - - while ((c = getopt (argc, argv, "he:d:i:o:H:")) != -1) - switch (c) - { - case 'e': - encrypt = 1; - key = optarg; - break; - case 'd': - decrypt = 1; - key = optarg; - break; - case 'i': - infile = optarg; - break; - case 'o': - outfile = optarg; - break; - case 'H': - encrypt = 1; - host = optarg; - break; - case 'h': - usage(); - exit(1); - default: - printf("ERROR: Unknown option '-%c'\n", optopt); - usage(); - exit(1); - } - - if(host){ - key = get_host_pubkey(host); - if(!key){ - fprintf(stderr,"ERROR: Unable to locate public key for host %s\n",host); - exit(1); - } - } - - if(encrypt > 0 && key && infile && outfile){ - size = rsa_encrypt(key, infile, outfile); - if(size < 0) - exit(1); - }else if(decrypt > 0 && key && infile && outfile){ - size = rsa_decrypt(key, infile, outfile); - if(size < 0) - exit(1); - }else{ - usage(); - exit(1); - } - exit(0); +int main(int argc, char *argv[]) +{ + OpenSSL_add_all_algorithms(); + ERR_load_crypto_strings(); + + opterr = 0; + char *key = NULL; + char *infile = NULL; + char *outfile = NULL; + char *host = NULL; + int encrypt = 0; + int decrypt = 0; + int c = 0; + int size = 0; + + //strcpy(WORKDIR,CFCR_WORKDIR); + + while ((c = getopt (argc, argv, "he:d:i:o:H:")) != -1) + { + switch (c) + { + case 'e': + encrypt = 1; + key = optarg; + break; + case 'd': + decrypt = 1; + key = optarg; + break; + case 'i': + infile = optarg; + break; + case 'o': + outfile = optarg; + break; + case 'H': + encrypt = 1; + host = optarg; + break; + case 'h': + usage(); + exit(1); + default: + printf("ERROR: Unknown option '-%c'\n", optopt); + usage(); + exit(1); + } + } + if (host) + { + key = get_host_pubkey(host); + if (!key) + { + fprintf(stderr,"ERROR: Unable to locate public key for host %s\n", host); + exit(1); + } + } + + if (encrypt > 0 && key && infile && outfile) + { + size = rsa_encrypt(key, infile, outfile); + if (size < 0) + exit(1); + } + else if (decrypt > 0 && key && infile && outfile) + { + size = rsa_decrypt(key, infile, outfile); + if (size < 0) + exit(1); + } + else + { + usage(); + exit(1); + } + exit(0); } diff --git a/configure.ac b/configure.ac index 2d89faf84e..06b7438402 100644 --- a/configure.ac +++ b/configure.ac @@ -1772,6 +1772,7 @@ AC_CONFIG_FILES([Makefile cf-serverd/Makefile cf-testd/Makefile cf-net/Makefile + cf-keycrypt/Makefile config.post.h misc/Makefile misc/selinux/Makefile diff --git a/libenv/sysinfo.c b/libenv/sysinfo.c index d2446aa92a..fcdad92e83 100644 --- a/libenv/sysinfo.c +++ b/libenv/sysinfo.c @@ -435,7 +435,7 @@ static void GetNameInfo3(EvalContext *ctx) // This is used for $(sys.cf_agent), $(sys.cf_serverd) ... : char *components[COMPONENTS_SIZE] = { "cf-twin", "cf-agent", "cf-serverd", "cf-monitord", "cf-know", "cf-report", "cf-key", "cf-runagent", "cf-execd", "cf-hub", - "cf-promises", "cf-upgrade", "cf-net", "cf-check", + "cf-promises", "cf-upgrade", "cf-net", "cf-check", "cf-keycrypt", NULL }; int have_component[COMPONENTS_SIZE]; From 8b21cf32d247a43d81a4d7d6d23a0abb112a5927 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Thu, 6 Jul 2017 12:14:37 +0200 Subject: [PATCH 085/333] Acceptance tests for cf-keycrypt Squashed commits: cf-keycrypt: Fixes to acceptance test cf-keycrypt: Added to testall cf-keycrypt: Added keys and expected output cf-keycrypt: add seperate encryption and decryption tests Fixup encryption and decryption tests Really fix the encryption and decryption tests Expect encryption does not produce the same output cf-keycrypt: Add Example usage Fix example All the example pllicy blocks must be within the being and end src tags. Convert to static example (cherry picked from commit 6ab4cc09006e636dae510e1654724e5383a2146b) --- examples/cf-keycrypt.cf | 65 +++++++++++++++++++ examples/cf-keycrypt.cf.cfcrypt | 2 + examples/cf-keycrypt.cf.priv | 30 +++++++++ examples/cf-keycrypt.cf.pub | 8 +++ .../00_basics/cf-keycrypt/decrypt.cf | 27 ++++++++ .../00_basics/cf-keycrypt/encrypt-decrypt.cf | 53 +++++++++++++++ .../00_basics/cf-keycrypt/encrypt.cf.x | 29 +++++++++ .../00_basics/cf-keycrypt/encrypted | 2 + .../00_basics/cf-keycrypt/plaintext | 1 + .../00_basics/cf-keycrypt/testkey.priv | 30 +++++++++ .../00_basics/cf-keycrypt/testkey.pub | 8 +++ tests/acceptance/testall | 15 ++++- 12 files changed, 267 insertions(+), 3 deletions(-) create mode 100644 examples/cf-keycrypt.cf create mode 100644 examples/cf-keycrypt.cf.cfcrypt create mode 100644 examples/cf-keycrypt.cf.priv create mode 100644 examples/cf-keycrypt.cf.pub create mode 100644 tests/acceptance/00_basics/cf-keycrypt/decrypt.cf create mode 100644 tests/acceptance/00_basics/cf-keycrypt/encrypt-decrypt.cf create mode 100644 tests/acceptance/00_basics/cf-keycrypt/encrypt.cf.x create mode 100644 tests/acceptance/00_basics/cf-keycrypt/encrypted create mode 100644 tests/acceptance/00_basics/cf-keycrypt/plaintext create mode 100644 tests/acceptance/00_basics/cf-keycrypt/testkey.priv create mode 100644 tests/acceptance/00_basics/cf-keycrypt/testkey.pub diff --git a/examples/cf-keycrypt.cf b/examples/cf-keycrypt.cf new file mode 100644 index 0000000000..b8b17a3b40 --- /dev/null +++ b/examples/cf-keycrypt.cf @@ -0,0 +1,65 @@ +# Copyright (C) Cfengine AS + +# This file is part of Cfengine 3 - written and maintained by Cfengine AS. + +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation; version 3. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA + +# To the extent this program is licensed as part of the Enterprise +# versions of Cfengine, the applicable Commercial Open Source License +# (COSL) may apply to this file if you as a licensee so wish it. See +# included file COSL.txt. + +############################################################################### +#+begin_src cfengine3 +bundle agent main +{ + vars: + + "private_key" + comment => "The decryption key", + string => "$(this.promise_filename).priv"; + + "encrypted_file" string => "$(this.promise_filename).cfcrypt"; + + "secret" + comment => "We decrypt the encrypted file directly into a variable.", + string => execresult("$(sys.cf_keycrypt) -d $(private_key) -i $(encrypted_file) -o -", noshell); + + reports: + "Encrypted file content:" + printfile => cat( $(encrypted_file) ); + + "Decrypted content:$(const.n)$(secret)"; +} + +body printfile cat(file) +# @brief Report the contents of a file +# @param file The full path of the file to report +{ + file_to_print => "$(file)"; + number_of_lines => "inf"; +} +#+end_src + +############################################################################### +#+begin_src static_example_output +#@ ``` +#@ R: Encrypted file content: +#@ R: ��Q�������VO�["N߀ �1�}΢n�#��h�J�2�}5���N1m�n�,��E��\����٣��g>��oqVÛ��P�-{.�G��w$ +#@ R: Decrypted content: +#@ Super secret message is here +#@ ``` +#+end_src diff --git a/examples/cf-keycrypt.cf.cfcrypt b/examples/cf-keycrypt.cf.cfcrypt new file mode 100644 index 0000000000..6f1ab1ff18 --- /dev/null +++ b/examples/cf-keycrypt.cf.cfcrypt @@ -0,0 +1,2 @@ +®Q–ßíéîù’VO["N߀ 1ü}΢n„#©ìh‡J¸2Ê}5ƒ›ôN1múnµ,­áE=§Ð\Õó ‡Ù£­ƒg>ôèoqVÛ¸PÐ-{.ÿG‹w$ \ No newline at end of file diff --git a/examples/cf-keycrypt.cf.priv b/examples/cf-keycrypt.cf.priv new file mode 100644 index 0000000000..9057519827 --- /dev/null +++ b/examples/cf-keycrypt.cf.priv @@ -0,0 +1,30 @@ +-----BEGIN RSA PRIVATE KEY----- +Proc-Type: 4,ENCRYPTED +DEK-Info: DES-EDE3-CBC,C213C4387710A6A4 + +L1J7NLoZ0DogIHAQXb0MR62kSGyIqRkZX5mLDKAf76mP/ejKGo27s8gwV7wVioFC +uTAY5j9066/XsUdRN2vjBmX+/VM9SPciRrXwsKFUxqd23swxEEG9FjTWmp8z5mKK +VNUh3C1CcSMVYjigSKInQP3M4/iMZ5EvzXgCsPxgfLgIf0j2+lF3Qxsdy0uzBCxz +t19IjgvG4LCn26Vd1tm8QomPBK5y1e6OEuxh7Ke8TcOh/93nACIpuo/hGw1FtJ6Z +zQXNdrDklP96FwrzsUdaDd6tuHTTTiqHkviaF9jkEUbfkIZJkWzqAFGhcW0So+28 +OiCzpfESTDTaj3aVagc9UovJ2/S+AnlbHxrf24VCo5tD0dQzk9h5Pjg8Ze5N5el8 +tmGkPMcvSDwr1RkPm1Xqipa1FbPGnfnebRNxc95vRw1JIf0eF0PCvVS0f4XjncBG +QhUVeM6XKsqxoq3GC6ab/6cC3ZiVZNQpcptzt6Qy9+cDz0Et21Y8RJPdOT8wR+Sr +SQST/wKW+H/xNh4sld+gLF3v6MACq5Wk2trAZYS+MdmelVps7LSQBnHPD1oDOkkq +cL21cdevScDMjvbRnL2oj2054320rGBXS2VkT4XI7Qw/JeqNfT0Ue/9n+TABx60u +pbPKpCfJzxQIErE5XEQU27+2IVYu5fK+BVi6s7UYUU40NePRlWFTfLRCVskrm63u +UnZ4o8nKz+5DLtCdtTlRGOZDCwYQYOFawCfwLN9wToHqQZxC1wzqRTsqXoTEVmby +hp+Qji55e93dFgLpIKB54flj2v3Wdbk8FPx3mjzf9MCxRaaGhsuX0WHQyHzIDb9K +raFWYcZJ1S8DUz5o7Q8otJaxj+nPGLCKTpBw/UyfXi1tXkGSRSIDo1dUtMaE2MpL +J44u4BuokKSOZbOi/piJy5wR0GSN6FHiDJo3yNbz/ZFXCq8l4ZQsVFNtB+9BzKX1 +ZkTSe8JX1guGLo3+MKsLOpriBzmjDJ2JvziXa1S4P9l3SziwCfreDqMM3dw5+N2z +rmtruWDMCURiVQXyuUToKRvzxiroOWIUofHrBoCG23Ztz3R+qlpXtYNy93z9tcg1 +R8CeFXymXq8q2TpUorhLrZGf7Q6fZxLV66tWfkYje4oicH4qo+SFqN8T1/T6kJd4 +kVj28Vb1RB4Phz7dkOyjKNo7qYXcedUbygUiAepLfnFgk7L9NEldn5CR6e1TxJoZ +fD0ib/naOuUKdesc2YF2EAoS0Un9O4/YgZMCBNBUI412MGteiqOWwn0K6MNgiIKr +0ToBtNRsEXM/7p5/q32NDBuWOfL1/82ptMShRIlCMemZAlswbibuJnA6KI6smUIi +LD4fti2143FqLQg3pkvjc6PEuAHUFVxfUSx3Kn/8mhKNbZxJuU8V/OfTEnfPKo+3 +DyesHX7fTvnea5K+XJZY4fZRMxuzXtCcbIfq2wc5D4XPUy32wGqfsV9fmiEGqG7/ +ul1uquFUg4qfBhKqbFdIt7x7XSDiCSzdQsC0kQ9XI5OiIQBrPIctXEsQxfsvGzx5 +tI9xJ40eiNbFZfOLNyPOZ9RS5E7SHe0YeOQGkcbb1/+yY8GjaxTTYg== +-----END RSA PRIVATE KEY----- diff --git a/examples/cf-keycrypt.cf.pub b/examples/cf-keycrypt.cf.pub new file mode 100644 index 0000000000..4fc29eff00 --- /dev/null +++ b/examples/cf-keycrypt.cf.pub @@ -0,0 +1,8 @@ +-----BEGIN RSA PUBLIC KEY----- +MIIBCgKCAQEAt96mvMUSXyJSr4jjcfrADkIm+8uqLGp0pHL4zrJLo8o/o8WnQyKl +hK614sAA/XlQmOIZ1KZ8x5Nw3fd/EFxEJ9wugz2iMbQTyRNgyJ4y68wrlBumKDgL +rtVDgb1ozIVh9Q5SEdGuV6MgHwNI7ubaLyuLetn+mlLcJXbzs6V75yWvCdfPfAt+ +aVONUChFRi1HydH34KEmEOgEca8YQW3FxjRYOnzeU81B9C3UGv3aFs8UQtNBvvd5 +15GQGnjimkZ3Ukpp0M/CrpcpoAU7X4AywvZLB3X5/JhZVw4wgFlprIdbbL4wd9ov +Q+4KoG+3BaMf5Y6NKvLRshONh5e3j7V4kQIDAQAB +-----END RSA PUBLIC KEY----- diff --git a/tests/acceptance/00_basics/cf-keycrypt/decrypt.cf b/tests/acceptance/00_basics/cf-keycrypt/decrypt.cf new file mode 100644 index 0000000000..78b5520cf5 --- /dev/null +++ b/tests/acceptance/00_basics/cf-keycrypt/decrypt.cf @@ -0,0 +1,27 @@ +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +bundle agent test +{ + meta: + "description" + string => "Test that cf-keycrypt can still decrypt content encrypted at the time of initial implementation"; + + commands: + "$(sys.cf_keycrypt)" + args => "-d $(this.promise_dirname)/testkey.priv -i $(this.promise_dirname)/encrypted -o $(G.testfile)"; + +} + +bundle agent check +{ + methods: + "any" + usebundle => dcs_check_diff("$(this.promise_dirname)/plaintext", + "$(G.testfile)", + "$(this.promise_filename)"); +} diff --git a/tests/acceptance/00_basics/cf-keycrypt/encrypt-decrypt.cf b/tests/acceptance/00_basics/cf-keycrypt/encrypt-decrypt.cf new file mode 100644 index 0000000000..1b44f2172c --- /dev/null +++ b/tests/acceptance/00_basics/cf-keycrypt/encrypt-decrypt.cf @@ -0,0 +1,53 @@ +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +bundle agent init +{ + methods: + "any" usebundle => dcs_fini("$(G.testfile).plain"); + "any" usebundle => dcs_fini("$(G.testfile).encrypted"); + "any" usebundle => dcs_fini("$(G.testfile).decrypted"); + "any" usebundle => generate_key; + "any" usebundle => trust_key; +} + +bundle agent test +{ + meta: + "description" + string => "Test that cf-keycrypt basic key based encryption and decryption work"; + + + vars: + "text" string => "This secret sauce should be encrypted and decrypted."; + + files: + "$(G.testfile).plain" + create => "true", + edit_defaults => empty, + edit_line => insert_lines( "$(text)" ); + + commands: + "$(sys.cf_keycrypt)" + args => "-e $(sys.workdir)/ppkeys/localhost.pub -i $(G.testfile).plain -o $(G.testfile).encrypted"; + "$(sys.cf_keycrypt)" + args => "-d $(sys.workdir)/ppkeys/localhost.priv -i $(G.testfile).encrypted -o $(G.testfile).decrypted"; + reports: + "Binaries/folders:"; + "$(sys.cf_keycrypt)"; + "$(sys.cf_agent)"; + "$(sys.workdir)"; +} + +bundle agent check +{ + methods: + "any" + usebundle => dcs_check_diff("$(G.testfile).plain", + "$(G.testfile).decrypted", + "$(this.promise_filename)"); +} diff --git a/tests/acceptance/00_basics/cf-keycrypt/encrypt.cf.x b/tests/acceptance/00_basics/cf-keycrypt/encrypt.cf.x new file mode 100644 index 0000000000..040cc6098e --- /dev/null +++ b/tests/acceptance/00_basics/cf-keycrypt/encrypt.cf.x @@ -0,0 +1,29 @@ +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +bundle agent test +{ + meta: + + "description" + string => "Test that cf-keycrypt can still encrypt content encrypted at the time of initial implementation"; + + # if this breaks then something may have changed with supported key types (this was originally introduced in 3.11.0). + + commands: + "$(sys.cf_keycrypt)" + args => "-e $(this.promise_dirname)/testkey.pub -i $(this.promise_dirname)/plaintext -o $(G.testfile)"; +} + +bundle agent check +{ + methods: + "any" + usebundle => dcs_check_diff("$(this.promise_dirname)/encrypted", + "$(G.testfile)", + "$(this.promise_filename)"); +} diff --git a/tests/acceptance/00_basics/cf-keycrypt/encrypted b/tests/acceptance/00_basics/cf-keycrypt/encrypted new file mode 100644 index 0000000000..6f1ab1ff18 --- /dev/null +++ b/tests/acceptance/00_basics/cf-keycrypt/encrypted @@ -0,0 +1,2 @@ +®Q–ßíéîù’VO["N߀ 1ü}΢n„#©ìh‡J¸2Ê}5ƒ›ôN1múnµ,­áE=§Ð\Õó ‡Ù£­ƒg>ôèoqVÛ¸PÐ-{.ÿG‹w$ \ No newline at end of file diff --git a/tests/acceptance/00_basics/cf-keycrypt/plaintext b/tests/acceptance/00_basics/cf-keycrypt/plaintext new file mode 100644 index 0000000000..ab9c75451f --- /dev/null +++ b/tests/acceptance/00_basics/cf-keycrypt/plaintext @@ -0,0 +1 @@ +Super secret message is here diff --git a/tests/acceptance/00_basics/cf-keycrypt/testkey.priv b/tests/acceptance/00_basics/cf-keycrypt/testkey.priv new file mode 100644 index 0000000000..9057519827 --- /dev/null +++ b/tests/acceptance/00_basics/cf-keycrypt/testkey.priv @@ -0,0 +1,30 @@ +-----BEGIN RSA PRIVATE KEY----- +Proc-Type: 4,ENCRYPTED +DEK-Info: DES-EDE3-CBC,C213C4387710A6A4 + +L1J7NLoZ0DogIHAQXb0MR62kSGyIqRkZX5mLDKAf76mP/ejKGo27s8gwV7wVioFC +uTAY5j9066/XsUdRN2vjBmX+/VM9SPciRrXwsKFUxqd23swxEEG9FjTWmp8z5mKK +VNUh3C1CcSMVYjigSKInQP3M4/iMZ5EvzXgCsPxgfLgIf0j2+lF3Qxsdy0uzBCxz +t19IjgvG4LCn26Vd1tm8QomPBK5y1e6OEuxh7Ke8TcOh/93nACIpuo/hGw1FtJ6Z +zQXNdrDklP96FwrzsUdaDd6tuHTTTiqHkviaF9jkEUbfkIZJkWzqAFGhcW0So+28 +OiCzpfESTDTaj3aVagc9UovJ2/S+AnlbHxrf24VCo5tD0dQzk9h5Pjg8Ze5N5el8 +tmGkPMcvSDwr1RkPm1Xqipa1FbPGnfnebRNxc95vRw1JIf0eF0PCvVS0f4XjncBG +QhUVeM6XKsqxoq3GC6ab/6cC3ZiVZNQpcptzt6Qy9+cDz0Et21Y8RJPdOT8wR+Sr +SQST/wKW+H/xNh4sld+gLF3v6MACq5Wk2trAZYS+MdmelVps7LSQBnHPD1oDOkkq +cL21cdevScDMjvbRnL2oj2054320rGBXS2VkT4XI7Qw/JeqNfT0Ue/9n+TABx60u +pbPKpCfJzxQIErE5XEQU27+2IVYu5fK+BVi6s7UYUU40NePRlWFTfLRCVskrm63u +UnZ4o8nKz+5DLtCdtTlRGOZDCwYQYOFawCfwLN9wToHqQZxC1wzqRTsqXoTEVmby +hp+Qji55e93dFgLpIKB54flj2v3Wdbk8FPx3mjzf9MCxRaaGhsuX0WHQyHzIDb9K +raFWYcZJ1S8DUz5o7Q8otJaxj+nPGLCKTpBw/UyfXi1tXkGSRSIDo1dUtMaE2MpL +J44u4BuokKSOZbOi/piJy5wR0GSN6FHiDJo3yNbz/ZFXCq8l4ZQsVFNtB+9BzKX1 +ZkTSe8JX1guGLo3+MKsLOpriBzmjDJ2JvziXa1S4P9l3SziwCfreDqMM3dw5+N2z +rmtruWDMCURiVQXyuUToKRvzxiroOWIUofHrBoCG23Ztz3R+qlpXtYNy93z9tcg1 +R8CeFXymXq8q2TpUorhLrZGf7Q6fZxLV66tWfkYje4oicH4qo+SFqN8T1/T6kJd4 +kVj28Vb1RB4Phz7dkOyjKNo7qYXcedUbygUiAepLfnFgk7L9NEldn5CR6e1TxJoZ +fD0ib/naOuUKdesc2YF2EAoS0Un9O4/YgZMCBNBUI412MGteiqOWwn0K6MNgiIKr +0ToBtNRsEXM/7p5/q32NDBuWOfL1/82ptMShRIlCMemZAlswbibuJnA6KI6smUIi +LD4fti2143FqLQg3pkvjc6PEuAHUFVxfUSx3Kn/8mhKNbZxJuU8V/OfTEnfPKo+3 +DyesHX7fTvnea5K+XJZY4fZRMxuzXtCcbIfq2wc5D4XPUy32wGqfsV9fmiEGqG7/ +ul1uquFUg4qfBhKqbFdIt7x7XSDiCSzdQsC0kQ9XI5OiIQBrPIctXEsQxfsvGzx5 +tI9xJ40eiNbFZfOLNyPOZ9RS5E7SHe0YeOQGkcbb1/+yY8GjaxTTYg== +-----END RSA PRIVATE KEY----- diff --git a/tests/acceptance/00_basics/cf-keycrypt/testkey.pub b/tests/acceptance/00_basics/cf-keycrypt/testkey.pub new file mode 100644 index 0000000000..4fc29eff00 --- /dev/null +++ b/tests/acceptance/00_basics/cf-keycrypt/testkey.pub @@ -0,0 +1,8 @@ +-----BEGIN RSA PUBLIC KEY----- +MIIBCgKCAQEAt96mvMUSXyJSr4jjcfrADkIm+8uqLGp0pHL4zrJLo8o/o8WnQyKl +hK614sAA/XlQmOIZ1KZ8x5Nw3fd/EFxEJ9wugz2iMbQTyRNgyJ4y68wrlBumKDgL +rtVDgb1ozIVh9Q5SEdGuV6MgHwNI7ubaLyuLetn+mlLcJXbzs6V75yWvCdfPfAt+ +aVONUChFRi1HydH34KEmEOgEca8YQW3FxjRYOnzeU81B9C3UGv3aFs8UQtNBvvd5 +15GQGnjimkZ3Ukpp0M/CrpcpoAU7X4AywvZLB3X5/JhZVw4wgFlprIdbbL4wd9ov +Q+4KoG+3BaMf5Y6NKvLRshONh5e3j7V4kQIDAQAB +-----END RSA PUBLIC KEY----- diff --git a/tests/acceptance/testall b/tests/acceptance/testall index ba0eacf9a4..807db54974 100755 --- a/tests/acceptance/testall +++ b/tests/acceptance/testall @@ -100,6 +100,7 @@ CF_PROMISES=${CF_PROMISES:-} ; export CF_PROMISES CF_SERVERD=${CF_SERVERD:-} ; export CF_SERVERD CF_EXECD=${CF_EXECD:-} ; export CF_EXECD CF_KEY=${CF_KEY:-} ; export CF_KEY +CF_KEYCRYPT=${CF_KEYCRYPT:-} ; export CF_KEYCRYPT CF_NET=${CF_NET:-} ; export CF_NET CF_CHECK=${CF_CHECK:-} ; export CF_CHECK CF_RUNAGENT=${CF_RUNAGENT:-} ; export CF_RUNAGENT @@ -207,7 +208,7 @@ unix_seconds() { } usage() { - echo "testall [-h|--help] [-q|--quiet] [--gainroot=] [--agent=] [--cfpromises=] [--cfserverd=] [--cfexecd=] [--cfkey=] [--cfnet=] [--cfcheck=] [--bindir=] [--tests=...] [--gdb] [--printlog] [ ...]" + echo "testall [-h|--help] [-q|--quiet] [--gainroot=] [--agent=] [--cfpromises=] [--cfserverd=] [--cfexecd=] [--cfkey=] [--cfkeycrypt=] [--cfnet=] [--cfcheck=] [--bindir=] [--tests=...] [--gdb] [--printlog] [ ...]" echo echo "If no test is given, all standard tests are run:" echo " Tests with names of form .cf are expected to run successfully" @@ -237,6 +238,8 @@ usage() { echo " and defaults to $DEFCF_EXECD." echo " --cfkey provides a way to specify non-default cf-key location," echo " and defaults to $DEFCF_KEY." + echo " --cfkeycrypt provides a way to specify non-default cf-keycrypt location," + echo " and defaults to $DEFCF_KEYCRYPT." echo " --cfnet provides a way to specify non-default cf-net location," echo " and defaults to $DEFCF_NET." echo " --cfcheck provides a way to specify non-default cf-check location," @@ -422,6 +425,7 @@ runtest() { $LN_CMD "$CF_SERVERD" "$WORKDIR/bin" $LN_CMD "$CF_EXECD" "$WORKDIR/bin" $LN_CMD "$CF_KEY" "$WORKDIR/bin" + $LN_CMD "$CF_KEYCRYPT" "$WORKDIR/bin" $LN_CMD "$CF_NET" "$WORKDIR/bin" $LN_CMD "$CF_CHECK" "$WORKDIR/bin" $LN_CMD "$CF_RUNAGENT" "$WORKDIR/bin" @@ -827,6 +831,8 @@ do CF_EXECD=${1#--cfexecd=};; --cfkey=*) CF_KEY=${1#--cfkey=};; + --cfkeycrypt=*) + CF_KEYCRYPT=${1#--cfkeycrypt=};; --cfnet=*) CF_NET=${1#--cfnet=};; --cfcheck=*) @@ -914,7 +920,7 @@ do esac done -if [ -n "$AGENT" -o -n "$CF_PROMISES" -o -n "$CF_SERVERD" -o -n "$CF_EXECD" -o -n "$CF_KEY" -o -n "$CF_NET" -o -n "$CF_CHECK" -o -n "$CF_RUNAGENT" -o -n "$RPMVERCMP" ] +if [ -n "$AGENT" -o -n "$CF_PROMISES" -o -n "$CF_SERVERD" -o -n "$CF_EXECD" -o -n "$CF_KEY" -o -n "$CF_KEYCRYPT" -o -n "$CF_NET" -o -n "$CF_CHECK" -o -n "$CF_RUNAGENT" -o -n "$RPMVERCMP" ] then if [ -n "$BINDIR" ] then @@ -943,6 +949,7 @@ find_default_binary DEFCF_PROMISES cf-promises find_default_binary DEFCF_SERVERD cf-serverd find_default_binary DEFCF_EXECD cf-execd find_default_binary DEFCF_KEY cf-key +find_default_binary DEFCF_KEYCRYPT cf-keycrypt find_default_binary DEFCF_NET cf-net find_default_binary DEFCF_CHECK cf-check find_default_binary DEFCF_RUNAGENT cf-runagent @@ -956,13 +963,14 @@ CF_PROMISES=${CF_PROMISES:-${DEFCF_PROMISES}} CF_SERVERD=${CF_SERVERD:-${DEFCF_SERVERD}} CF_EXECD=${CF_EXECD:-${DEFCF_EXECD}} CF_KEY=${CF_KEY:-${DEFCF_KEY}} +CF_KEYCRYPT=${CF_KEYCRYPT:-${DEFCF_KEYCRYPT}} CF_NET=${CF_NET:-${DEFCF_NET}} CF_CHECK=${CF_CHECK:-${DEFCF_CHECK}} CF_RUNAGENT=${CF_RUNAGENT:-${DEFCF_RUNAGENT}} RPMVERCMP=${RPMVERCMP:-${DEFRPMVERCMP}} LIBTOOL=${LIBTOOL:-${DEFLIBTOOL}} -if [ ! -x "$AGENT" -o ! -x "$CF_PROMISES" -o ! -x "$CF_SERVERD" -o ! -x "$CF_EXECD" -o ! -x "$CF_KEY" -o ! -x "$CF_NET" -o ! -x "$CF_CHECK" -o ! -x "$CF_RUNAGENT" -o ! -x "$RPMVERCMP" ] +if [ ! -x "$AGENT" -o ! -x "$CF_PROMISES" -o ! -x "$CF_SERVERD" -o ! -x "$CF_EXECD" -o ! -x "$CF_KEY" -o ! -x "$CF_KEYCRYPT" -o ! -x "$CF_NET" -o ! -x "$CF_CHECK" -o ! -x "$CF_RUNAGENT" -o ! -x "$RPMVERCMP" ] then echo "ERROR: can't find cf-agent or other binary;" \ " Are you sure you're running this from OBJDIR" \ @@ -972,6 +980,7 @@ then echo '$CF_SERVERD =' "$CF_SERVERD" echo '$CF_EXECD =' "$CF_EXECD" echo '$CF_KEY =' "$CF_KEY" + echo '$CF_KEYCRYPT =' "$CF_KEYCRYPT" echo '$CF_RUNAGENT =' "$CF_RUNAGENT" echo '$CF_NET =' "$CF_NET" echo '$CF_CHECK =' "$CF_CHECK" From 68e2a62e11a2586bbf5bffd36c8d3793007f3a8d Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Thu, 13 Jul 2017 09:38:50 +0200 Subject: [PATCH 086/333] Changed the syntax of cf-keycrypt to match other components Also did a lot of refactoring, error checking, memory cleaning, et.c. Squashed commits: cf-keycrypt: license cf-keycrypt: PR fixes #1 cf-keycrypt: Fixed copyright in cf-keycrypt.c cf-keycrypt: CryptoInitialize() cf-keycrypt: Implemented new syntax as discussed in PR cf-keycrypt: Acceptance test for new arguments cf-keycrypt: Print help message when no arguments are added cf-keycrypt: PR fixes #2 cf-keycrypt: PR fixes #3 cf-keycrypt: PR fixes #4 cf-keycrypt: PR fixes #5 cf-keycrypt: Changed syntax to jimis' suggestion cf-keycrypt: Moved and fixed acceptance tests Changelog: Title Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 53d77697e9cba834b323bddf15bdac11af68df70) --- cf-keycrypt/cf-keycrypt.c | 496 +++++++++++------- .../00_basics/cf-keycrypt/decrypt.cf | 27 - .../00_basics/cf-keycrypt/encrypt-decrypt.cf | 53 -- .../00_basics/cf-keycrypt/encrypt.cf.x | 29 - tests/acceptance/27_cf-keycrypt/decrypt.cf | 26 + .../27_cf-keycrypt/encrypt-decrypt-args.cf | 53 ++ .../27_cf-keycrypt/encrypt-decrypt.cf | 55 ++ tests/acceptance/27_cf-keycrypt/encrypt.x.cf | 28 + .../cf-keycrypt => 27_cf-keycrypt}/encrypted | 0 .../cf-keycrypt => 27_cf-keycrypt}/plaintext | 0 .../testkey.priv | 0 .../testkey.pub | 0 12 files changed, 468 insertions(+), 299 deletions(-) delete mode 100644 tests/acceptance/00_basics/cf-keycrypt/decrypt.cf delete mode 100644 tests/acceptance/00_basics/cf-keycrypt/encrypt-decrypt.cf delete mode 100644 tests/acceptance/00_basics/cf-keycrypt/encrypt.cf.x create mode 100644 tests/acceptance/27_cf-keycrypt/decrypt.cf create mode 100644 tests/acceptance/27_cf-keycrypt/encrypt-decrypt-args.cf create mode 100644 tests/acceptance/27_cf-keycrypt/encrypt-decrypt.cf create mode 100644 tests/acceptance/27_cf-keycrypt/encrypt.x.cf rename tests/acceptance/{00_basics/cf-keycrypt => 27_cf-keycrypt}/encrypted (100%) rename tests/acceptance/{00_basics/cf-keycrypt => 27_cf-keycrypt}/plaintext (100%) rename tests/acceptance/{00_basics/cf-keycrypt => 27_cf-keycrypt}/testkey.priv (100%) rename tests/acceptance/{00_basics/cf-keycrypt => 27_cf-keycrypt}/testkey.pub (100%) diff --git a/cf-keycrypt/cf-keycrypt.c b/cf-keycrypt/cf-keycrypt.c index 4c4894a372..bd172de326 100644 --- a/cf-keycrypt/cf-keycrypt.c +++ b/cf-keycrypt/cf-keycrypt.c @@ -1,3 +1,27 @@ +/* + Copyright 2017 Northern.tech AS + + This file is part of CFEngine 3 - written and maintained by Northern.tech AS. + + This program is free software; you can redistribute it and/or modify it + under the terms of the GNU General Public License as published by the + Free Software Foundation; version 3. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA + + To the extent this program is licensed as part of the Enterprise + versions of CFEngine, the applicable Commercial Open Source License + (COSL) may apply to this file if you as a licensee so wish it. See + included file COSL.txt. +*/ + /* cf-keycrypt.c @@ -19,59 +43,63 @@ */ -#include #include #include #include #include +#include +#include +#include #include #include #include #include #include #include -/* -#ifdef LMDB -#include -#endif -*/ - -#define STDIN 0 -#define STDOUT 1 #define BUFSIZE 1024 -void usage() +static const char passphrase[] = "Cfengine passphrase"; + +//******************************************************************* +// DOCUMENTATION / GETOPT CONSTS: +//******************************************************************* + +static const char *const CF_KEYCRYPT_SHORT_DESCRIPTION = + "cf-keycrypt: Use CFEngine cryptographic keys to encrypt and decrypt files"; + +static const char *const CF_KEYCRYPT_MANPAGE_LONG_DESCRIPTION = + "cf-keycrypt offers a simple way to encrypt or decrypt files using keys " + "generated by cf-key. CFEngine uses asymmetric cryptography, and " + "cf-keycrypt allows you to encrypt a file using a public key file. " + "The encrypted file can only be decrypted on the host with the " + "corresponding private key. Original author: Jon Henrik Bjornstad " + ""; + +static const struct option OPTIONS[] = { - printf("Workdir is defined as %s\n", WORKDIR); - printf( - "\n" - "Usage: cf-keycrypt [-e public-key|-d private-key|-H hostname-or-ip] -o outfile -i infile [-h]\n" - "\n" - "Use CFEngine cryptographic keys to encrypt and decrypt files, eg. files containing\n" - "passwords or certificates.\n" - "\n" - " -e Encrypt with key.\n" - " -d Decrypt with key.\n" - " -i File to encrypt/decrypt. '-' reads from stdin.\n" - " -o File to write encrypted/decrypted contents to. '-' writes to stdout.\n" - " -H Hostname/IP to encrypt for, gets data from lastseen database\n" - " -h Print help.\n" - "\n" - "Examples:\n" - " Encrypt:\n" - " cf-keycrypt -e /var/cfengine/ppkeys/localhost.pub -i myplain.dat -o mycrypt.dat\n" - "\n" - " Decrypt:\n" - " cf-keycrypt -d /var/cfengine/ppkeys/localhost.priv -i mycrypt.dat -o myplain.dat\n" - "\n" - "Written and maintained by Jon Henrik Bjornstad \n" - "\n" - "Copyright (C) cfengineers.net\n" - "\n" - ); -} + {"help", no_argument, 0, 'h'}, + {"manpage", no_argument, 0, 'M'}, + {"encrypt", no_argument, 0, 'e'}, + {"decrypt", no_argument, 0, 'd'}, + {"key", required_argument, 0, 'k'}, + {"host", required_argument, 0, 'H'}, + {"output", required_argument, 0, 'o'}, + {NULL, 0, 0, '\0'} +}; + +static const char *const HINTS[] = +{ + "Print the help message", + "Print the man page", + "Encrypt file", + "Decrypt file", + "Use key file", + "Encrypt for host (get key from lastseen database)", + "Output file", + NULL +}; void *get_in_addr(struct sockaddr *sa) { @@ -82,22 +110,17 @@ void *get_in_addr(struct sockaddr *sa) return &(((struct sockaddr_in6*)sa)->sin6_addr); } -int file_exist(char *filename) +int file_exist(const char *filename) { struct stat buffer; - return (stat (filename, &buffer) == 0); + return stat(filename, &buffer) == 0; } -char *get_host_pubkey(char *host) +char *get_host_pubkey(const char *host) { - // char *key; // TODO: this was unused(!) - // char value[BUFSIZE]; // TODO: this was unused(!) char *buffer = (char *) malloc(BUFSIZE *sizeof(char)); - //char *keyname = NULL; char hash[CF_HOSTKEY_STRING_SIZE]; - //char buffer[BUFSIZE]; char ipaddress[BUFSIZE]; - // int ecode; // TODO: this was unused(!) struct addrinfo *result; struct addrinfo *res; bool found = false; @@ -124,16 +147,24 @@ char *get_host_pubkey(char *host) if (found) { snprintf(buffer, BUFSIZE * sizeof(char), "%s/ppkeys/root-%s.pub", WORKDIR, hash); + freeaddrinfo(result); return buffer; } else { for(res = result; res != NULL; res = res->ai_next) { - inet_ntop(res->ai_family, get_in_addr((struct sockaddr *)res->ai_addr), ipaddress, sizeof(ipaddress)); - snprintf(buffer, BUFSIZE * sizeof(char), "%s/ppkeys/root-%s.pub", WORKDIR, ipaddress); + inet_ntop( + res->ai_family, + get_in_addr((struct sockaddr *)res->ai_addr), + ipaddress, + sizeof(ipaddress)); + snprintf( + buffer, BUFSIZE * sizeof(char), "%s/ppkeys/root-%s.pub", + WORKDIR, ipaddress); if (file_exist(buffer)) { + freeaddrinfo(result); return buffer; } } @@ -141,248 +172,333 @@ char *get_host_pubkey(char *host) return NULL; } -void *readseckey(char *secfile) +RSA *readseckey(const char *privkey_path) { - FILE *fp = NULL; - RSA* PRIVKEY = NULL; - static char *passphrase = "Cfengine passphrase"; - unsigned long err; + FILE *fp = fopen(privkey_path,"r"); - if ((fp = fopen(secfile,"r")) == NULL) + if (fp == NULL) { - printf("Couldn't find a private key - use cf-key to get one"); - return (void *)NULL; - } - - if ((PRIVKEY = PEM_read_RSAPrivateKey(fp,(RSA **)NULL,NULL,passphrase)) == NULL) - { - err = ERR_get_error(); - printf("PEM_readError reading Private Key = %s\n", ERR_reason_error_string(err)); - PRIVKEY = NULL; - fclose(fp); - return (void *)NULL; + Log(LOG_LEVEL_ERR, "Could not open private key '%s'", privkey_path); + return NULL; } - else + RSA *privkey = PEM_read_RSAPrivateKey(fp, (RSA **)NULL, NULL, + (void *) passphrase); + if (privkey == NULL) { - return (void *)PRIVKEY; + unsigned long err = ERR_get_error(); + Log(LOG_LEVEL_ERR, "Could not read private key '%s': %s", + privkey_path, ERR_reason_error_string(err)); } + fclose(fp); + return privkey; } -void *readpubkey(char *pubfile) +RSA *readpubkey(const char *pubkey_path) { - FILE *fp; - RSA *key = NULL; - static char *passphrase = "Cfengine passphrase"; + FILE *fp = fopen(pubkey_path, "r"); - if ((fp = fopen(pubfile, "r")) == NULL) + if (fp == NULL) { - fprintf(stderr,"Error: Cannot locate Public Key file '%s'.\n", pubfile); + Log(LOG_LEVEL_ERR, "Could not open public key '%s'", pubkey_path); return NULL; } - if ((key = PEM_read_RSAPublicKey(fp,(RSA **)NULL,NULL,passphrase)) == NULL) + + RSA *pubkey = PEM_read_RSAPublicKey( + fp, NULL, NULL, (void *)passphrase); + if (pubkey == NULL) { - fprintf(stderr,"Error: failed reading Public Key in '%s' file.\n", pubfile); - return NULL; + Log(LOG_LEVEL_ERR, "Could not read public key '%s'", pubkey_path); } fclose(fp); - return (void *)key; + return pubkey; } -long int rsa_encrypt(char *pubfile, char *filein, char *fileout) +long int rsa_encrypt( + const char *pubkey_path, const char *input_path, const char *output_path) { int ks = 0; - unsigned long size = 0, len = 0, ciphsz = 0; - RSA* key = NULL; - FILE *infile = NULL; - FILE *outfile = NULL; - char *tmpciph = NULL, *tmpplain = NULL; - - key = (RSA *)readpubkey(pubfile); - if (!key) + unsigned long len = 0, ciphsz = 0; + + RSA* pubkey = readpubkey(pubkey_path); + if (pubkey == NULL) { return -1; } - if ((strcmp("-",filein) == 0)) - { - infile = stdin; - } - else if (!(infile = fopen(filein, "r"))) + FILE *input_file = fopen(input_path, "r"); + if (input_file == NULL) { - fprintf(stderr, "Error: Cannot locate input file.\n"); + Log(LOG_LEVEL_ERR, "Could not open input file '%s'", input_path); + RSA_free(pubkey); return -1; } - if ((strcmp("-",fileout) == 0)) + FILE *output_file = fopen(output_path, "w"); + if (output_file == NULL) { - outfile = stdout; - //printf("Write to stdout\n"); - } - else if (!(outfile = fopen(fileout, "w"))) - { - fprintf(stderr, "Error: Cannot create output file.\n"); + Log(LOG_LEVEL_ERR, "Could not create output file '%s'", output_path); + fclose(input_file); + RSA_free(pubkey); return -1; } - ks = RSA_size(key); - tmpplain = (unsigned char *)malloc(ks * sizeof(unsigned char)); - tmpciph = (unsigned char *)malloc(ks * sizeof(unsigned char)); - srand(time(NULL)); + ks = RSA_size(pubkey); + char tmp_ciphertext[ks], tmp_plaintext[ks]; - while(!feof(infile)) + srand(time(NULL)); + bool error_return = false; + while (!feof(input_file)) { - memset(tmpplain, '\0', ks); - memset(tmpciph, '\0', ks); - len = fread(tmpplain, 1, ks-RSA_PKCS1_PADDING_SIZE, infile); - if (len <= 0) + memset(tmp_plaintext, '\0', ks); + memset(tmp_ciphertext, '\0', ks); + len = fread( + tmp_plaintext, 1, ks - RSA_PKCS1_PADDING_SIZE, input_file); + if (len <= 0 || ferror(input_file) != 0) + { + Log(LOG_LEVEL_ERR, "Could not read file '%s'", input_path); + error_return = true; + break; + } + unsigned long size = RSA_public_encrypt( + strlen(tmp_plaintext), tmp_plaintext, tmp_ciphertext, + pubkey, RSA_PKCS1_PADDING); + if (size == -1) { - Log(LOG_LEVEL_ERR, "Could not read file '%s'(fread: %lu)", - filein, len); + Log(LOG_LEVEL_ERR, "%s", + ERR_error_string(ERR_get_error(), NULL)); + error_return = true; + break; } - if ((size = RSA_public_encrypt(strlen(tmpplain), tmpplain, tmpciph, key, RSA_PKCS1_PADDING)) == -1) + fwrite(tmp_ciphertext, sizeof(unsigned char), ks, output_file); + if (ferror(output_file) != 0) { - fprintf(stderr, "%s\n", ERR_error_string(ERR_get_error(), NULL)); - return -1; + Log(LOG_LEVEL_ERR, "Could not write file '%s'", output_path); + error_return = true; + break; } - fwrite(tmpciph,sizeof(unsigned char),ks,outfile); } - fclose(infile); - fclose(outfile); - free(tmpciph); - free(tmpplain); - RSA_free(key); + fclose(input_file); + fclose(output_file); + RSA_free(pubkey); + if (error_return) + { + return -1; + } return ciphsz; } -long int rsa_decrypt(char *secfile, char *cryptfile, char *plainfile) +long int rsa_decrypt( + const char *privkey_path, const char *input_path, const char *output_path) { - unsigned long int plsz = 0, size = 0, ks = 0, len = 0; // TODO: Unused: len = 0; - RSA *key = NULL; - char *tmpplain = NULL, *tmpciph = NULL; - - FILE *infile = NULL; - FILE *outfile = NULL; + unsigned long int plsz = 0, size = 0; - key = (RSA *)readseckey(secfile); - if (!key) { + RSA *privkey = (RSA *)readseckey(privkey_path); + if (privkey == NULL) { return -1; } - if ((strcmp("-",cryptfile) == 0)) { - infile = stdin; - }else if (!(infile = fopen(cryptfile, "r"))) { - fprintf(stderr, "Error: Cannot locate input file.\n"); + FILE *input_file = fopen(input_path, "r"); + if (input_file == NULL) { + Log(LOG_LEVEL_ERR, "Cannot open input file '%s'", input_path); + RSA_free(privkey); return -1; } - if ((strcmp("-",plainfile) == 0)) { - outfile = stdout; - //printf("Write to stdout\n"); - }else if (!(outfile = fopen(plainfile, "w"))){ - fprintf(stderr, "Error: Cannot create output file.\n"); + FILE *output_file = fopen(output_path, "w"); + if (output_file == NULL){ + Log(LOG_LEVEL_ERR, "Cannot open output file '%s'", output_path); + fclose(input_file); + RSA_free(privkey); return -1; } - ks = RSA_size(key); - tmpciph = (unsigned char *)malloc(ks * sizeof(unsigned char)); - tmpplain = (unsigned char *)malloc(ks * sizeof(unsigned char)); - //printf("Keysize: %d\n", ks); - while(!feof(infile)) + unsigned long int ks = RSA_size(privkey); + char tmp_ciphertext[ks], tmp_plaintext[ks]; + + bool error_return = false; + while (!feof(input_file)) { - memset(tmpciph, '\0', ks); - memset(tmpplain, '\0', ks); - len = fread(tmpciph, 1, ks, infile); + memset(tmp_ciphertext, '\0', ks); + memset(tmp_plaintext, '\0', ks); + unsigned long int len = fread( + tmp_ciphertext, 1, ks, input_file); + if (ferror(input_file) != 0) + { + Log(LOG_LEVEL_ERR, "Could not read from '%s'", input_path); + error_return = true; + break; + } if (len > 0){ - if ((size = RSA_private_decrypt(ks, tmpciph, tmpplain, key, RSA_PKCS1_PADDING)) == -1) + size = RSA_private_decrypt( + ks, tmp_ciphertext, tmp_plaintext, privkey, RSA_PKCS1_PADDING); + if (size == -1) { - fprintf(stderr, "%s\n", ERR_error_string(ERR_get_error(), NULL)); - return -1; + Log(LOG_LEVEL_ERR, "%s", + ERR_error_string(ERR_get_error(), NULL)); + error_return = true; + break; } } - fwrite(tmpplain,1,strlen(tmpplain),outfile); + + fwrite(tmp_plaintext, 1, strlen(tmp_plaintext), output_file); + if (ferror(output_file) != 0) + { + Log(LOG_LEVEL_ERR, "Could not write to '%s'", output_path); + error_return = true; + break; + } + } + fclose(input_file); + fclose(output_file); + RSA_free(privkey); + if (error_return) + { + return -1; } - fclose(infile); - fclose(outfile); - free(tmpplain); - free(tmpciph); - RSA_free(key); return plsz; } +void cf_keycrypt_help() +{ + Writer *w = FileWriter(stdout); + WriterWriteHelp(w, "cf-keycrypt", OPTIONS, HINTS, false, NULL); + FileWriterDetach(w); +} + +void cf_keycrypt_man() +{ + Writer *out = FileWriter(stdout); + ManPageWrite(out, "cf-keycrypt", time(NULL), + CF_KEYCRYPT_SHORT_DESCRIPTION, + CF_KEYCRYPT_MANPAGE_LONG_DESCRIPTION, + OPTIONS, HINTS, true); + FileWriterDetach(out); +} int main(int argc, char *argv[]) { - OpenSSL_add_all_algorithms(); - ERR_load_crypto_strings(); + if (argc == 1) + { + cf_keycrypt_help(); + exit(EXIT_FAILURE); + } opterr = 0; char *key = NULL; - char *infile = NULL; - char *outfile = NULL; + char *input_path = NULL; + char *output_path = NULL; char *host = NULL; - int encrypt = 0; - int decrypt = 0; + bool encrypt = false; + bool decrypt = false; int c = 0; - int size = 0; - - //strcpy(WORKDIR,CFCR_WORKDIR); - - while ((c = getopt (argc, argv, "he:d:i:o:H:")) != -1) + while ((c = getopt_long(argc, argv, "hMedk:o:H:", OPTIONS, NULL)) != -1) { switch (c) { + case 'h': + cf_keycrypt_help(); + exit(EXIT_SUCCESS); + break; + case 'M': + cf_keycrypt_man(); + exit(EXIT_SUCCESS); + break; case 'e': - encrypt = 1; - key = optarg; + encrypt = true; break; case 'd': - decrypt = 1; - key = optarg; + decrypt = true; break; - case 'i': - infile = optarg; + case 'k': + key = optarg; break; case 'o': - outfile = optarg; + output_path = optarg; break; case 'H': - encrypt = 1; host = optarg; break; - case 'h': - usage(); - exit(1); default: - printf("ERROR: Unknown option '-%c'\n", optopt); - usage(); - exit(1); + Log(LOG_LEVEL_ERR, "Unknown option '-%c'", optopt); + cf_keycrypt_help(); + exit(EXIT_FAILURE); } } + + input_path = argv[optind]; + optind += 1; + if (argv[optind] != NULL) + { + Log(LOG_LEVEL_ERR, "Unexpected non-option argument: '%s'",argv[optind]); + exit(EXIT_FAILURE); + } + + // Check for argument errors: + if (encrypt == decrypt) + { + Log(LOG_LEVEL_ERR, "Must specify either encrypt or decrypt (and not both)"); + exit(EXIT_FAILURE); + } + + if (host != NULL && key != NULL) + { + Log(LOG_LEVEL_ERR, "--host/-H is used to specify a public key and cannot be used with --key/-k"); + exit(EXIT_FAILURE); + } + if (input_path == NULL) + { + Log(LOG_LEVEL_ERR, "No input file specified (Use -h for help)"); + exit(EXIT_FAILURE); + } + if (output_path == NULL) + { + Log(LOG_LEVEL_ERR, "No output file specified (Use -h for help)"); + exit(EXIT_FAILURE); + } + + // Resolve host to public key: + CryptoInitialize(); if (host) { key = get_host_pubkey(host); if (!key) { - fprintf(stderr,"ERROR: Unable to locate public key for host %s\n", host); - exit(1); + Log(LOG_LEVEL_ERR, "Unable to locate public key for host '%s'", host); + exit(EXIT_FAILURE); } } - if (encrypt > 0 && key && infile && outfile) + // Additional error checking: + if (key == NULL) + { + Log(LOG_LEVEL_ERR, "No key specified (Use -h for help)"); + exit(EXIT_FAILURE); + } + + // Encrypt or decrypt + int size = 0; + if (encrypt) { - size = rsa_encrypt(key, infile, outfile); + size = rsa_encrypt(key, input_path, output_path); if (size < 0) - exit(1); + { + Log(LOG_LEVEL_ERR, "Encryption failed"); + exit(EXIT_FAILURE); + } } - else if (decrypt > 0 && key && infile && outfile) + else if (decrypt) { - size = rsa_decrypt(key, infile, outfile); + size = rsa_decrypt(key, input_path, output_path); if (size < 0) - exit(1); + { + Log(LOG_LEVEL_ERR, "Decryption failed"); + exit(EXIT_FAILURE); + } } else { - usage(); - exit(1); + ProgrammingError("Unexpected error in cf-keycrypt"); + exit(EXIT_FAILURE); } - exit(0); + + return 0; } diff --git a/tests/acceptance/00_basics/cf-keycrypt/decrypt.cf b/tests/acceptance/00_basics/cf-keycrypt/decrypt.cf deleted file mode 100644 index 78b5520cf5..0000000000 --- a/tests/acceptance/00_basics/cf-keycrypt/decrypt.cf +++ /dev/null @@ -1,27 +0,0 @@ -body common control -{ - inputs => { "../../default.cf.sub" }; - bundlesequence => { default("$(this.promise_filename)") }; - version => "1.0"; -} - -bundle agent test -{ - meta: - "description" - string => "Test that cf-keycrypt can still decrypt content encrypted at the time of initial implementation"; - - commands: - "$(sys.cf_keycrypt)" - args => "-d $(this.promise_dirname)/testkey.priv -i $(this.promise_dirname)/encrypted -o $(G.testfile)"; - -} - -bundle agent check -{ - methods: - "any" - usebundle => dcs_check_diff("$(this.promise_dirname)/plaintext", - "$(G.testfile)", - "$(this.promise_filename)"); -} diff --git a/tests/acceptance/00_basics/cf-keycrypt/encrypt-decrypt.cf b/tests/acceptance/00_basics/cf-keycrypt/encrypt-decrypt.cf deleted file mode 100644 index 1b44f2172c..0000000000 --- a/tests/acceptance/00_basics/cf-keycrypt/encrypt-decrypt.cf +++ /dev/null @@ -1,53 +0,0 @@ -body common control -{ - inputs => { "../../default.cf.sub" }; - bundlesequence => { default("$(this.promise_filename)") }; - version => "1.0"; -} - -bundle agent init -{ - methods: - "any" usebundle => dcs_fini("$(G.testfile).plain"); - "any" usebundle => dcs_fini("$(G.testfile).encrypted"); - "any" usebundle => dcs_fini("$(G.testfile).decrypted"); - "any" usebundle => generate_key; - "any" usebundle => trust_key; -} - -bundle agent test -{ - meta: - "description" - string => "Test that cf-keycrypt basic key based encryption and decryption work"; - - - vars: - "text" string => "This secret sauce should be encrypted and decrypted."; - - files: - "$(G.testfile).plain" - create => "true", - edit_defaults => empty, - edit_line => insert_lines( "$(text)" ); - - commands: - "$(sys.cf_keycrypt)" - args => "-e $(sys.workdir)/ppkeys/localhost.pub -i $(G.testfile).plain -o $(G.testfile).encrypted"; - "$(sys.cf_keycrypt)" - args => "-d $(sys.workdir)/ppkeys/localhost.priv -i $(G.testfile).encrypted -o $(G.testfile).decrypted"; - reports: - "Binaries/folders:"; - "$(sys.cf_keycrypt)"; - "$(sys.cf_agent)"; - "$(sys.workdir)"; -} - -bundle agent check -{ - methods: - "any" - usebundle => dcs_check_diff("$(G.testfile).plain", - "$(G.testfile).decrypted", - "$(this.promise_filename)"); -} diff --git a/tests/acceptance/00_basics/cf-keycrypt/encrypt.cf.x b/tests/acceptance/00_basics/cf-keycrypt/encrypt.cf.x deleted file mode 100644 index 040cc6098e..0000000000 --- a/tests/acceptance/00_basics/cf-keycrypt/encrypt.cf.x +++ /dev/null @@ -1,29 +0,0 @@ -body common control -{ - inputs => { "../../default.cf.sub" }; - bundlesequence => { default("$(this.promise_filename)") }; - version => "1.0"; -} - -bundle agent test -{ - meta: - - "description" - string => "Test that cf-keycrypt can still encrypt content encrypted at the time of initial implementation"; - - # if this breaks then something may have changed with supported key types (this was originally introduced in 3.11.0). - - commands: - "$(sys.cf_keycrypt)" - args => "-e $(this.promise_dirname)/testkey.pub -i $(this.promise_dirname)/plaintext -o $(G.testfile)"; -} - -bundle agent check -{ - methods: - "any" - usebundle => dcs_check_diff("$(this.promise_dirname)/encrypted", - "$(G.testfile)", - "$(this.promise_filename)"); -} diff --git a/tests/acceptance/27_cf-keycrypt/decrypt.cf b/tests/acceptance/27_cf-keycrypt/decrypt.cf new file mode 100644 index 0000000000..340b672bd0 --- /dev/null +++ b/tests/acceptance/27_cf-keycrypt/decrypt.cf @@ -0,0 +1,26 @@ +body common control +{ + inputs => { "../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +bundle agent test +{ + meta: + "description" + string => "Test that cf-keycrypt can still decrypt content encrypted at the time of initial implementation"; + + commands: + "$(sys.cf_keycrypt)" + args => "-k $(this.promise_dirname)/testkey.priv -d $(this.promise_dirname)/encrypted -o $(G.testfile)"; +} + +bundle agent check +{ + methods: + "any" + usebundle => dcs_check_diff("$(this.promise_dirname)/plaintext", + "$(G.testfile)", + "$(this.promise_filename)"); +} diff --git a/tests/acceptance/27_cf-keycrypt/encrypt-decrypt-args.cf b/tests/acceptance/27_cf-keycrypt/encrypt-decrypt-args.cf new file mode 100644 index 0000000000..f87dcb310f --- /dev/null +++ b/tests/acceptance/27_cf-keycrypt/encrypt-decrypt-args.cf @@ -0,0 +1,53 @@ +body common control +{ + inputs => { "../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +bundle agent init +{ + methods: + "any" usebundle => dcs_fini("$(G.testfile).plain"); + "any" usebundle => dcs_fini("$(G.testfile).encrypted"); + "any" usebundle => dcs_fini("$(G.testfile).decrypted"); + "any" usebundle => generate_key; + "any" usebundle => trust_key; +} + +bundle agent test +{ + meta: + "description" + string => "Test cf-keycrypt with different arguments/order"; + + + vars: + "text" string => "This secret sauce should be encrypted and decrypted."; + + files: + "$(G.testfile).plain" + create => "true", + edit_defaults => empty, + edit_line => insert_lines( "$(text)" ); + + commands: + "$(sys.cf_keycrypt)" + args => "--encrypt $(G.testfile).plain --key $(sys.workdir)/ppkeys/localhost.pub --output $(G.testfile).encrypted"; + "$(sys.cf_keycrypt)" + args => "--decrypt -o $(G.testfile).decrypted --key $(sys.workdir)/ppkeys/localhost.priv $(G.testfile).encrypted"; + reports: + "Binaries/folders:"; + "$(sys.cf_keycrypt)"; + "$(sys.cf_agent)"; + "$(sys.workdir)"; +} + +bundle agent check +{ + methods: + "any" + usebundle => dcs_check_diff("$(G.testfile).plain", + "$(G.testfile).decrypted", + "$(this.promise_filename)"); +} diff --git a/tests/acceptance/27_cf-keycrypt/encrypt-decrypt.cf b/tests/acceptance/27_cf-keycrypt/encrypt-decrypt.cf new file mode 100644 index 0000000000..2dc5fc4916 --- /dev/null +++ b/tests/acceptance/27_cf-keycrypt/encrypt-decrypt.cf @@ -0,0 +1,55 @@ +body common control +{ + inputs => { "../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +bundle agent init +{ + methods: + "any" usebundle => dcs_fini("$(G.testfile).plain"); + "any" usebundle => dcs_fini("$(G.testfile).encrypted"); + "any" usebundle => dcs_fini("$(G.testfile).decrypted"); + "any" usebundle => generate_key; + "any" usebundle => trust_key; +} + +bundle agent test +{ + meta: + "description" + string => "Test that cf-keycrypt basic key based encryption and decryption work"; + + + vars: + "text" + string => "This secret sauce should be encrypted and decrypted."; + + files: + "$(G.testfile).plain" + create => "true", + edit_defaults => empty, + edit_line => insert_lines( "$(text)" ); + + commands: + "$(sys.cf_keycrypt)" + args => "-k $(sys.workdir)/ppkeys/localhost.pub -e $(G.testfile).plain -o $(G.testfile).encrypted"; + "$(sys.cf_keycrypt)" + args => "-k $(sys.workdir)/ppkeys/localhost.priv -d $(G.testfile).encrypted -o $(G.testfile).decrypted"; + + reports: + "Binaries/folders:"; + "$(sys.cf_keycrypt)"; + "$(sys.cf_agent)"; + "$(sys.workdir)"; +} + +bundle agent check +{ + methods: + "any" + usebundle => dcs_check_diff("$(G.testfile).plain", + "$(G.testfile).decrypted", + "$(this.promise_filename)"); +} diff --git a/tests/acceptance/27_cf-keycrypt/encrypt.x.cf b/tests/acceptance/27_cf-keycrypt/encrypt.x.cf new file mode 100644 index 0000000000..23e72337aa --- /dev/null +++ b/tests/acceptance/27_cf-keycrypt/encrypt.x.cf @@ -0,0 +1,28 @@ +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +bundle agent test +{ + meta: + "description" + string => "Test that cf-keycrypt encryption uses random padding"; + + commands: + "$(sys.cf_keycrypt)" + args => "-e $(this.promise_dirname)/plaintext -k $(this.promise_dirname)/testkey.pub -o $(G.testfile)"; + "$(sys.cf_keycrypt)" + args => "-e $(this.promise_dirname)/plaintext -k $(this.promise_dirname)/testkey.pub -o $(G.testfile).2"; +} + +bundle agent check +{ + methods: + "any" + usebundle => dcs_check_diff("$(G.testfile)", + "$(G.testfile).2", + "$(this.promise_filename)"); +} diff --git a/tests/acceptance/00_basics/cf-keycrypt/encrypted b/tests/acceptance/27_cf-keycrypt/encrypted similarity index 100% rename from tests/acceptance/00_basics/cf-keycrypt/encrypted rename to tests/acceptance/27_cf-keycrypt/encrypted diff --git a/tests/acceptance/00_basics/cf-keycrypt/plaintext b/tests/acceptance/27_cf-keycrypt/plaintext similarity index 100% rename from tests/acceptance/00_basics/cf-keycrypt/plaintext rename to tests/acceptance/27_cf-keycrypt/plaintext diff --git a/tests/acceptance/00_basics/cf-keycrypt/testkey.priv b/tests/acceptance/27_cf-keycrypt/testkey.priv similarity index 100% rename from tests/acceptance/00_basics/cf-keycrypt/testkey.priv rename to tests/acceptance/27_cf-keycrypt/testkey.priv diff --git a/tests/acceptance/00_basics/cf-keycrypt/testkey.pub b/tests/acceptance/27_cf-keycrypt/testkey.pub similarity index 100% rename from tests/acceptance/00_basics/cf-keycrypt/testkey.pub rename to tests/acceptance/27_cf-keycrypt/testkey.pub From 2b19f82b39aedc940f4ef4cea3ee8573f4ab5fe4 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 5 Feb 2020 11:21:01 +0100 Subject: [PATCH 087/333] Include 'statistics.h' from lastseen.h It's using the 'Qpoint' type from there. (cherry picked from commit 9969a828f852b1fb1af4b727b84e25bfefea2163) --- libpromises/lastseen.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/libpromises/lastseen.h b/libpromises/lastseen.h index 56125304fd..6460ce311e 100644 --- a/libpromises/lastseen.h +++ b/libpromises/lastseen.h @@ -25,6 +25,8 @@ #ifndef CFENGINE_LASTSEEN_H #define CFENGINE_LASTSEEN_H +#include + typedef struct { time_t lastseen; From 650f533f383f039086e0e47a50947b2e6a6237b9 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 5 Feb 2020 11:21:34 +0100 Subject: [PATCH 088/333] Adapt cf-keycrypt source to changes in the codebase The codebase has evolved since the last attempt to merge cf-keycrypt in #2883. (cherry picked from commit bd3e840d55707f8612818b8dc51ede8d6c5121d5) --- cf-keycrypt/Makefile.am | 2 +- cf-keycrypt/cf-keycrypt.c | 8 ++------ 2 files changed, 3 insertions(+), 7 deletions(-) diff --git a/cf-keycrypt/Makefile.am b/cf-keycrypt/Makefile.am index 937d0f6220..fff519f0ef 100644 --- a/cf-keycrypt/Makefile.am +++ b/cf-keycrypt/Makefile.am @@ -23,7 +23,7 @@ # noinst_LTLIBRARIES = libcf-keycrypt.la -AM_CPPFLAGS = -I$(srcdir)/../libpromises -I$(srcdir)/../libutils \ +AM_CPPFLAGS = -I$(srcdir)/../libpromises -I$(srcdir)/../libntech/libutils \ -I$(srcdir)/../libcfnet \ $(OPENSSL_CPPFLAGS) \ $(PCRE_CPPFLAGS) \ diff --git a/cf-keycrypt/cf-keycrypt.c b/cf-keycrypt/cf-keycrypt.c index bd172de326..0e106cb138 100644 --- a/cf-keycrypt/cf-keycrypt.c +++ b/cf-keycrypt/cf-keycrypt.c @@ -43,8 +43,7 @@ */ -#include -#include +#include #include #include @@ -52,11 +51,8 @@ #include #include #include -#include -#include -#include +#include #include -#include #define BUFSIZE 1024 From d6eac6386ce14e9ef7bc99be1b48dc054a78ae06 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 5 Feb 2020 12:43:12 +0100 Subject: [PATCH 089/333] Address LGTM alerts in cf-keycrypt Except for an alert that we are using the command line argument directly as a path to open and read/write which we actually want to do. Also make helper functions static. (cherry picked from commit 65e490ad262044738da0ac5de00d7fd63495ca7d) --- cf-keycrypt/cf-keycrypt.c | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/cf-keycrypt/cf-keycrypt.c b/cf-keycrypt/cf-keycrypt.c index 0e106cb138..032c776978 100644 --- a/cf-keycrypt/cf-keycrypt.c +++ b/cf-keycrypt/cf-keycrypt.c @@ -1,5 +1,5 @@ /* - Copyright 2017 Northern.tech AS + Copyright 2020 Northern.tech AS This file is part of CFEngine 3 - written and maintained by Northern.tech AS. @@ -53,6 +53,7 @@ #include #include #include +#include #define BUFSIZE 1024 @@ -97,8 +98,9 @@ static const char *const HINTS[] = NULL }; -void *get_in_addr(struct sockaddr *sa) +static inline void *get_in_addr(struct sockaddr *sa) { + assert(sa != NULL); if (sa->sa_family == AF_INET) { return &(((struct sockaddr_in*)sa)->sin_addr); @@ -106,13 +108,13 @@ void *get_in_addr(struct sockaddr *sa) return &(((struct sockaddr_in6*)sa)->sin6_addr); } -int file_exist(const char *filename) +static inline bool file_exist(const char *filename) { struct stat buffer; return stat(filename, &buffer) == 0; } -char *get_host_pubkey(const char *host) +static char *get_host_pubkey(const char *host) { char *buffer = (char *) malloc(BUFSIZE *sizeof(char)); char hash[CF_HOSTKEY_STRING_SIZE]; @@ -168,9 +170,9 @@ char *get_host_pubkey(const char *host) return NULL; } -RSA *readseckey(const char *privkey_path) +static RSA *readseckey(const char *privkey_path) { - FILE *fp = fopen(privkey_path,"r"); + FILE *fp = safe_fopen(privkey_path,"r"); if (fp == NULL) { @@ -189,9 +191,9 @@ RSA *readseckey(const char *privkey_path) return privkey; } -RSA *readpubkey(const char *pubkey_path) +static RSA *readpubkey(const char *pubkey_path) { - FILE *fp = fopen(pubkey_path, "r"); + FILE *fp = safe_fopen(pubkey_path, "r"); if (fp == NULL) { @@ -209,7 +211,7 @@ RSA *readpubkey(const char *pubkey_path) return pubkey; } -long int rsa_encrypt( +static long rsa_encrypt( const char *pubkey_path, const char *input_path, const char *output_path) { int ks = 0; @@ -221,7 +223,7 @@ long int rsa_encrypt( return -1; } - FILE *input_file = fopen(input_path, "r"); + FILE *input_file = safe_fopen(input_path, "r"); if (input_file == NULL) { Log(LOG_LEVEL_ERR, "Could not open input file '%s'", input_path); @@ -229,7 +231,7 @@ long int rsa_encrypt( return -1; } - FILE *output_file = fopen(output_path, "w"); + FILE *output_file = safe_fopen(output_path, "w"); if (output_file == NULL) { Log(LOG_LEVEL_ERR, "Could not create output file '%s'", output_path); @@ -293,14 +295,14 @@ long int rsa_decrypt( return -1; } - FILE *input_file = fopen(input_path, "r"); + FILE *input_file = safe_fopen(input_path, "r"); if (input_file == NULL) { Log(LOG_LEVEL_ERR, "Cannot open input file '%s'", input_path); RSA_free(privkey); return -1; } - FILE *output_file = fopen(output_path, "w"); + FILE *output_file = safe_fopen(output_path, "w"); if (output_file == NULL){ Log(LOG_LEVEL_ERR, "Cannot open output file '%s'", output_path); fclose(input_file); From f5620443b42927c86c6fc3fe039ca7f6bf4676b8 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 5 Feb 2020 12:58:53 +0100 Subject: [PATCH 090/333] Increment the number of components in libenv/sysinfo.c cf-keycrypt is a new component (binary) so the array holding all binaries' names needs to be one element bigger. (cherry picked from commit 1ba429792a1e33158c09580a3f7f963cdab99a89) --- libenv/sysinfo.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libenv/sysinfo.c b/libenv/sysinfo.c index fcdad92e83..b88a7de4af 100644 --- a/libenv/sysinfo.c +++ b/libenv/sysinfo.c @@ -431,7 +431,7 @@ static void GetNameInfo3(EvalContext *ctx) long sz; #endif -#define COMPONENTS_SIZE 15 +#define COMPONENTS_SIZE 16 // This is used for $(sys.cf_agent), $(sys.cf_serverd) ... : char *components[COMPONENTS_SIZE] = { "cf-twin", "cf-agent", "cf-serverd", "cf-monitord", "cf-know", "cf-report", "cf-key", "cf-runagent", "cf-execd", "cf-hub", From 6e5047d7ea783b798f636ad9f9508239098f8dcc Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 5 Feb 2020 13:02:47 +0100 Subject: [PATCH 091/333] Fix the check for unexpected arguments in cf-keycrypt (cherry picked from commit 4f62b1e3f00cc206f3694d15bbd2ff979509e95c) --- cf-keycrypt/cf-keycrypt.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cf-keycrypt/cf-keycrypt.c b/cf-keycrypt/cf-keycrypt.c index 032c776978..f3ac63f88a 100644 --- a/cf-keycrypt/cf-keycrypt.c +++ b/cf-keycrypt/cf-keycrypt.c @@ -424,9 +424,9 @@ int main(int argc, char *argv[]) input_path = argv[optind]; optind += 1; - if (argv[optind] != NULL) + if (optind < argc) { - Log(LOG_LEVEL_ERR, "Unexpected non-option argument: '%s'",argv[optind]); + Log(LOG_LEVEL_ERR, "Unexpected non-option argument: '%s'", argv[optind]); exit(EXIT_FAILURE); } From fdf157e031671f86e6a9966db90434d69de1f6bd Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Mon, 10 Feb 2020 11:39:18 +0100 Subject: [PATCH 092/333] Replace file_exist() with a call to access() in cf-keycrypt No reason to have a function for what already exists. (cherry picked from commit 73fb49202c82eb9c126e56cf01e61f4e16ce7d56) --- cf-keycrypt/cf-keycrypt.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/cf-keycrypt/cf-keycrypt.c b/cf-keycrypt/cf-keycrypt.c index f3ac63f88a..d0235c499f 100644 --- a/cf-keycrypt/cf-keycrypt.c +++ b/cf-keycrypt/cf-keycrypt.c @@ -54,6 +54,7 @@ #include #include #include +#include #define BUFSIZE 1024 @@ -108,12 +109,6 @@ static inline void *get_in_addr(struct sockaddr *sa) return &(((struct sockaddr_in6*)sa)->sin6_addr); } -static inline bool file_exist(const char *filename) -{ - struct stat buffer; - return stat(filename, &buffer) == 0; -} - static char *get_host_pubkey(const char *host) { char *buffer = (char *) malloc(BUFSIZE *sizeof(char)); @@ -160,7 +155,7 @@ static char *get_host_pubkey(const char *host) snprintf( buffer, BUFSIZE * sizeof(char), "%s/ppkeys/root-%s.pub", WORKDIR, ipaddress); - if (file_exist(buffer)) + if (access(buffer, F_OK) == 0) { freeaddrinfo(result); return buffer; From 67a13a0a6e6fcc4a6044a0fb3080d546ea6a8965 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Tue, 11 Feb 2020 13:37:52 +0100 Subject: [PATCH 093/333] Fix and clean cf-keycrypt Fix logical errors in the code like always reading the public key even when decrypting and make the code more in accordance with our coding standards and conventions. Ticket: CFE-2613 Changelog: none (cherry picked from commit 5f021b52fa1590f67679ee1c7b1de1ab649d58d5) --- cf-keycrypt/cf-keycrypt.c | 251 +++++++++++++++++++------------------- 1 file changed, 124 insertions(+), 127 deletions(-) diff --git a/cf-keycrypt/cf-keycrypt.c b/cf-keycrypt/cf-keycrypt.c index d0235c499f..f829f2006e 100644 --- a/cf-keycrypt/cf-keycrypt.c +++ b/cf-keycrypt/cf-keycrypt.c @@ -53,11 +53,17 @@ #include #include #include +#include #include #include #define BUFSIZE 1024 +typedef enum { + HOST_RSA_KEY_PRIVATE, + HOST_RSA_KEY_PUBLIC, +} HostRSAKeyType; + static const char passphrase[] = "Cfengine passphrase"; //******************************************************************* @@ -99,7 +105,7 @@ static const char *const HINTS[] = NULL }; -static inline void *get_in_addr(struct sockaddr *sa) +static inline void *GetIPAddress(struct sockaddr *sa) { assert(sa != NULL); if (sa->sa_family == AF_INET) @@ -109,15 +115,22 @@ static inline void *get_in_addr(struct sockaddr *sa) return &(((struct sockaddr_in6*)sa)->sin6_addr); } -static char *get_host_pubkey(const char *host) +/** + * Get path of the RSA key for the given host. + */ +static char *GetHostRSAKey(const char *host, HostRSAKeyType type) { - char *buffer = (char *) malloc(BUFSIZE *sizeof(char)); - char hash[CF_HOSTKEY_STRING_SIZE]; - char ipaddress[BUFSIZE]; - struct addrinfo *result; - struct addrinfo *res; - bool found = false; + const char *key_ext = NULL; + if (type == HOST_RSA_KEY_PRIVATE) + { + key_ext = ".priv"; + } + else + { + key_ext = ".pub"; + } + struct addrinfo *result; int error = getaddrinfo(host, NULL, NULL, &result); if (error != 0) { @@ -126,35 +139,37 @@ static char *get_host_pubkey(const char *host) return NULL; } - for(res = result; res != NULL && !found; res = res->ai_next) + char *buffer = malloc(BUFSIZE); + char hash[CF_HOSTKEY_STRING_SIZE]; + char ipaddress[64]; + bool found = false; + for (struct addrinfo *res = result; !found && (res != NULL); res = res->ai_next) { - inet_ntop(res->ai_family, get_in_addr((struct sockaddr *)res->ai_addr), ipaddress, sizeof(ipaddress)); - if ((strcmp(ipaddress, "127.0.0.1") == 0) || (strcmp(ipaddress, "::1") == 0)) + inet_ntop(res->ai_family, + GetIPAddress((struct sockaddr *) res->ai_addr), + ipaddress, sizeof(ipaddress)); + if (StringStartsWith(ipaddress, "127.") || StringSafeEqual(ipaddress, "::1")) { found = true; - snprintf(buffer, BUFSIZE * sizeof(char), "%s/ppkeys/localhost.pub", WORKDIR); + snprintf(buffer, BUFSIZE, "%s/ppkeys/localhost%s", WORKDIR, key_ext); return buffer; } found = Address2Hostkey(hash, sizeof(hash), ipaddress); } if (found) { - snprintf(buffer, BUFSIZE * sizeof(char), "%s/ppkeys/root-%s.pub", WORKDIR, hash); + snprintf(buffer, BUFSIZE, "%s/ppkeys/root-%s%s", WORKDIR, hash, key_ext); freeaddrinfo(result); return buffer; } else { - for(res = result; res != NULL; res = res->ai_next) + for (struct addrinfo *res = result; res != NULL; res = res->ai_next) { - inet_ntop( - res->ai_family, - get_in_addr((struct sockaddr *)res->ai_addr), - ipaddress, - sizeof(ipaddress)); - snprintf( - buffer, BUFSIZE * sizeof(char), "%s/ppkeys/root-%s.pub", - WORKDIR, ipaddress); + inet_ntop(res->ai_family, + GetIPAddress((struct sockaddr *) res->ai_addr), + ipaddress, sizeof(ipaddress)); + snprintf(buffer, BUFSIZE, "%s/ppkeys/root-%s%s", WORKDIR, ipaddress, key_ext); if (access(buffer, F_OK) == 0) { freeaddrinfo(result); @@ -165,7 +180,7 @@ static char *get_host_pubkey(const char *host) return NULL; } -static RSA *readseckey(const char *privkey_path) +static RSA *ReadPrivateKey(const char *privkey_path) { FILE *fp = safe_fopen(privkey_path,"r"); @@ -174,8 +189,7 @@ static RSA *readseckey(const char *privkey_path) Log(LOG_LEVEL_ERR, "Could not open private key '%s'", privkey_path); return NULL; } - RSA *privkey = PEM_read_RSAPrivateKey(fp, (RSA **)NULL, NULL, - (void *) passphrase); + RSA *privkey = PEM_read_RSAPrivateKey(fp, (RSA **) NULL, NULL, (void *) passphrase); if (privkey == NULL) { unsigned long err = ERR_get_error(); @@ -186,7 +200,7 @@ static RSA *readseckey(const char *privkey_path) return privkey; } -static RSA *readpubkey(const char *pubkey_path) +static RSA *ReadPublicKey(const char *pubkey_path) { FILE *fp = safe_fopen(pubkey_path, "r"); @@ -196,26 +210,23 @@ static RSA *readpubkey(const char *pubkey_path) return NULL; } - RSA *pubkey = PEM_read_RSAPublicKey( - fp, NULL, NULL, (void *)passphrase); + RSA *pubkey = PEM_read_RSAPublicKey(fp, NULL, NULL, (void *) passphrase); if (pubkey == NULL) { - Log(LOG_LEVEL_ERR, "Could not read public key '%s'", pubkey_path); + unsigned long err = ERR_get_error(); + Log(LOG_LEVEL_ERR, "Could not read public key '%s': %s", + pubkey_path, ERR_reason_error_string(err)); } fclose(fp); return pubkey; } -static long rsa_encrypt( - const char *pubkey_path, const char *input_path, const char *output_path) +static bool RSAEncrypt(const char *pubkey_path, const char *input_path, const char *output_path) { - int ks = 0; - unsigned long len = 0, ciphsz = 0; - - RSA* pubkey = readpubkey(pubkey_path); + RSA* pubkey = ReadPublicKey(pubkey_path); if (pubkey == NULL) { - return -1; + return false; } FILE *input_file = safe_fopen(input_path, "r"); @@ -223,7 +234,7 @@ static long rsa_encrypt( { Log(LOG_LEVEL_ERR, "Could not open input file '%s'", input_path); RSA_free(pubkey); - return -1; + return false; } FILE *output_file = safe_fopen(output_path, "w"); @@ -232,132 +243,121 @@ static long rsa_encrypt( Log(LOG_LEVEL_ERR, "Could not create output file '%s'", output_path); fclose(input_file); RSA_free(pubkey); - return -1; + return false; } - ks = RSA_size(pubkey); - char tmp_ciphertext[ks], tmp_plaintext[ks]; + const int key_size = RSA_size(pubkey); + char tmp_ciphertext[key_size], tmp_plaintext[key_size]; srand(time(NULL)); - bool error_return = false; - while (!feof(input_file)) + unsigned long len = 0; + bool error = false; + while (!error && !feof(input_file)) { - memset(tmp_plaintext, '\0', ks); - memset(tmp_ciphertext, '\0', ks); - len = fread( - tmp_plaintext, 1, ks - RSA_PKCS1_PADDING_SIZE, input_file); + len = fread(tmp_plaintext, 1, key_size - RSA_PKCS1_PADDING_SIZE, input_file); if (len <= 0 || ferror(input_file) != 0) { Log(LOG_LEVEL_ERR, "Could not read file '%s'", input_path); - error_return = true; + error = true; break; } - unsigned long size = RSA_public_encrypt( - strlen(tmp_plaintext), tmp_plaintext, tmp_ciphertext, - pubkey, RSA_PKCS1_PADDING); + unsigned long size = RSA_public_encrypt(len, tmp_plaintext, tmp_ciphertext, + pubkey, RSA_PKCS1_PADDING); if (size == -1) { - Log(LOG_LEVEL_ERR, "%s", + Log(LOG_LEVEL_ERR, "Failed to encrypt data: %s", ERR_error_string(ERR_get_error(), NULL)); - error_return = true; + error = true; break; } - fwrite(tmp_ciphertext, sizeof(unsigned char), ks, output_file); + fwrite(tmp_ciphertext, 1, key_size, output_file); if (ferror(output_file) != 0) { Log(LOG_LEVEL_ERR, "Could not write file '%s'", output_path); - error_return = true; + error = true; break; } } + OPENSSL_cleanse(tmp_plaintext, key_size); fclose(input_file); fclose(output_file); RSA_free(pubkey); - if (error_return) - { - return -1; - } - return ciphsz; + return !error; } -long int rsa_decrypt( - const char *privkey_path, const char *input_path, const char *output_path) +static bool RSADecrypt(const char *privkey_path, const char *input_path, const char *output_path) { - unsigned long int plsz = 0, size = 0; - - RSA *privkey = (RSA *)readseckey(privkey_path); - if (privkey == NULL) { - return -1; + RSA *privkey = ReadPrivateKey(privkey_path); + if (privkey == NULL) + { + return false; } FILE *input_file = safe_fopen(input_path, "r"); - if (input_file == NULL) { + if (input_file == NULL) + { Log(LOG_LEVEL_ERR, "Cannot open input file '%s'", input_path); RSA_free(privkey); - return -1; + return false; } FILE *output_file = safe_fopen(output_path, "w"); - if (output_file == NULL){ + if (output_file == NULL) + { Log(LOG_LEVEL_ERR, "Cannot open output file '%s'", output_path); fclose(input_file); RSA_free(privkey); - return -1; + return false; } - unsigned long int ks = RSA_size(privkey); - char tmp_ciphertext[ks], tmp_plaintext[ks]; + const int key_size = RSA_size(privkey); + char tmp_ciphertext[key_size], tmp_plaintext[key_size]; - bool error_return = false; - while (!feof(input_file)) + bool error = false; + while (!error && !feof(input_file)) { - memset(tmp_ciphertext, '\0', ks); - memset(tmp_plaintext, '\0', ks); - unsigned long int len = fread( - tmp_ciphertext, 1, ks, input_file); + unsigned long int len = fread(tmp_ciphertext, 1, key_size, input_file); if (ferror(input_file) != 0) { Log(LOG_LEVEL_ERR, "Could not read from '%s'", input_path); - error_return = true; - break; + error = true; } - if (len > 0){ - size = RSA_private_decrypt( - ks, tmp_ciphertext, tmp_plaintext, privkey, RSA_PKCS1_PADDING); + else if (len > 0) + { + int size = RSA_private_decrypt(key_size, tmp_ciphertext, tmp_plaintext, + privkey, RSA_PKCS1_PADDING); if (size == -1) { - Log(LOG_LEVEL_ERR, "%s", + Log(LOG_LEVEL_ERR, "Failed to decrypt data: %s", ERR_error_string(ERR_get_error(), NULL)); - error_return = true; - break; + error = true; + } + else + { + fwrite(tmp_plaintext, 1, size, output_file); + if (ferror(output_file) != 0) + { + Log(LOG_LEVEL_ERR, "Could not write to '%s'", output_path); + error = true; + } } - } - - fwrite(tmp_plaintext, 1, strlen(tmp_plaintext), output_file); - if (ferror(output_file) != 0) - { - Log(LOG_LEVEL_ERR, "Could not write to '%s'", output_path); - error_return = true; - break; } } + OPENSSL_cleanse(tmp_plaintext, key_size); fclose(input_file); fclose(output_file); RSA_free(privkey); - if (error_return) - { - return -1; - } - return plsz; + return !error; } -void cf_keycrypt_help() + +static void CFKeyCryptHelp() { Writer *w = FileWriter(stdout); WriterWriteHelp(w, "cf-keycrypt", OPTIONS, HINTS, false, NULL); FileWriterDetach(w); } -void cf_keycrypt_man() +void CFKeyCryptMan() { Writer *out = FileWriter(stdout); ManPageWrite(out, "cf-keycrypt", time(NULL), @@ -371,28 +371,29 @@ int main(int argc, char *argv[]) { if (argc == 1) { - cf_keycrypt_help(); + CFKeyCryptHelp(); exit(EXIT_FAILURE); } opterr = 0; - char *key = NULL; + char *key_path = NULL; char *input_path = NULL; char *output_path = NULL; char *host = NULL; bool encrypt = false; bool decrypt = false; + int c = 0; while ((c = getopt_long(argc, argv, "hMedk:o:H:", OPTIONS, NULL)) != -1) { switch (c) { case 'h': - cf_keycrypt_help(); + CFKeyCryptHelp(); exit(EXIT_SUCCESS); break; case 'M': - cf_keycrypt_man(); + CFKeyCryptMan(); exit(EXIT_SUCCESS); break; case 'e': @@ -402,7 +403,7 @@ int main(int argc, char *argv[]) decrypt = true; break; case 'k': - key = optarg; + key_path = optarg; break; case 'o': output_path = optarg; @@ -412,7 +413,7 @@ int main(int argc, char *argv[]) break; default: Log(LOG_LEVEL_ERR, "Unknown option '-%c'", optopt); - cf_keycrypt_help(); + CFKeyCryptHelp(); exit(EXIT_FAILURE); } } @@ -426,17 +427,19 @@ int main(int argc, char *argv[]) } // Check for argument errors: - if (encrypt == decrypt) + if ((encrypt && decrypt) || (!encrypt && !decrypt)) { Log(LOG_LEVEL_ERR, "Must specify either encrypt or decrypt (and not both)"); exit(EXIT_FAILURE); } - if (host != NULL && key != NULL) + if ((host != NULL) && (key_path != NULL)) { - Log(LOG_LEVEL_ERR, "--host/-H is used to specify a public key and cannot be used with --key/-k"); + Log(LOG_LEVEL_ERR, + "--host/-H is used to specify a public key and cannot be used with --key/-k"); exit(EXIT_FAILURE); } + if (input_path == NULL) { Log(LOG_LEVEL_ERR, "No input file specified (Use -h for help)"); @@ -448,31 +451,25 @@ int main(int argc, char *argv[]) exit(EXIT_FAILURE); } - // Resolve host to public key: CryptoInitialize(); - if (host) + if (host != NULL) { - key = get_host_pubkey(host); - if (!key) + HostRSAKeyType key_type = encrypt ? HOST_RSA_KEY_PUBLIC : HOST_RSA_KEY_PRIVATE; + key_path = GetHostRSAKey(host, key_type); + if (!key_path) { - Log(LOG_LEVEL_ERR, "Unable to locate public key for host '%s'", host); + Log(LOG_LEVEL_ERR, "Unable to locate key for host '%s'", host); exit(EXIT_FAILURE); } } - - // Additional error checking: - if (key == NULL) - { - Log(LOG_LEVEL_ERR, "No key specified (Use -h for help)"); - exit(EXIT_FAILURE); - } + assert (key_path != NULL); // Encrypt or decrypt - int size = 0; + bool success; if (encrypt) { - size = rsa_encrypt(key, input_path, output_path); - if (size < 0) + success = RSAEncrypt(key_path, input_path, output_path); + if (!success) { Log(LOG_LEVEL_ERR, "Encryption failed"); exit(EXIT_FAILURE); @@ -480,8 +477,8 @@ int main(int argc, char *argv[]) } else if (decrypt) { - size = rsa_decrypt(key, input_path, output_path); - if (size < 0) + success = RSADecrypt(key_path, input_path, output_path); + if (!success) { Log(LOG_LEVEL_ERR, "Decryption failed"); exit(EXIT_FAILURE); From 220fa32197422491a42b18000a5db42e12708243 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Tue, 11 Feb 2020 13:43:43 +0100 Subject: [PATCH 094/333] Add cf-keycrypt/Makefile to .gitignore It's produced by autotools. (cherry picked from commit 6cb68557dde9519536bed0c666aba0df2b82c03a) --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 6baaf6bb33..a79387dd0f 100644 --- a/.gitignore +++ b/.gitignore @@ -35,6 +35,7 @@ stamp-h1 /cf-check/Makefile /cf-execd/Makefile /cf-key/Makefile +/cf-keycrypt/Makefile /cf-monitord/Makefile /cf-net/Makefile /cf-promises/Makefile From ab1c75673a4a332c709d4cc67b8475f892330a2b Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Tue, 11 Feb 2020 15:33:18 +0100 Subject: [PATCH 095/333] Default to localhost.priv key for decryption in cf-keycrypt Usually the private key is only available for the local host. And that's where the decryption takes place and makes sense. Ticket: CFE-2613 Changelog: none (cherry picked from commit c7f2ee53ad1a958267c98d9a40de4b20ca258d0d) --- cf-keycrypt/cf-keycrypt.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/cf-keycrypt/cf-keycrypt.c b/cf-keycrypt/cf-keycrypt.c index f829f2006e..e814329436 100644 --- a/cf-keycrypt/cf-keycrypt.c +++ b/cf-keycrypt/cf-keycrypt.c @@ -433,6 +433,15 @@ int main(int argc, char *argv[]) exit(EXIT_FAILURE); } + if (decrypt && (host == NULL) && (key_path == NULL)) + { + /* Decryption requires a private key which is usually only available for + * the local host. Let's just default to localhost if no other specific + * host/key is given for decryption. */ + Log(LOG_LEVEL_VERBOSE, "Using the localhost private key for decryption"); + host = "localhost"; + } + if ((host != NULL) && (key_path != NULL)) { Log(LOG_LEVEL_ERR, From 84581d2eddfa203a4608ef78432796a674c2f8d0 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 12 Feb 2020 17:12:27 +0100 Subject: [PATCH 096/333] Do proper encryption using RSA+AES in cf-keycrypt Encrypting just with the RSA key is slow and also potentially insecure because the security of the encryption with RSA degrades with the data being bigger than the RSA key itself. The standard way of encryption using RSA is to generate a AES key, encrypt the key with the RSA key and then encrypt the actual payload with the generated AES key using symmetric encryption. Of course, the encrypted AES key then has to be stored next to the encrypted payload for decryption. The same applies to a randomly generated initialization vector (IV) needed by the AES cipher. Ticket: CFE-2613 Changelog: none (cherry picked from commit f27b70a3b65ef157a4dcbe200459090e7503af31) --- cf-keycrypt/cf-keycrypt.c | 319 ++++++++++++++++++---- tests/acceptance/27_cf-keycrypt/encrypted | Bin 256 -> 318 bytes 2 files changed, 271 insertions(+), 48 deletions(-) diff --git a/cf-keycrypt/cf-keycrypt.c b/cf-keycrypt/cf-keycrypt.c index e814329436..b5f5376508 100644 --- a/cf-keycrypt/cf-keycrypt.c +++ b/cf-keycrypt/cf-keycrypt.c @@ -59,11 +59,23 @@ #define BUFSIZE 1024 +#define MAX_HEADER_LEN 256 /* "Key[126]: Value[128]" */ +#define MAX_HEADER_KEY_LEN 126 +#define MAX_HEADER_VAL_LEN 128 + +#define MAX_HEADER_KEY_LEN_STR STRING(MAX_HEADER_KEY_LEN) +#define MAX_HEADER_VAL_LEN_STR STRING(MAX_HEADER_VAL_LEN) + typedef enum { HOST_RSA_KEY_PRIVATE, HOST_RSA_KEY_PUBLIC, } HostRSAKeyType; +/* see README.md for details about the format */ +typedef enum { + CF_KEYCRYPT_FORMAT_V_1_0, +} CFKeyCryptFormatVersion; + static const char passphrase[] = "Cfengine passphrase"; //******************************************************************* @@ -241,48 +253,190 @@ static bool RSAEncrypt(const char *pubkey_path, const char *input_path, const ch if (output_file == NULL) { Log(LOG_LEVEL_ERR, "Could not create output file '%s'", output_path); + RSA_free(pubkey); fclose(input_file); + return false; + } + + EVP_PKEY *evp_key = EVP_PKEY_new(); + if (EVP_PKEY_set1_RSA(evp_key, pubkey) == 0) + { + Log(LOG_LEVEL_ERR, "Failed to initialize encryption context"); RSA_free(pubkey); + fclose(input_file); + fclose(output_file); return false; } - const int key_size = RSA_size(pubkey); - char tmp_ciphertext[key_size], tmp_plaintext[key_size]; + bool success = true; + + const EVP_CIPHER *cipher = EVP_aes_256_cbc(); + EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new(); + const int key_size = EVP_PKEY_size(evp_key); + + unsigned char *ek = xmalloc(key_size); + int ek_size; + + const int iv_size = EVP_CIPHER_iv_length(cipher); + unsigned char iv[iv_size]; + + const int block_size = EVP_CIPHER_block_size(cipher); + char plaintext[block_size], ciphertext[2 * block_size]; + int ct_len; + + int ret = EVP_SealInit(ctx, cipher, &ek, &ek_size, iv, &evp_key, 1); + if (ret == 0) + { + Log(LOG_LEVEL_ERR, "Failed to initialize encryption context"); + success = false; + goto cleanup; + } + + const char *headers = "Version: 1.0\n" + "\n"; + size_t headers_len = strlen(headers); + ssize_t n_written = FullWrite(fileno(output_file), headers, headers_len); + if (n_written != headers_len) + { + Log(LOG_LEVEL_ERR, "Failed to write headers to the output file '%s'", output_path); + success = false; + goto cleanup; + } - srand(time(NULL)); - unsigned long len = 0; - bool error = false; - while (!error && !feof(input_file)) + n_written = FullWrite(fileno(output_file), iv, iv_size); + if (n_written != iv_size) + { + Log(LOG_LEVEL_ERR, "Failed to write IV to the output file '%s'", output_path); + success = false; + goto cleanup; + } + n_written = FullWrite(fileno(output_file), ek, ek_size); + if (n_written != ek_size) { - len = fread(tmp_plaintext, 1, key_size - RSA_PKCS1_PADDING_SIZE, input_file); - if (len <= 0 || ferror(input_file) != 0) + Log(LOG_LEVEL_ERR, "Failed to write key to the output file '%s'", output_path); + success = false; + goto cleanup; + } + + while (success && !feof(input_file)) + { + ssize_t n_read = ReadFileStreamToBuffer(input_file, block_size, plaintext); + if (n_read == FILE_ERROR_READ) { Log(LOG_LEVEL_ERR, "Could not read file '%s'", input_path); - error = true; + success = false; break; } - unsigned long size = RSA_public_encrypt(len, tmp_plaintext, tmp_ciphertext, - pubkey, RSA_PKCS1_PADDING); - if (size == -1) + ret = EVP_SealUpdate(ctx, ciphertext, &ct_len, plaintext, n_read); + if (ret == 0) { Log(LOG_LEVEL_ERR, "Failed to encrypt data: %s", ERR_error_string(ERR_get_error(), NULL)); - error = true; + success = false; break; } - fwrite(tmp_ciphertext, 1, key_size, output_file); - if (ferror(output_file) != 0) + n_written = FullWrite(fileno(output_file), ciphertext, ct_len); + if (n_written < 0) { Log(LOG_LEVEL_ERR, "Could not write file '%s'", output_path); - error = true; + success = false; break; } } - OPENSSL_cleanse(tmp_plaintext, key_size); + ret = EVP_SealFinal(ctx, ciphertext, &ct_len); + if (ret == 0) + { + Log(LOG_LEVEL_ERR, "Failed to encrypt data: %s", + ERR_error_string(ERR_get_error(), NULL)); + success = false; + } + if (ct_len > 0) + { + n_written = FullWrite(fileno(output_file), ciphertext, ct_len); + if (n_written < 0) + { + Log(LOG_LEVEL_ERR, "Could not write file '%s'", output_path); + success = false; + } + } + OPENSSL_cleanse(plaintext, block_size); + + cleanup: + RSA_free(pubkey); fclose(input_file); fclose(output_file); - RSA_free(pubkey); - return !error; + EVP_PKEY_free(evp_key); + EVP_CIPHER_CTX_free(ctx); + free(ek); + return success; +} + +static bool CheckHeader(const char *key, const char *value) +{ + if (StringSafeEqual(key, "Version")) + { + if (!StringSafeEqual(value, "1.0")) + { + Log(LOG_LEVEL_ERR, "Unsupported file format version: '%s'", value); + return false; + } + else + { + return true; + } + } + else + { + Log(LOG_LEVEL_ERR, "Unsupported header: '%s'", key); + return false; + } +} + +static bool ParseHeaders(FILE *input_file) +{ + char key[MAX_HEADER_KEY_LEN + 1]; + char value[MAX_HEADER_VAL_LEN + 1]; + int ret = fscanf(input_file, + "%"MAX_HEADER_KEY_LEN_STR"[^:]: %"MAX_HEADER_VAL_LEN_STR"[^\n]", + key, value); + bool version_specified = false; + while (ret == 2) + { + if (!CheckHeader(key, value)) + { + return false; + } + version_specified = (version_specified || (StringSafeEqual(key, "Version"))); + + /* each header should be terminated by a newline */ + char next = fgetc(input_file); + if (next == '\n') + { + next = fgetc(input_file); + if (next == '\n') + { + /* headers are supposed to be separated by blank line */ + if (!version_specified) + { + Log(LOG_LEVEL_ERR, "File format version not specified"); + } + return version_specified; + } + else + { + /* keep trying */ + ungetc(next, input_file); + } + } + else + { + return false; + } + ret = fscanf(input_file, + "%"MAX_HEADER_KEY_LEN_STR"[^:]: %"MAX_HEADER_VAL_LEN_STR"[^\n]\n", + key, value); + } + return false; } static bool RSADecrypt(const char *privkey_path, const char *input_path, const char *output_path) @@ -310,44 +464,113 @@ static bool RSADecrypt(const char *privkey_path, const char *input_path, const c return false; } - const int key_size = RSA_size(privkey); - char tmp_ciphertext[key_size], tmp_plaintext[key_size]; + EVP_PKEY *evp_key = EVP_PKEY_new(); + if (EVP_PKEY_set1_RSA(evp_key, privkey) == 0) + { + Log(LOG_LEVEL_ERR, "Failed to initialize decryption context"); + RSA_free(privkey); + fclose(input_file); + fclose(output_file); + return false; + } + + bool success = true; + + if (!ParseHeaders(input_file)) + { + Log(LOG_LEVEL_ERR, "Failed to parse headers from '%s'", input_path); + RSA_free(privkey); + fclose(input_file); + fclose(output_file); + return false; + } + + const EVP_CIPHER *cipher = EVP_aes_256_cbc(); + EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new(); + + const int iv_size = EVP_CIPHER_iv_length(cipher); + unsigned char iv[iv_size]; + + const int key_size = EVP_PKEY_size(evp_key); + unsigned char ek[key_size]; + + const int block_size = EVP_CIPHER_block_size(cipher); + + char plaintext[block_size], ciphertext[2 * block_size]; + int pt_len; + + ssize_t n_read = ReadFileStreamToBuffer(input_file, iv_size, iv); + if (n_read != iv_size) + { + Log(LOG_LEVEL_ERR, "Failed to read the IV from '%s'", input_path); + goto cleanup; + } + n_read = ReadFileStreamToBuffer(input_file, key_size, ek); + if (n_read != key_size) + { + Log(LOG_LEVEL_ERR, "Failed to read the key from '%s'", input_path); + goto cleanup; + } - bool error = false; - while (!error && !feof(input_file)) + int ret = EVP_OpenInit(ctx, cipher, ek, key_size, iv, evp_key); + if (ret == 0) { - unsigned long int len = fread(tmp_ciphertext, 1, key_size, input_file); - if (ferror(input_file) != 0) + Log(LOG_LEVEL_ERR, "Failed to initialize decryption context"); + success = false; + goto cleanup; + } + + ssize_t n_written; + while (success && !feof(input_file)) + { + n_read = ReadFileStreamToBuffer(input_file, block_size, ciphertext); + if (n_read == FILE_ERROR_READ) { - Log(LOG_LEVEL_ERR, "Could not read from '%s'", input_path); - error = true; + Log(LOG_LEVEL_ERR, "Could not read file '%s'", input_path); + success = false; + break; } - else if (len > 0) + ret = EVP_OpenUpdate(ctx, plaintext, &pt_len, ciphertext, n_read); + if (ret == 0) { - int size = RSA_private_decrypt(key_size, tmp_ciphertext, tmp_plaintext, - privkey, RSA_PKCS1_PADDING); - if (size == -1) - { - Log(LOG_LEVEL_ERR, "Failed to decrypt data: %s", - ERR_error_string(ERR_get_error(), NULL)); - error = true; - } - else - { - fwrite(tmp_plaintext, 1, size, output_file); - if (ferror(output_file) != 0) - { - Log(LOG_LEVEL_ERR, "Could not write to '%s'", output_path); - error = true; - } - } + Log(LOG_LEVEL_ERR, "Failed to encrypt data: %s", + ERR_error_string(ERR_get_error(), NULL)); + success = false; + break; + } + n_written = FullWrite(fileno(output_file), plaintext, pt_len); + if (n_written < 0) + { + Log(LOG_LEVEL_ERR, "Could not write file '%s'", output_path); + success = false; + break; + } + } + ret = EVP_OpenFinal(ctx, plaintext, &pt_len); + if (ret == 0) + { + Log(LOG_LEVEL_ERR, "Failed to encrypt data: %s", + ERR_error_string(ERR_get_error(), NULL)); + success = false; + } + if (pt_len > 0) + { + n_written = FullWrite(fileno(output_file), plaintext, pt_len); + if (n_written < 0) + { + Log(LOG_LEVEL_ERR, "Could not write file '%s'", output_path); + success = false; } } - OPENSSL_cleanse(tmp_plaintext, key_size); + OPENSSL_cleanse(plaintext, block_size); + + cleanup: + RSA_free(privkey); fclose(input_file); fclose(output_file); - RSA_free(privkey); - return !error; + EVP_PKEY_free(evp_key); + EVP_CIPHER_CTX_free(ctx); + return success; } static void CFKeyCryptHelp() diff --git a/tests/acceptance/27_cf-keycrypt/encrypted b/tests/acceptance/27_cf-keycrypt/encrypted index 6f1ab1ff185197c03d839820313c36266b8b22d7..f710d5a094e54670251c329aff4d4598f4fd0880 100644 GIT binary patch literal 318 zcmV-E0m1%OWpZFMQnIJ$yqf|hoRXEvfr)BnZ;CzGb#kTvs znx3aPSyg>WYdp%G9DHV!AuQZA^5#K_q8>%F#zs}ShqC^3&6QsgOiqS}D+zg-eegXM zelMLdWfb2M4f6ND;$coL#BRj>n};|5)og1k;Ywij<)#!oupXj?`$t7vFCr|zjN8Mo zM^QD{3BHrwcU3IP6v$`}+MGFl?vfz!&zwi_;XMpm#LA^9(ZaZaYa{=rAl%wQ1>eCVw8zEPw@C$!s80g-hdS?ZG{sse8j6i2$1f9Q3z2MAbqdmzzI z%S1%5^rdK<19j#frDiFk2^F$#^xz}vioP$XdB1N4K0fs5Z*f-JfseRQ&@FoqF8@c1 Gop&TAz Date: Thu, 13 Feb 2020 11:40:57 +0100 Subject: [PATCH 097/333] Add a README.md file for cf-keycrypt Explaining the purpose of the tool together with the file format it uses. Ticket: CFE-2613 Changelog: none (cherry picked from commit 134252f8db73797de02d5470cc1d12fe80ce6237) --- cf-keycrypt/README.md | 44 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100644 cf-keycrypt/README.md diff --git a/cf-keycrypt/README.md b/cf-keycrypt/README.md new file mode 100644 index 0000000000..831c07f33a --- /dev/null +++ b/cf-keycrypt/README.md @@ -0,0 +1,44 @@ +# cf-keycrypt + +*cf-keycrypt* is a utility for encrypting sensitive data for use on +CFEngine-managed hosts. It is using the existing CFEngine key pairs for strong +cryptography based on the combination of RSA and AES ciphers. + +## File format + +The file format used by *cf-keycrypt* has the following schema: + +``` + ------------ + | Headers | + ------------ + | AES IV | + ------------ + | AES key | + ------------ + | data | + ------------ +``` + +The header format is similar to HTTP headers -- colon-separated key-value pairs +each on one line: + +`Key: Value\n` + +The header section is terminated by a blank line. + +Supported headers are: + + * `Version` (required) -- version of the file format to allow backwards + compatibility + +The AES initialization vector is 16 bytes long (256 bits) and serves the purpose +of the seed for the CBC (Cipher Block Chain) mode of operation of the AES +cipher. + +The AES key is a randomly generated AES key encrypted by the specific RSA public +key and is as long as the RSA public key, currently 256 bytes (2048 bits). + +The future versions of *cf-keycrypt* are expected to support more headers, +multiple keys (encryption for multiple hosts in a single file) and varying key +sizes. \ No newline at end of file From 98798d00af6c5ceafa0b50e779e341b90acd6351 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 13 Feb 2020 11:52:32 +0100 Subject: [PATCH 098/333] Support '-' as input/output file in cf-keycrypt So that it can read/write from/to standard input/output. Ticket: CFE-2613 Changelog: none (cherry picked from commit fd59433f2f2b1932ff94ac262ea48fda5e88338b) --- cf-keycrypt/cf-keycrypt.c | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/cf-keycrypt/cf-keycrypt.c b/cf-keycrypt/cf-keycrypt.c index b5f5376508..f820888803 100644 --- a/cf-keycrypt/cf-keycrypt.c +++ b/cf-keycrypt/cf-keycrypt.c @@ -233,6 +233,24 @@ static RSA *ReadPublicKey(const char *pubkey_path) return pubkey; } +static FILE *OpenInputOutput(const char *path, const char *mode) +{ + assert(path != NULL); + assert(mode != NULL); + if (StringSafeEqual(path, "-")) + { + if (*mode == 'r') + { + return stdin; + } + else + { + return stdout; + } + } + return safe_fopen(path, mode); +} + static bool RSAEncrypt(const char *pubkey_path, const char *input_path, const char *output_path) { RSA* pubkey = ReadPublicKey(pubkey_path); @@ -241,7 +259,7 @@ static bool RSAEncrypt(const char *pubkey_path, const char *input_path, const ch return false; } - FILE *input_file = safe_fopen(input_path, "r"); + FILE *input_file = OpenInputOutput(input_path, "r"); if (input_file == NULL) { Log(LOG_LEVEL_ERR, "Could not open input file '%s'", input_path); @@ -249,7 +267,7 @@ static bool RSAEncrypt(const char *pubkey_path, const char *input_path, const ch return false; } - FILE *output_file = safe_fopen(output_path, "w"); + FILE *output_file = OpenInputOutput(output_path, "w"); if (output_file == NULL) { Log(LOG_LEVEL_ERR, "Could not create output file '%s'", output_path); @@ -447,7 +465,7 @@ static bool RSADecrypt(const char *privkey_path, const char *input_path, const c return false; } - FILE *input_file = safe_fopen(input_path, "r"); + FILE *input_file = OpenInputOutput(input_path, "r"); if (input_file == NULL) { Log(LOG_LEVEL_ERR, "Cannot open input file '%s'", input_path); @@ -455,7 +473,7 @@ static bool RSADecrypt(const char *privkey_path, const char *input_path, const c return false; } - FILE *output_file = safe_fopen(output_path, "w"); + FILE *output_file = OpenInputOutput(output_path, "w"); if (output_file == NULL) { Log(LOG_LEVEL_ERR, "Cannot open output file '%s'", output_path); From c0244cde3c3dc7d0c543bd0edc9f51afe6b86e7c Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 13 Feb 2020 11:57:45 +0100 Subject: [PATCH 099/333] Adapt the cf-keycrypt example to changes done in cf-keycrypt The output format is different now. Ticket: CFE-2613 Changelog: none (cherry picked from commit 1600c9c61d454f775be1b1efa31c11fed0e26b79) --- examples/cf-keycrypt.cf | 7 ++++--- examples/cf-keycrypt.cf.cfcrypt | 6 ++++-- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/examples/cf-keycrypt.cf b/examples/cf-keycrypt.cf index b8b17a3b40..1dd13fb092 100644 --- a/examples/cf-keycrypt.cf +++ b/examples/cf-keycrypt.cf @@ -56,9 +56,10 @@ body printfile cat(file) #+begin_src static_example_output #@ ``` #@ R: Encrypted file content: -#@ R: ��Q�������VO�["N߀ �1�}΢n�#��h�J�2�}5���N1m�n�,��E��\����٣��g>��oqVÛ��P�-{.�G��w$ +#@ R: Version: 1.0 +#@ R: +#@ R: ���V�cv�#�P��, ��-O�8æ—¼[i����pò¢¢¦ï¿½Q� +#@ R: Φ&l�x'�#j���qQ����[�F�1����v�Q��ˮ�J'�թ�|^HG%)�`&�����~k�$wd]"�%4X\(Q�~�O����s�A~���/��:�" gi�Rn&Ù�E^���߬3��M�ə�%2s�SB��b3���K4wm����o�B�:P��O�#��1�t8��`�@��j/��+����j��g஡����Z�D�iJ��͞j��8ĉ�ag�9vz?+�暢��Sî•Ÿo��.Org]�"+�S����_HÑ¢=_O% #@ R: Decrypted content: #@ Super secret message is here #@ ``` diff --git a/examples/cf-keycrypt.cf.cfcrypt b/examples/cf-keycrypt.cf.cfcrypt index 6f1ab1ff18..273c496dd6 100644 --- a/examples/cf-keycrypt.cf.cfcrypt +++ b/examples/cf-keycrypt.cf.cfcrypt @@ -1,2 +1,4 @@ -®Q–ßíéîù’VO["N߀ 1ü}΢n„#©ìh‡J¸2Ê}5ƒ›ôN1múnµ,­áE=§Ð\Õó ‡Ù£­ƒg>ôèoqVÛ¸PÐ-{.ÿG‹w$ \ No newline at end of file +Version: 1.0 + +ÿþÛV‡cvŸ#ªPîô, ¥„-OŠ8æ—¼[iþþêñpò¢¢¦ð•QÞ +Φ&lúx'Ô#jÖqQ– ¦à®[·F”1óÓéªÎvÙQù×Ë®ŠJ'ÝÕ©“|^HG%)ê`&Ÿùíšè~kß$wd]"º%4X\(Qª~¢O°ðîà§sÕA~¹ÿ/ÄÅ:¸" giƒRn&Ù¢E^å¯ñÌ߬3àšMÆÉ™§%2sÏSBæÜb3Œð¨†çK4wm©„ûÏo›Bà:Pš¬O—#¼Þ1Ãt8¸`¤@¾Öj/‰¶+®‡üj ¬g஡ÐðõûZÍD¦iJ¨ÝPÍžj³œ8ĉ„ag¤9vz?+œæš¢ÎþSî•Ÿo¸€.Org]ã"+¹S¡°“_HÑ¢=_O% \ No newline at end of file From c12f1a7c585c3285b14c0fa3233a51fdb5a9b3d9 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 3 Apr 2020 12:37:20 +0200 Subject: [PATCH 100/333] Add multi-key/multi-host support to cf-keycrypt Encrypting secrets for hosts individually is error-prone and results in many files. A better approach is to encrypt the payload for multiple hosts (keys) at once and store the result in a file the format of which allows the individual hosts to decrypt the payload back using their keys. This commit introduces a new 'Encrypted-for' header for the encrypted file format which tells which keys the payload was encrypted for and also what is the ordering of the segments containing the encrypted symmetric key. Thus the format now looks as follows: ------------ | Headers | ------------ | AES IV | ------------ | AES key | ------------ | AES key | ------------ | ... | ------------ | AES key | ------------ | data | ------------ Where each 'AES key' block is the same randomly generated AES key encrypted by different RSA (host) key. The host then needs to identify which 'AES key' block to decrypt and use as the key to decrypt the payload. Ticket: CFE-3271 Changelog: None (cherry picked from commit e34f8819039b458b62ec5b6c7f6fd8caaba247df) --- cf-keycrypt/README.md | 28 +- cf-keycrypt/cf-keycrypt.c | 384 +++++++++++++++++----- tests/acceptance/27_cf-keycrypt/encrypted | Bin 318 -> 370 bytes 3 files changed, 314 insertions(+), 98 deletions(-) diff --git a/cf-keycrypt/README.md b/cf-keycrypt/README.md index 831c07f33a..69246e3aa2 100644 --- a/cf-keycrypt/README.md +++ b/cf-keycrypt/README.md @@ -16,10 +16,19 @@ The file format used by *cf-keycrypt* has the following schema: ------------ | AES key | ------------ + | AES key | + ------------ + | ... | + ------------ + | AES key | + ------------ | data | ------------ ``` +Where each 'AES key' block is the same randomly generated AES key +encrypted by different RSA (host) key. + The header format is similar to HTTP headers -- colon-separated key-value pairs each on one line: @@ -32,13 +41,16 @@ Supported headers are: * `Version` (required) -- version of the file format to allow backwards compatibility -The AES initialization vector is 16 bytes long (256 bits) and serves the purpose -of the seed for the CBC (Cipher Block Chain) mode of operation of the AES -cipher. + * `Encrypted-for` (required) -- which (host) keys the payload was encrypted for, one header (line) + per host + +The AES initialization vector (IV) is 16 bytes long (256 bits) and serves the +purpose of the seed for the CBC (Cipher Block Chain) mode of operation of the +AES cipher. -The AES key is a randomly generated AES key encrypted by the specific RSA public -key and is as long as the RSA public key, currently 256 bytes (2048 bits). +The `AES key` is a randomly generated AES key encrypted by RSA public keys and +each encrypted `AES key` block is as long as the RSA public key, currently 256 +bytes (2048 bits). -The future versions of *cf-keycrypt* are expected to support more headers, -multiple keys (encryption for multiple hosts in a single file) and varying key -sizes. \ No newline at end of file +The future versions of *cf-keycrypt* are expected to support more headers and +varying key sizes. diff --git a/cf-keycrypt/cf-keycrypt.c b/cf-keycrypt/cf-keycrypt.c index f820888803..27f69251ea 100644 --- a/cf-keycrypt/cf-keycrypt.c +++ b/cf-keycrypt/cf-keycrypt.c @@ -48,6 +48,7 @@ #include #include +#include /* GenericAgentSetDefaultDigest */ #include #include #include @@ -55,6 +56,8 @@ #include #include #include +#include +#include #include #define BUFSIZE 1024 @@ -63,9 +66,6 @@ #define MAX_HEADER_KEY_LEN 126 #define MAX_HEADER_VAL_LEN 128 -#define MAX_HEADER_KEY_LEN_STR STRING(MAX_HEADER_KEY_LEN) -#define MAX_HEADER_VAL_LEN_STR STRING(MAX_HEADER_VAL_LEN) - typedef enum { HOST_RSA_KEY_PRIVATE, HOST_RSA_KEY_PUBLIC, @@ -251,19 +251,14 @@ static FILE *OpenInputOutput(const char *path, const char *mode) return safe_fopen(path, mode); } -static bool RSAEncrypt(const char *pubkey_path, const char *input_path, const char *output_path) +static bool RSAEncrypt(Seq *rsa_keys, const char *input_path, const char *output_path) { - RSA* pubkey = ReadPublicKey(pubkey_path); - if (pubkey == NULL) - { - return false; - } + assert((rsa_keys != NULL) && (SeqLength(rsa_keys) > 0)); FILE *input_file = OpenInputOutput(input_path, "r"); if (input_file == NULL) { Log(LOG_LEVEL_ERR, "Could not open input file '%s'", input_path); - RSA_free(pubkey); return false; } @@ -271,29 +266,41 @@ static bool RSAEncrypt(const char *pubkey_path, const char *input_path, const ch if (output_file == NULL) { Log(LOG_LEVEL_ERR, "Could not create output file '%s'", output_path); - RSA_free(pubkey); fclose(input_file); return false; } - EVP_PKEY *evp_key = EVP_PKEY_new(); - if (EVP_PKEY_set1_RSA(evp_key, pubkey) == 0) + const size_t n_keys = SeqLength(rsa_keys); + Seq *evp_keys = SeqNew(n_keys, EVP_PKEY_free); + for (size_t i = 0; i < n_keys; i++) { - Log(LOG_LEVEL_ERR, "Failed to initialize encryption context"); - RSA_free(pubkey); - fclose(input_file); - fclose(output_file); - return false; + RSA *rsa_key = SeqAt(rsa_keys, i); + EVP_PKEY *evp_key = EVP_PKEY_new(); + if (EVP_PKEY_set1_RSA(evp_key, rsa_key) == 0) + { + Log(LOG_LEVEL_ERR, "Failed to initialize encryption context"); + SeqDestroy(evp_keys); + fclose(input_file); + fclose(output_file); + return false; + } + SeqAppend(evp_keys, evp_key); } bool success = true; const EVP_CIPHER *cipher = EVP_aes_256_cbc(); EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new(); - const int key_size = EVP_PKEY_size(evp_key); + const int key_size = EVP_PKEY_size((EVP_PKEY*) SeqAt(evp_keys, 0)); - unsigned char *ek = xmalloc(key_size); - int ek_size; + /* This sequence and the 'enc_key_sizes' array are both populated by the + * EVP_SealInit() call below. */ + Seq *enc_keys = SeqNew(n_keys, free); + for (size_t i = 0; i < n_keys; i++) + { + SeqAppend(enc_keys, xmalloc(key_size)); + } + int enc_key_sizes[n_keys]; const int iv_size = EVP_CIPHER_iv_length(cipher); unsigned char iv[iv_size]; @@ -302,7 +309,9 @@ static bool RSAEncrypt(const char *pubkey_path, const char *input_path, const ch char plaintext[block_size], ciphertext[2 * block_size]; int ct_len; - int ret = EVP_SealInit(ctx, cipher, &ek, &ek_size, iv, &evp_key, 1); + int ret = EVP_SealInit(ctx, cipher, + (unsigned char**) SeqGetData(enc_keys), enc_key_sizes, iv, + (EVP_PKEY**) SeqGetData(evp_keys), n_keys); if (ret == 0) { Log(LOG_LEVEL_ERR, "Failed to initialize encryption context"); @@ -310,13 +319,34 @@ static bool RSAEncrypt(const char *pubkey_path, const char *input_path, const ch goto cleanup; } - const char *headers = "Version: 1.0\n" - "\n"; - size_t headers_len = strlen(headers); - ssize_t n_written = FullWrite(fileno(output_file), headers, headers_len); - if (n_written != headers_len) + /* newline and NUL-byte => +2 */ + char header[MAX_HEADER_LEN + 2] = "Version: 1.0\n"; + size_t header_len = strlen(header); + ssize_t n_written = FullWrite(fileno(output_file), header, header_len); + if (n_written != header_len) { - Log(LOG_LEVEL_ERR, "Failed to write headers to the output file '%s'", output_path); + Log(LOG_LEVEL_ERR, "Failed to write header to the output file '%s'", output_path); + success = false; + goto cleanup; + } + for (size_t i = 0; i < n_keys; i++) + { + char *key_digest = GetPubkeyDigest(SeqAt(rsa_keys, i)); + header_len = snprintf(header, MAX_HEADER_LEN + 2, "Encrypted-for: %s\n", key_digest); + free(key_digest); + assert(header_len <= (MAX_HEADER_LEN + 2)); + n_written = FullWrite(fileno(output_file), header, header_len); + if (n_written != header_len) + { + Log(LOG_LEVEL_ERR, "Failed to write header to the output file '%s'", output_path); + success = false; + goto cleanup; + } + } + n_written = FullWrite(fileno(output_file), "\n", 1); + if (n_written != 1) + { + Log(LOG_LEVEL_ERR, "Failed to write header to the output file '%s'", output_path); success = false; goto cleanup; } @@ -328,12 +358,17 @@ static bool RSAEncrypt(const char *pubkey_path, const char *input_path, const ch success = false; goto cleanup; } - n_written = FullWrite(fileno(output_file), ek, ek_size); - if (n_written != ek_size) + + for (size_t i = 0; i < n_keys; i++) { - Log(LOG_LEVEL_ERR, "Failed to write key to the output file '%s'", output_path); - success = false; - goto cleanup; + const char *enc_key = SeqAt(enc_keys, i); + n_written = FullWrite(fileno(output_file), enc_key, enc_key_sizes[i]); + if (n_written != enc_key_sizes[i]) + { + Log(LOG_LEVEL_ERR, "Failed to write key to the output file '%s'", output_path); + success = false; + goto cleanup; + } } while (success && !feof(input_file)) @@ -380,16 +415,15 @@ static bool RSAEncrypt(const char *pubkey_path, const char *input_path, const ch OPENSSL_cleanse(plaintext, block_size); cleanup: - RSA_free(pubkey); fclose(input_file); fclose(output_file); - EVP_PKEY_free(evp_key); + SeqDestroy(evp_keys); + SeqDestroy(enc_keys); EVP_CIPHER_CTX_free(ctx); - free(ek); return success; } -static bool CheckHeader(const char *key, const char *value) +static inline bool CheckHeader(const char *key, const char *value) { if (StringSafeEqual(key, "Version")) { @@ -403,6 +437,11 @@ static bool CheckHeader(const char *key, const char *value) return true; } } + else if (StringSafeEqual(key, "Encrypted-for")) + { + /* TODO: do some verification that 'value' is valid hash digest? */ + return true; + } else { Log(LOG_LEVEL_ERR, "Unsupported header: '%s'", key); @@ -410,66 +449,149 @@ static bool CheckHeader(const char *key, const char *value) } } -static bool ParseHeaders(FILE *input_file) +/** + * Read from #input_file into #buffer at most #max_chars or until #delimiter is + * encountered. If #stop_on_space, also stop when ' ' is read. #buffer is + * NUL-terminated when this function returns. + * + * @warning #buffer has to be at least #max_chars+1 big to accommodate for the + * terminating NUL byte. + */ +static inline bool ReadUntilDelim(FILE *input_file, char *buffer, size_t max_chars, char delimiter, bool stop_on_space) +{ + bool done = false; + size_t i = 0; + int c = fgetc(input_file); + while (!done && (i < max_chars)) + { + if (c == EOF) + { + done = true; + } + else if (c == delimiter) + { + done = true; + ungetc(c, input_file); + } + else if (stop_on_space && (c == ' ')) + { + done = true; + ungetc(c, input_file); + } + else + { + buffer[i] = (char) c; + i++; + if (i < max_chars) + { + c = fgetc(input_file); + } + } + } + buffer[i] = '\0'; + + bool ran_out = ((i > max_chars) || (c == EOF)); + return !ran_out; +} + +static inline void SkipSpaces(FILE *input_file) +{ + int c = ' '; + while (!feof(input_file) && (c == ' ')) + { + c = fgetc(input_file); + } + if (c != ' ') + { + ungetc(c, input_file); + } +} + +static inline bool ParseHeader(FILE *input_file, char *key, char *value) { + bool ok = ReadUntilDelim(input_file, key, MAX_HEADER_KEY_LEN, ':', true); + if (ok) + { + SkipSpaces(input_file); + ok = (fgetc(input_file) == ':'); + SkipSpaces(input_file); + ok = ReadUntilDelim(input_file, value, MAX_HEADER_VAL_LEN, '\n', false); + } + ok = (fgetc(input_file) == '\n'); + return ok; +} + +static bool ParseHeaders(FILE *input_file, RSA *privkey, size_t *enc_key_pos, size_t *n_enc_keys) +{ + /* Actually works for a private RSA key too because it contains the public + * key. */ + char *key_digest = GetPubkeyDigest(privkey); + bool version_specified = false; + bool found_matching_digest = false; + size_t n_enc_for_headers = 0; + char key[MAX_HEADER_KEY_LEN + 1]; char value[MAX_HEADER_VAL_LEN + 1]; - int ret = fscanf(input_file, - "%"MAX_HEADER_KEY_LEN_STR"[^:]: %"MAX_HEADER_VAL_LEN_STR"[^\n]", - key, value); - bool version_specified = false; - while (ret == 2) + + while (ParseHeader(input_file, key, value)) { if (!CheckHeader(key, value)) { + free(key_digest); return false; } - version_specified = (version_specified || (StringSafeEqual(key, "Version"))); - /* each header should be terminated by a newline */ - char next = fgetc(input_file); + if (StringSafeEqual(key, "Version")) + { + version_specified = true; + } + else if (StringSafeEqual(key, "Encrypted-for")) + { + if (StringSafeEqual(value, key_digest)) + { + found_matching_digest = true; + *enc_key_pos = n_enc_for_headers; + } + n_enc_for_headers++; + } + + /* headers are supposed to be terminated by a blank line */ + int next = fgetc(input_file); if (next == '\n') { - next = fgetc(input_file); - if (next == '\n') + if (!version_specified) { - /* headers are supposed to be separated by blank line */ - if (!version_specified) - { - Log(LOG_LEVEL_ERR, "File format version not specified"); - } - return version_specified; + Log(LOG_LEVEL_ERR, "File format version not specified"); } - else + if (!found_matching_digest) { - /* keep trying */ - ungetc(next, input_file); + Log(LOG_LEVEL_ERR, "File not encrypted for host '%s'", key_digest); } + *n_enc_keys = n_enc_for_headers; + free(key_digest); + return (version_specified && found_matching_digest); } - else + else if (next == EOF) { + free(key_digest); return false; } - ret = fscanf(input_file, - "%"MAX_HEADER_KEY_LEN_STR"[^:]: %"MAX_HEADER_VAL_LEN_STR"[^\n]\n", - key, value); + else + { + /* keep trying */ + ungetc(next, input_file); + } } + free(key_digest); return false; } -static bool RSADecrypt(const char *privkey_path, const char *input_path, const char *output_path) +static bool RSADecrypt(RSA *privkey, const char *input_path, const char *output_path) { - RSA *privkey = ReadPrivateKey(privkey_path); - if (privkey == NULL) - { - return false; - } - FILE *input_file = OpenInputOutput(input_path, "r"); if (input_file == NULL) { Log(LOG_LEVEL_ERR, "Cannot open input file '%s'", input_path); - RSA_free(privkey); return false; } @@ -478,7 +600,6 @@ static bool RSADecrypt(const char *privkey_path, const char *input_path, const c { Log(LOG_LEVEL_ERR, "Cannot open output file '%s'", output_path); fclose(input_file); - RSA_free(privkey); return false; } @@ -486,7 +607,6 @@ static bool RSADecrypt(const char *privkey_path, const char *input_path, const c if (EVP_PKEY_set1_RSA(evp_key, privkey) == 0) { Log(LOG_LEVEL_ERR, "Failed to initialize decryption context"); - RSA_free(privkey); fclose(input_file); fclose(output_file); return false; @@ -494,10 +614,11 @@ static bool RSADecrypt(const char *privkey_path, const char *input_path, const c bool success = true; - if (!ParseHeaders(input_file)) + size_t our_key_pos; + size_t n_enc_keys; + if (!ParseHeaders(input_file, privkey, &our_key_pos, &n_enc_keys)) { Log(LOG_LEVEL_ERR, "Failed to parse headers from '%s'", input_path); - RSA_free(privkey); fclose(input_file); fclose(output_file); return false; @@ -511,6 +632,7 @@ static bool RSADecrypt(const char *privkey_path, const char *input_path, const c const int key_size = EVP_PKEY_size(evp_key); unsigned char ek[key_size]; + unsigned char dev_null[key_size]; const int block_size = EVP_CIPHER_block_size(cipher); @@ -523,12 +645,36 @@ static bool RSADecrypt(const char *privkey_path, const char *input_path, const c Log(LOG_LEVEL_ERR, "Failed to read the IV from '%s'", input_path); goto cleanup; } + + /* just skip the keys that are not ours */ + size_t nth_key = 0; + for (; nth_key < our_key_pos; nth_key++) + { + n_read = ReadFileStreamToBuffer(input_file, key_size, dev_null); + if (n_read != key_size) + { + Log(LOG_LEVEL_ERR, "Failed to read the key from '%s'", input_path); + goto cleanup; + } + } + /* read our key */ n_read = ReadFileStreamToBuffer(input_file, key_size, ek); if (n_read != key_size) { Log(LOG_LEVEL_ERR, "Failed to read the key from '%s'", input_path); goto cleanup; } + nth_key++; + /* skip the remaining keys */ + for (; nth_key < n_enc_keys; nth_key++) + { + n_read = ReadFileStreamToBuffer(input_file, key_size, dev_null); + if (n_read != key_size) + { + Log(LOG_LEVEL_ERR, "Failed to read the key from '%s'", input_path); + goto cleanup; + } + } int ret = EVP_OpenInit(ctx, cipher, ek, key_size, iv, evp_key); if (ret == 0) @@ -583,7 +729,6 @@ static bool RSADecrypt(const char *privkey_path, const char *input_path, const c OPENSSL_cleanse(plaintext, block_size); cleanup: - RSA_free(privkey); fclose(input_file); fclose(output_file); EVP_PKEY_free(evp_key); @@ -591,6 +736,24 @@ static bool RSADecrypt(const char *privkey_path, const char *input_path, const c return success; } +static Seq *LoadPublicKeys(Seq *key_paths) +{ + const size_t n_keys = SeqLength(key_paths); + Seq *pub_keys = SeqNew(n_keys, RSA_free); + for (size_t i = 0; i < n_keys; i++) + { + const char *key_path = SeqAt(key_paths, i); + RSA *pubkey = ReadPublicKey(key_path); + if (pubkey == NULL) + { + SeqDestroy(pub_keys); + return NULL; + } + SeqAppend(pub_keys, pubkey); + } + return pub_keys; +} + static void CFKeyCryptHelp() { Writer *w = FileWriter(stdout); @@ -617,10 +780,10 @@ int main(int argc, char *argv[]) } opterr = 0; - char *key_path = NULL; + char *key_path_arg = NULL; char *input_path = NULL; char *output_path = NULL; - char *host = NULL; + char *host_arg = NULL; bool encrypt = false; bool decrypt = false; @@ -644,13 +807,13 @@ int main(int argc, char *argv[]) decrypt = true; break; case 'k': - key_path = optarg; + key_path_arg = optarg; break; case 'o': output_path = optarg; break; case 'H': - host = optarg; + host_arg = optarg; break; default: Log(LOG_LEVEL_ERR, "Unknown option '-%c'", optopt); @@ -674,16 +837,16 @@ int main(int argc, char *argv[]) exit(EXIT_FAILURE); } - if (decrypt && (host == NULL) && (key_path == NULL)) + if (decrypt && (host_arg == NULL) && (key_path_arg == NULL)) { /* Decryption requires a private key which is usually only available for * the local host. Let's just default to localhost if no other specific * host/key is given for decryption. */ Log(LOG_LEVEL_VERBOSE, "Using the localhost private key for decryption"); - host = "localhost"; + host_arg = "localhost"; } - if ((host != NULL) && (key_path != NULL)) + if ((host_arg != NULL) && (key_path_arg != NULL)) { Log(LOG_LEVEL_ERR, "--host/-H is used to specify a public key and cannot be used with --key/-k"); @@ -702,23 +865,54 @@ int main(int argc, char *argv[]) } CryptoInitialize(); - if (host != NULL) + GenericAgentSetDefaultDigest(&CF_DEFAULT_DIGEST, &CF_DEFAULT_DIGEST_LEN); + + Seq *key_paths; + if (key_path_arg != NULL) + { + key_paths = SeqStringFromString(key_path_arg, ','); + } + else { - HostRSAKeyType key_type = encrypt ? HOST_RSA_KEY_PUBLIC : HOST_RSA_KEY_PRIVATE; - key_path = GetHostRSAKey(host, key_type); - if (!key_path) + key_paths = SeqNew(16, free); + } + + if (host_arg != NULL) + { + Seq *hosts = SeqStringFromString(host_arg, ','); + const size_t n_hosts = SeqLength(hosts); + for (size_t i = 0; i < n_hosts; i++) { - Log(LOG_LEVEL_ERR, "Unable to locate key for host '%s'", host); - exit(EXIT_FAILURE); + HostRSAKeyType key_type = encrypt ? HOST_RSA_KEY_PUBLIC : HOST_RSA_KEY_PRIVATE; + char *host = SeqAt(hosts, i); + char *host_key_path = GetHostRSAKey(host, key_type); + if (!host_key_path) + { + Log(LOG_LEVEL_ERR, "Unable to locate key for host '%s'", host); + SeqDestroy(hosts); + SeqDestroy(key_paths); + exit(EXIT_FAILURE); + } + SeqAppend(key_paths, host_key_path); } + SeqDestroy(hosts); } - assert (key_path != NULL); + assert ((key_paths != NULL) && (SeqLength(key_paths) > 0)); // Encrypt or decrypt bool success; if (encrypt) { - success = RSAEncrypt(key_path, input_path, output_path); + Seq *pub_keys = LoadPublicKeys(key_paths); + SeqDestroy(key_paths); + if (pub_keys == NULL) + { + Log(LOG_LEVEL_ERR, "Failed to load public key(s)"); + exit(EXIT_FAILURE); + } + + success = RSAEncrypt(pub_keys, input_path, output_path); + SeqDestroy(pub_keys); if (!success) { Log(LOG_LEVEL_ERR, "Encryption failed"); @@ -727,7 +921,17 @@ int main(int argc, char *argv[]) } else if (decrypt) { - success = RSADecrypt(key_path, input_path, output_path); + const size_t n_keys = SeqLength(key_paths); + if (n_keys > 1) + { + Log(LOG_LEVEL_ERR, "--decrypt requires only one key/host to be specified"); + SeqDestroy(key_paths); + exit(EXIT_FAILURE); + } + RSA *private_key = ReadPrivateKey((char *) SeqAt(key_paths, 0)); + SeqDestroy(key_paths); + success = RSADecrypt(private_key, input_path, output_path); + RSA_free(private_key); if (!success) { Log(LOG_LEVEL_ERR, "Decryption failed"); diff --git a/tests/acceptance/27_cf-keycrypt/encrypted b/tests/acceptance/27_cf-keycrypt/encrypted index f710d5a094e54670251c329aff4d4598f4fd0880..d37e09ef6badb94aa0764bc9427c561c735b2d4b 100644 GIT binary patch literal 370 zcmV-&0ge7vWpZ>{w^01wldWyf%BFfgx# zxcjbUM``Gk1#e?&GACCsQw(pX0K3*ia4LDS5Qs3T_!iUPS4ZuS;96872?L5qtFzhh znb@NT^zZAtC#FpVTF6BeLTKtlfgt6hOK;B1Jf=p`cu1RwJ-8jyaG{6nY!y_5U4=Ge zocGNcXLQjss0(2Drf@QHGhi0bLEnf{ipb-^9haXZ^&0H(gv=D-?SF3R3D>?cBav)E zx@?D!KxVB(ZPCP_2Ky5pe55jj8E=Iu++gx0UWD6bTg~hoq3$Vc2YV+pQ|xE#KHBqe zJEBzloB^{o?n+ES3_t=kFwPj%Z7jW>gZEJe-NG%JIa&i|l$Xnq^4{d$Idq?5>l2(z Q6;3N)SM8a-bgV}0(uUrrBme*a literal 318 zcmV-E0m1%OWpZFMQnIJ$yqf|hoRXEvfr)BnZ;CzGb#kTvs znx3aPSyg>WYdp%G9DHV!AuQZA^5#K_q8>%F#zs}ShqC^3&6QsgOiqS}D+zg-eegXM zelMLdWfb2M4f6ND;$coL#BRj>n};|5)og1k;Ywij<)#!oupXj?`$t7vFCr|zjN8Mo zM^QD{3BHrwcU3IP6v$`}+MGFl?vfz!&zwi_;XMpm#LA^9(ZaZaYa{ Date: Thu, 9 Apr 2020 12:00:50 +0200 Subject: [PATCH 101/333] Replace --encrypt/--decrypt cf-keycrypt options with commands It's more natural and common these days to write 'cf-keycrypt encrypt OPTS' than 'cf-keycrypt --encrypt OPTS'. Ticket: CFE-3272 Changelog: None (cherry picked from commit 163ecc3b884ae3368319de897a1f91db216d6b3f) --- cf-keycrypt/cf-keycrypt.c | 52 ++++++++++++------- tests/acceptance/27_cf-keycrypt/decrypt.cf | 2 +- .../27_cf-keycrypt/encrypt-decrypt-args.cf | 4 +- .../27_cf-keycrypt/encrypt-decrypt.cf | 4 +- tests/acceptance/27_cf-keycrypt/encrypt.x.cf | 4 +- 5 files changed, 39 insertions(+), 27 deletions(-) diff --git a/cf-keycrypt/cf-keycrypt.c b/cf-keycrypt/cf-keycrypt.c index 27f69251ea..1a0c2f4c16 100644 --- a/cf-keycrypt/cf-keycrypt.c +++ b/cf-keycrypt/cf-keycrypt.c @@ -97,8 +97,6 @@ static const struct option OPTIONS[] = { {"help", no_argument, 0, 'h'}, {"manpage", no_argument, 0, 'M'}, - {"encrypt", no_argument, 0, 'e'}, - {"decrypt", no_argument, 0, 'd'}, {"key", required_argument, 0, 'k'}, {"host", required_argument, 0, 'H'}, {"output", required_argument, 0, 'o'}, @@ -109,14 +107,19 @@ static const char *const HINTS[] = { "Print the help message", "Print the man page", - "Encrypt file", - "Decrypt file", "Use key file", "Encrypt for host (get key from lastseen database)", "Output file", NULL }; +static const Description COMMANDS[] = +{ + {"encrypt", "Encrypt data for one or more hosts/keys", "cf-keycrypt encrypt -k/-H KEY/HOST -o OUTPUT INPUT"}, + {"decrypt", "Decrypt data", "cf-keycrypt decrypt [-k/-H KEY/HOST] -o OUTPUT INPUT"}, + {NULL, NULL, NULL} +}; + static inline void *GetIPAddress(struct sockaddr *sa) { assert(sa != NULL); @@ -757,7 +760,7 @@ static Seq *LoadPublicKeys(Seq *key_paths) static void CFKeyCryptHelp() { Writer *w = FileWriter(stdout); - WriterWriteHelp(w, "cf-keycrypt", OPTIONS, HINTS, false, NULL); + WriterWriteHelp(w, "cf-keycrypt", OPTIONS, HINTS, COMMANDS, true, true); FileWriterDetach(w); } @@ -773,7 +776,7 @@ void CFKeyCryptMan() int main(int argc, char *argv[]) { - if (argc == 1) + if (argc < 2) { CFKeyCryptHelp(); exit(EXIT_FAILURE); @@ -787,8 +790,20 @@ int main(int argc, char *argv[]) bool encrypt = false; bool decrypt = false; + size_t offset = 0; + if (StringSafeEqual(argv[1], "encrypt")) + { + encrypt = true; + offset++; + } + else if (StringSafeEqual(argv[1], "decrypt")) + { + offset++; + decrypt = true; + } + int c = 0; - while ((c = getopt_long(argc, argv, "hMedk:o:H:", OPTIONS, NULL)) != -1) + while ((c = getopt_long(argc - offset, argv + offset, "hMedk:o:H:", OPTIONS, NULL)) != -1) { switch (c) { @@ -800,12 +815,6 @@ int main(int argc, char *argv[]) CFKeyCryptMan(); exit(EXIT_SUCCESS); break; - case 'e': - encrypt = true; - break; - case 'd': - decrypt = true; - break; case 'k': key_path_arg = optarg; break; @@ -822,18 +831,21 @@ int main(int argc, char *argv[]) } } - input_path = argv[optind]; - optind += 1; - if (optind < argc) + if (!(decrypt || encrypt)) { - Log(LOG_LEVEL_ERR, "Unexpected non-option argument: '%s'", argv[optind]); + printf("Command required. Specify either 'encrypt' or 'decrypt'\n"); + CFKeyCryptHelp(); exit(EXIT_FAILURE); } - // Check for argument errors: - if ((encrypt && decrypt) || (!encrypt && !decrypt)) + /* Increment 'optind' because of command being argv[0]. */ + optind++; + input_path = argv[optind]; + + /* Some more unexpected arguments? */ + if (argc > (optind + offset)) { - Log(LOG_LEVEL_ERR, "Must specify either encrypt or decrypt (and not both)"); + Log(LOG_LEVEL_ERR, "Unexpected non-option argument: '%s'", argv[optind + 1]); exit(EXIT_FAILURE); } diff --git a/tests/acceptance/27_cf-keycrypt/decrypt.cf b/tests/acceptance/27_cf-keycrypt/decrypt.cf index 340b672bd0..6f73819c0d 100644 --- a/tests/acceptance/27_cf-keycrypt/decrypt.cf +++ b/tests/acceptance/27_cf-keycrypt/decrypt.cf @@ -13,7 +13,7 @@ bundle agent test commands: "$(sys.cf_keycrypt)" - args => "-k $(this.promise_dirname)/testkey.priv -d $(this.promise_dirname)/encrypted -o $(G.testfile)"; + args => "decrypt -k $(this.promise_dirname)/testkey.priv -o $(G.testfile) $(this.promise_dirname)/encrypted"; } bundle agent check diff --git a/tests/acceptance/27_cf-keycrypt/encrypt-decrypt-args.cf b/tests/acceptance/27_cf-keycrypt/encrypt-decrypt-args.cf index f87dcb310f..66bd5b0d0a 100644 --- a/tests/acceptance/27_cf-keycrypt/encrypt-decrypt-args.cf +++ b/tests/acceptance/27_cf-keycrypt/encrypt-decrypt-args.cf @@ -33,9 +33,9 @@ bundle agent test commands: "$(sys.cf_keycrypt)" - args => "--encrypt $(G.testfile).plain --key $(sys.workdir)/ppkeys/localhost.pub --output $(G.testfile).encrypted"; + args => "encrypt --key $(sys.workdir)/ppkeys/localhost.pub --output $(G.testfile).encrypted $(G.testfile).plain"; "$(sys.cf_keycrypt)" - args => "--decrypt -o $(G.testfile).decrypted --key $(sys.workdir)/ppkeys/localhost.priv $(G.testfile).encrypted"; + args => "decrypt --key $(sys.workdir)/ppkeys/localhost.priv -o $(G.testfile).decrypted $(G.testfile).encrypted"; reports: "Binaries/folders:"; "$(sys.cf_keycrypt)"; diff --git a/tests/acceptance/27_cf-keycrypt/encrypt-decrypt.cf b/tests/acceptance/27_cf-keycrypt/encrypt-decrypt.cf index 2dc5fc4916..29e217f3a3 100644 --- a/tests/acceptance/27_cf-keycrypt/encrypt-decrypt.cf +++ b/tests/acceptance/27_cf-keycrypt/encrypt-decrypt.cf @@ -34,9 +34,9 @@ bundle agent test commands: "$(sys.cf_keycrypt)" - args => "-k $(sys.workdir)/ppkeys/localhost.pub -e $(G.testfile).plain -o $(G.testfile).encrypted"; + args => "encrypt -k $(sys.workdir)/ppkeys/localhost.pub -o $(G.testfile).encrypted $(G.testfile).plain"; "$(sys.cf_keycrypt)" - args => "-k $(sys.workdir)/ppkeys/localhost.priv -d $(G.testfile).encrypted -o $(G.testfile).decrypted"; + args => "decrypt -k $(sys.workdir)/ppkeys/localhost.priv -o $(G.testfile).decrypted $(G.testfile).encrypted"; reports: "Binaries/folders:"; diff --git a/tests/acceptance/27_cf-keycrypt/encrypt.x.cf b/tests/acceptance/27_cf-keycrypt/encrypt.x.cf index 23e72337aa..fe6b017a9a 100644 --- a/tests/acceptance/27_cf-keycrypt/encrypt.x.cf +++ b/tests/acceptance/27_cf-keycrypt/encrypt.x.cf @@ -13,9 +13,9 @@ bundle agent test commands: "$(sys.cf_keycrypt)" - args => "-e $(this.promise_dirname)/plaintext -k $(this.promise_dirname)/testkey.pub -o $(G.testfile)"; + args => "encrypt -k $(this.promise_dirname)/testkey.pub -o $(G.testfile) $(this.promise_dirname)/plaintext"; "$(sys.cf_keycrypt)" - args => "-e $(this.promise_dirname)/plaintext -k $(this.promise_dirname)/testkey.pub -o $(G.testfile).2"; + args => "encrypt -k $(this.promise_dirname)/testkey.pub -o $(G.testfile).2 $(this.promise_dirname)/plaintext"; } bundle agent check From f09c498389aa481a1fff372938c14ac98b5f5561 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 9 Apr 2020 12:17:15 +0200 Subject: [PATCH 102/333] Provide more information in 'cf-keycrypt -h' output It should be clear which options are required, what individual options mean and what they expect as a value. Ticket: CFE-3272 Changelog: None (cherry picked from commit 8762e43000b2b2e6c5a065b354e93bcfe8c5c9cb) --- cf-keycrypt/cf-keycrypt.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cf-keycrypt/cf-keycrypt.c b/cf-keycrypt/cf-keycrypt.c index 1a0c2f4c16..4facb60bd7 100644 --- a/cf-keycrypt/cf-keycrypt.c +++ b/cf-keycrypt/cf-keycrypt.c @@ -107,9 +107,9 @@ static const char *const HINTS[] = { "Print the help message", "Print the man page", - "Use key file", - "Encrypt for host (get key from lastseen database)", - "Output file", + "Comma-separated list of key files to use (one of -k/-H options is required for encryption)", + "Comma-separated list of hosts to encrypt/decrypt for (defaults to 'localhost' for decryption)", + "Output file (required)", NULL }; From 629ace0f30366fe20e2a9f25087f7fc7d6867851 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 9 Apr 2020 13:51:35 +0200 Subject: [PATCH 103/333] Add standard options for setting log level to cf-keycrypt And add a lot more logging now that it is useful. Ticket: CFE-3272 Changelog: None (cherry picked from commit 8f1c3ec14217dd016cbab7b5e1247085ccd217b0) --- cf-keycrypt/cf-keycrypt.c | 66 +++++++++++++++++++++++++++++++++++---- 1 file changed, 60 insertions(+), 6 deletions(-) diff --git a/cf-keycrypt/cf-keycrypt.c b/cf-keycrypt/cf-keycrypt.c index 4facb60bd7..541f3a94e1 100644 --- a/cf-keycrypt/cf-keycrypt.c +++ b/cf-keycrypt/cf-keycrypt.c @@ -97,6 +97,10 @@ static const struct option OPTIONS[] = { {"help", no_argument, 0, 'h'}, {"manpage", no_argument, 0, 'M'}, + {"debug", no_argument, 0, 'd'}, + {"verbose", no_argument, 0, 'v'}, + {"log-level", required_argument, 0, 'g'}, + {"inform", no_argument, 0, 'I'}, {"key", required_argument, 0, 'k'}, {"host", required_argument, 0, 'H'}, {"output", required_argument, 0, 'o'}, @@ -107,6 +111,10 @@ static const char *const HINTS[] = { "Print the help message", "Print the man page", + "Enable debugging output", + "Enable verbose output", + "Specify how detailed logs should be. Possible values: 'error', 'warning', 'notice', 'info', 'verbose', 'debug'", + "Enable basic information output", "Comma-separated list of key files to use (one of -k/-H options is required for encryption)", "Comma-separated list of hosts to encrypt/decrypt for (defaults to 'localhost' for decryption)", "Output file (required)", @@ -165,6 +173,7 @@ static char *GetHostRSAKey(const char *host, HostRSAKeyType type) ipaddress, sizeof(ipaddress)); if (StringStartsWith(ipaddress, "127.") || StringSafeEqual(ipaddress, "::1")) { + Log(LOG_LEVEL_VERBOSE, "Using localhost%s key", key_ext); found = true; snprintf(buffer, BUFSIZE, "%s/ppkeys/localhost%s", WORKDIR, key_ext); return buffer; @@ -173,12 +182,14 @@ static char *GetHostRSAKey(const char *host, HostRSAKeyType type) } if (found) { + Log(LOG_LEVEL_DEBUG, "Found host '%s' for address '%s'", hash, ipaddress); snprintf(buffer, BUFSIZE, "%s/ppkeys/root-%s%s", WORKDIR, hash, key_ext); freeaddrinfo(result); return buffer; } else { + Log(LOG_LEVEL_DEBUG, "Searching key by IP"); for (struct addrinfo *res = result; res != NULL; res = res->ai_next) { inet_ntop(res->ai_family, @@ -187,6 +198,7 @@ static char *GetHostRSAKey(const char *host, HostRSAKeyType type) snprintf(buffer, BUFSIZE, "%s/ppkeys/root-%s%s", WORKDIR, ipaddress, key_ext); if (access(buffer, F_OK) == 0) { + Log(LOG_LEVEL_DEBUG, "Found matching key: '%s'", buffer); freeaddrinfo(result); return buffer; } @@ -268,7 +280,7 @@ static bool RSAEncrypt(Seq *rsa_keys, const char *input_path, const char *output FILE *output_file = OpenInputOutput(output_path, "w"); if (output_file == NULL) { - Log(LOG_LEVEL_ERR, "Could not create output file '%s'", output_path); + Log(LOG_LEVEL_ERR, "Could not create or open output file '%s'", output_path); fclose(input_file); return false; } @@ -323,6 +335,7 @@ static bool RSAEncrypt(Seq *rsa_keys, const char *input_path, const char *output } /* newline and NUL-byte => +2 */ + Log(LOG_LEVEL_VERBOSE, "Writing headers"); char header[MAX_HEADER_LEN + 2] = "Version: 1.0\n"; size_t header_len = strlen(header); ssize_t n_written = FullWrite(fileno(output_file), header, header_len); @@ -332,6 +345,7 @@ static bool RSAEncrypt(Seq *rsa_keys, const char *input_path, const char *output success = false; goto cleanup; } + Log(LOG_LEVEL_VERBOSE, "Writing Encrypted-for headers"); for (size_t i = 0; i < n_keys; i++) { char *key_digest = GetPubkeyDigest(SeqAt(rsa_keys, i)); @@ -353,7 +367,7 @@ static bool RSAEncrypt(Seq *rsa_keys, const char *input_path, const char *output success = false; goto cleanup; } - + Log(LOG_LEVEL_VERBOSE, "Writing IV"); n_written = FullWrite(fileno(output_file), iv, iv_size); if (n_written != iv_size) { @@ -362,6 +376,7 @@ static bool RSAEncrypt(Seq *rsa_keys, const char *input_path, const char *output goto cleanup; } + Log(LOG_LEVEL_VERBOSE, "Writing keys"); for (size_t i = 0; i < n_keys; i++) { const char *enc_key = SeqAt(enc_keys, i); @@ -374,6 +389,7 @@ static bool RSAEncrypt(Seq *rsa_keys, const char *input_path, const char *output } } + size_t processed = 0; while (success && !feof(input_file)) { ssize_t n_read = ReadFileStreamToBuffer(input_file, block_size, plaintext); @@ -398,7 +414,10 @@ static bool RSAEncrypt(Seq *rsa_keys, const char *input_path, const char *output success = false; break; } + processed += n_read; + Log(LOG_LEVEL_VERBOSE, "%zu bytes processed", processed); } + Log(LOG_LEVEL_VERBOSE, "Finalizing"); ret = EVP_SealFinal(ctx, ciphertext, &ct_len); if (ret == 0) { @@ -538,6 +557,7 @@ static bool ParseHeaders(FILE *input_file, RSA *privkey, size_t *enc_key_pos, si while (ParseHeader(input_file, key, value)) { + Log(LOG_LEVEL_DEBUG, "Parsed header '%s: %s'", key, value); if (!CheckHeader(key, value)) { free(key_digest); @@ -550,6 +570,7 @@ static bool ParseHeaders(FILE *input_file, RSA *privkey, size_t *enc_key_pos, si } else if (StringSafeEqual(key, "Encrypted-for")) { + Log(LOG_LEVEL_DEBUG, "Encrypted for '%s'", value); if (StringSafeEqual(value, key_digest)) { found_matching_digest = true; @@ -576,6 +597,7 @@ static bool ParseHeaders(FILE *input_file, RSA *privkey, size_t *enc_key_pos, si } else if (next == EOF) { + Log(LOG_LEVEL_ERR, "Failed to parse headers from"); free(key_digest); return false; } @@ -585,6 +607,7 @@ static bool ParseHeaders(FILE *input_file, RSA *privkey, size_t *enc_key_pos, si ungetc(next, input_file); } } + Log(LOG_LEVEL_ERR, "Failed to parse headers"); free(key_digest); return false; } @@ -617,15 +640,16 @@ static bool RSADecrypt(RSA *privkey, const char *input_path, const char *output_ bool success = true; + Log(LOG_LEVEL_VERBOSE, "Parsing headers"); size_t our_key_pos; size_t n_enc_keys; if (!ParseHeaders(input_file, privkey, &our_key_pos, &n_enc_keys)) { - Log(LOG_LEVEL_ERR, "Failed to parse headers from '%s'", input_path); fclose(input_file); fclose(output_file); return false; } + Log(LOG_LEVEL_DEBUG, "Parsed %zu keys", n_enc_keys); const EVP_CIPHER *cipher = EVP_aes_256_cbc(); EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new(); @@ -659,8 +683,10 @@ static bool RSADecrypt(RSA *privkey, const char *input_path, const char *output_ Log(LOG_LEVEL_ERR, "Failed to read the key from '%s'", input_path); goto cleanup; } + Log(LOG_LEVEL_DEBUG, "Skipping key"); } /* read our key */ + Log(LOG_LEVEL_DEBUG, "Reading key"); n_read = ReadFileStreamToBuffer(input_file, key_size, ek); if (n_read != key_size) { @@ -677,16 +703,21 @@ static bool RSADecrypt(RSA *privkey, const char *input_path, const char *output_ Log(LOG_LEVEL_ERR, "Failed to read the key from '%s'", input_path); goto cleanup; } + Log(LOG_LEVEL_DEBUG, "Skipping key"); } int ret = EVP_OpenInit(ctx, cipher, ek, key_size, iv, evp_key); if (ret == 0) { - Log(LOG_LEVEL_ERR, "Failed to initialize decryption context"); + char *key_digest = GetPubkeyDigest(privkey); + Log(LOG_LEVEL_ERR, "Failed to decrypt contents using key '%s'", key_digest); + free(key_digest); success = false; goto cleanup; } + Log(LOG_LEVEL_VERBOSE, "Decrypting data"); + size_t processed = 0; ssize_t n_written; while (success && !feof(input_file)) { @@ -700,7 +731,7 @@ static bool RSADecrypt(RSA *privkey, const char *input_path, const char *output_ ret = EVP_OpenUpdate(ctx, plaintext, &pt_len, ciphertext, n_read); if (ret == 0) { - Log(LOG_LEVEL_ERR, "Failed to encrypt data: %s", + Log(LOG_LEVEL_ERR, "Failed to decrypt data: %s", ERR_error_string(ERR_get_error(), NULL)); success = false; break; @@ -712,11 +743,14 @@ static bool RSADecrypt(RSA *privkey, const char *input_path, const char *output_ success = false; break; } + processed += n_read; + Log(LOG_LEVEL_VERBOSE, "%zu bytes processed", processed); } + Log(LOG_LEVEL_VERBOSE, "Finalizing"); ret = EVP_OpenFinal(ctx, plaintext, &pt_len); if (ret == 0) { - Log(LOG_LEVEL_ERR, "Failed to encrypt data: %s", + Log(LOG_LEVEL_ERR, "Failed to decrypt data: %s", ERR_error_string(ERR_get_error(), NULL)); success = false; } @@ -746,6 +780,7 @@ static Seq *LoadPublicKeys(Seq *key_paths) for (size_t i = 0; i < n_keys; i++) { const char *key_path = SeqAt(key_paths, i); + Log(LOG_LEVEL_VERBOSE, "Reading key '%s'", key_path); RSA *pubkey = ReadPublicKey(key_path); if (pubkey == NULL) { @@ -815,6 +850,21 @@ int main(int argc, char *argv[]) CFKeyCryptMan(); exit(EXIT_SUCCESS); break; + case 'd': + LogSetGlobalLevel(LOG_LEVEL_DEBUG); + Log(LOG_LEVEL_DEBUG, "Debug log level enabled"); + break; + case 'v': + LogSetGlobalLevel(LOG_LEVEL_VERBOSE); + Log(LOG_LEVEL_VERBOSE, "Verbose log level enabled"); + break; + case 'I': + LogSetGlobalLevel(LOG_LEVEL_INFO); + Log(LOG_LEVEL_INFO, "Inform log level enabled"); + break; + case 'g': + LogSetGlobalLevelArgOrExit(optarg); + break; case 'k': key_path_arg = optarg; break; @@ -882,6 +932,7 @@ int main(int argc, char *argv[]) Seq *key_paths; if (key_path_arg != NULL) { + Log(LOG_LEVEL_DEBUG, "-k/--key given: '%s'", key_path_arg); key_paths = SeqStringFromString(key_path_arg, ','); } else @@ -891,6 +942,7 @@ int main(int argc, char *argv[]) if (host_arg != NULL) { + Log(LOG_LEVEL_DEBUG, "-H/--host given: '%s'", host_arg); Seq *hosts = SeqStringFromString(host_arg, ','); const size_t n_hosts = SeqLength(hosts); for (size_t i = 0; i < n_hosts; i++) @@ -915,6 +967,7 @@ int main(int argc, char *argv[]) bool success; if (encrypt) { + Log(LOG_LEVEL_DEBUG, "Encrypting"); Seq *pub_keys = LoadPublicKeys(key_paths); SeqDestroy(key_paths); if (pub_keys == NULL) @@ -933,6 +986,7 @@ int main(int argc, char *argv[]) } else if (decrypt) { + Log(LOG_LEVEL_DEBUG, "Decrypting"); const size_t n_keys = SeqLength(key_paths); if (n_keys > 1) { From fbf043d9c36fed0b9a726cc27bc9245bee5cd8f5 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 9 Apr 2020 14:53:25 +0200 Subject: [PATCH 104/333] Add COMMANDS section to the cf-net and cf-keycrypt man pages Quite important piece of information that should be there. Ticket: CFE-3272 Changelog: None (cherry picked from commit c524680b88d594906f0f4690b17fd1e6490c2cc3) --- cf-keycrypt/cf-keycrypt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cf-keycrypt/cf-keycrypt.c b/cf-keycrypt/cf-keycrypt.c index 541f3a94e1..c8bdef4506 100644 --- a/cf-keycrypt/cf-keycrypt.c +++ b/cf-keycrypt/cf-keycrypt.c @@ -805,7 +805,7 @@ void CFKeyCryptMan() ManPageWrite(out, "cf-keycrypt", time(NULL), CF_KEYCRYPT_SHORT_DESCRIPTION, CF_KEYCRYPT_MANPAGE_LONG_DESCRIPTION, - OPTIONS, HINTS, true); + OPTIONS, HINTS, COMMANDS, true, true); FileWriterDetach(out); } From dce15e368e409bc7d346f5b20442f64da24f87d4 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 9 Apr 2020 15:25:28 +0200 Subject: [PATCH 105/333] Rename cf-keycrypt -> cf-secret Ticket: CFE-3272 Changelog: Squash this entry with the one introducing cf-keycrypt (cherry picked from commit b93fce47cf7d45d5e82c782e340e35fc82951b56) --- .gitignore | 6 +++--- Makefile.am | 2 +- {cf-keycrypt => cf-secret}/Makefile.am | 12 ++++++------ {cf-keycrypt => cf-secret}/README.md | 8 ++++---- .../cf-keycrypt.c => cf-secret/cf-secret.c | 18 +++++++++--------- configure.ac | 2 +- examples/{cf-keycrypt.cf => cf-secret.cf} | 2 +- ...ycrypt.cf.cfcrypt => cf-secret.cf.cfcrypt} | 0 ...{cf-keycrypt.cf.priv => cf-secret.cf.priv} | 0 .../{cf-keycrypt.cf.pub => cf-secret.cf.pub} | 0 libenv/sysinfo.c | 2 +- .../decrypt.cf | 4 ++-- .../encrypt-decrypt-args.cf | 8 ++++---- .../encrypt-decrypt.cf | 8 ++++---- .../encrypt.x.cf | 6 +++--- .../encrypted | Bin .../plaintext | 0 .../testkey.priv | 0 .../testkey.pub | 0 tests/acceptance/testall | 6 +++--- 20 files changed, 42 insertions(+), 42 deletions(-) rename {cf-keycrypt => cf-secret}/Makefile.am (85%) rename {cf-keycrypt => cf-secret}/README.md (85%) rename cf-keycrypt/cf-keycrypt.c => cf-secret/cf-secret.c (97%) rename examples/{cf-keycrypt.cf => cf-secret.cf} (96%) rename examples/{cf-keycrypt.cf.cfcrypt => cf-secret.cf.cfcrypt} (100%) rename examples/{cf-keycrypt.cf.priv => cf-secret.cf.priv} (100%) rename examples/{cf-keycrypt.cf.pub => cf-secret.cf.pub} (100%) rename tests/acceptance/{27_cf-keycrypt => 27_cf-secret}/decrypt.cf (79%) rename tests/acceptance/{27_cf-keycrypt => 27_cf-secret}/encrypt-decrypt-args.cf (89%) rename tests/acceptance/{27_cf-keycrypt => 27_cf-secret}/encrypt-decrypt.cf (88%) rename tests/acceptance/{27_cf-keycrypt => 27_cf-secret}/encrypt.x.cf (83%) rename tests/acceptance/{27_cf-keycrypt => 27_cf-secret}/encrypted (100%) rename tests/acceptance/{27_cf-keycrypt => 27_cf-secret}/plaintext (100%) rename tests/acceptance/{27_cf-keycrypt => 27_cf-secret}/testkey.priv (100%) rename tests/acceptance/{27_cf-keycrypt => 27_cf-secret}/testkey.pub (100%) diff --git a/.gitignore b/.gitignore index a79387dd0f..4a16d90378 100644 --- a/.gitignore +++ b/.gitignore @@ -35,7 +35,7 @@ stamp-h1 /cf-check/Makefile /cf-execd/Makefile /cf-key/Makefile -/cf-keycrypt/Makefile +/cf-secret/Makefile /cf-monitord/Makefile /cf-net/Makefile /cf-promises/Makefile @@ -92,8 +92,8 @@ stamp-h1 /cf-runagent/cf-runagent.exe /cf-net/cf-net /cf-net/cf-net.exe -/cf-keycrypt/cf-keycrypt -/cf-keycrypt/cf-keycrypt.exe +/cf-secret/cf-secret +/cf-secret/cf-secret.exe /cf-serverd/cf-serverd /cf-serverd/cf-serverd.exe /cf-testd/cf-testd diff --git a/Makefile.am b/Makefile.am index 60c7e329c8..9c9e2730db 100644 --- a/Makefile.am +++ b/Makefile.am @@ -43,7 +43,7 @@ SUBDIRS = \ cf-testd \ cf-upgrade \ cf-net \ - cf-keycrypt \ + cf-secret \ misc \ ext \ examples \ diff --git a/cf-keycrypt/Makefile.am b/cf-secret/Makefile.am similarity index 85% rename from cf-keycrypt/Makefile.am rename to cf-secret/Makefile.am index fff519f0ef..f78f07e54f 100644 --- a/cf-keycrypt/Makefile.am +++ b/cf-secret/Makefile.am @@ -21,7 +21,7 @@ # (COSL) may apply to this file if you as a licensee so wish it. See # included file COSL.txt. # -noinst_LTLIBRARIES = libcf-keycrypt.la +noinst_LTLIBRARIES = libcf-secret.la AM_CPPFLAGS = -I$(srcdir)/../libpromises -I$(srcdir)/../libntech/libutils \ -I$(srcdir)/../libcfnet \ @@ -33,14 +33,14 @@ AM_CFLAGS = @CFLAGS@ \ $(OPENSSL_CFLAGS) \ $(ENTERPRISE_CFLAGS) -libcf_keycrypt_la_LIBADD = ../libpromises/libpromises.la +libcf_secret_la_LIBADD = ../libpromises/libpromises.la -libcf_keycrypt_la_SOURCES = cf-keycrypt.c +libcf_secret_la_SOURCES = cf-secret.c if !BUILTIN_EXTENSIONS - bin_PROGRAMS = cf-keycrypt - cf_keycrypt_LDADD = libcf-keycrypt.la - cf_keycrypt_SOURCES = + bin_PROGRAMS = cf-secret + cf_secret_LDADD = libcf-secret.la + cf_secret_SOURCES = endif CLEANFILES = *.gcno *.gcda diff --git a/cf-keycrypt/README.md b/cf-secret/README.md similarity index 85% rename from cf-keycrypt/README.md rename to cf-secret/README.md index 69246e3aa2..e585d76c8f 100644 --- a/cf-keycrypt/README.md +++ b/cf-secret/README.md @@ -1,12 +1,12 @@ -# cf-keycrypt +# cf-secret -*cf-keycrypt* is a utility for encrypting sensitive data for use on +*cf-secret* is a utility for encrypting sensitive data for use on CFEngine-managed hosts. It is using the existing CFEngine key pairs for strong cryptography based on the combination of RSA and AES ciphers. ## File format -The file format used by *cf-keycrypt* has the following schema: +The file format used by *cf-secret* has the following schema: ``` ------------ @@ -52,5 +52,5 @@ The `AES key` is a randomly generated AES key encrypted by RSA public keys and each encrypted `AES key` block is as long as the RSA public key, currently 256 bytes (2048 bits). -The future versions of *cf-keycrypt* are expected to support more headers and +The future versions of *cf-secret* are expected to support more headers and varying key sizes. diff --git a/cf-keycrypt/cf-keycrypt.c b/cf-secret/cf-secret.c similarity index 97% rename from cf-keycrypt/cf-keycrypt.c rename to cf-secret/cf-secret.c index c8bdef4506..6a330f8358 100644 --- a/cf-keycrypt/cf-keycrypt.c +++ b/cf-secret/cf-secret.c @@ -23,7 +23,7 @@ */ /* - cf-keycrypt.c + cf-secret.c Copyright (C) 2017 cfengineers.net @@ -83,12 +83,12 @@ static const char passphrase[] = "Cfengine passphrase"; //******************************************************************* static const char *const CF_KEYCRYPT_SHORT_DESCRIPTION = - "cf-keycrypt: Use CFEngine cryptographic keys to encrypt and decrypt files"; + "cf-secret: Use CFEngine cryptographic keys to encrypt and decrypt files"; static const char *const CF_KEYCRYPT_MANPAGE_LONG_DESCRIPTION = - "cf-keycrypt offers a simple way to encrypt or decrypt files using keys " + "cf-secret offers a simple way to encrypt or decrypt files using keys " "generated by cf-key. CFEngine uses asymmetric cryptography, and " - "cf-keycrypt allows you to encrypt a file using a public key file. " + "cf-secret allows you to encrypt a file using a public key file. " "The encrypted file can only be decrypted on the host with the " "corresponding private key. Original author: Jon Henrik Bjornstad " ""; @@ -123,8 +123,8 @@ static const char *const HINTS[] = static const Description COMMANDS[] = { - {"encrypt", "Encrypt data for one or more hosts/keys", "cf-keycrypt encrypt -k/-H KEY/HOST -o OUTPUT INPUT"}, - {"decrypt", "Decrypt data", "cf-keycrypt decrypt [-k/-H KEY/HOST] -o OUTPUT INPUT"}, + {"encrypt", "Encrypt data for one or more hosts/keys", "cf-secret encrypt -k/-H KEY/HOST -o OUTPUT INPUT"}, + {"decrypt", "Decrypt data", "cf-secret decrypt [-k/-H KEY/HOST] -o OUTPUT INPUT"}, {NULL, NULL, NULL} }; @@ -795,14 +795,14 @@ static Seq *LoadPublicKeys(Seq *key_paths) static void CFKeyCryptHelp() { Writer *w = FileWriter(stdout); - WriterWriteHelp(w, "cf-keycrypt", OPTIONS, HINTS, COMMANDS, true, true); + WriterWriteHelp(w, "cf-secret", OPTIONS, HINTS, COMMANDS, true, true); FileWriterDetach(w); } void CFKeyCryptMan() { Writer *out = FileWriter(stdout); - ManPageWrite(out, "cf-keycrypt", time(NULL), + ManPageWrite(out, "cf-secret", time(NULL), CF_KEYCRYPT_SHORT_DESCRIPTION, CF_KEYCRYPT_MANPAGE_LONG_DESCRIPTION, OPTIONS, HINTS, COMMANDS, true, true); @@ -1006,7 +1006,7 @@ int main(int argc, char *argv[]) } else { - ProgrammingError("Unexpected error in cf-keycrypt"); + ProgrammingError("Unexpected error in cf-secret"); exit(EXIT_FAILURE); } diff --git a/configure.ac b/configure.ac index 06b7438402..c6bd2edb0e 100644 --- a/configure.ac +++ b/configure.ac @@ -1772,7 +1772,7 @@ AC_CONFIG_FILES([Makefile cf-serverd/Makefile cf-testd/Makefile cf-net/Makefile - cf-keycrypt/Makefile + cf-secret/Makefile config.post.h misc/Makefile misc/selinux/Makefile diff --git a/examples/cf-keycrypt.cf b/examples/cf-secret.cf similarity index 96% rename from examples/cf-keycrypt.cf rename to examples/cf-secret.cf index 1dd13fb092..51883cffd7 100644 --- a/examples/cf-keycrypt.cf +++ b/examples/cf-secret.cf @@ -34,7 +34,7 @@ bundle agent main "secret" comment => "We decrypt the encrypted file directly into a variable.", - string => execresult("$(sys.cf_keycrypt) -d $(private_key) -i $(encrypted_file) -o -", noshell); + string => execresult("$(sys.cf_secret) -d $(private_key) -i $(encrypted_file) -o -", noshell); reports: "Encrypted file content:" diff --git a/examples/cf-keycrypt.cf.cfcrypt b/examples/cf-secret.cf.cfcrypt similarity index 100% rename from examples/cf-keycrypt.cf.cfcrypt rename to examples/cf-secret.cf.cfcrypt diff --git a/examples/cf-keycrypt.cf.priv b/examples/cf-secret.cf.priv similarity index 100% rename from examples/cf-keycrypt.cf.priv rename to examples/cf-secret.cf.priv diff --git a/examples/cf-keycrypt.cf.pub b/examples/cf-secret.cf.pub similarity index 100% rename from examples/cf-keycrypt.cf.pub rename to examples/cf-secret.cf.pub diff --git a/libenv/sysinfo.c b/libenv/sysinfo.c index b88a7de4af..aa1bd79b60 100644 --- a/libenv/sysinfo.c +++ b/libenv/sysinfo.c @@ -435,7 +435,7 @@ static void GetNameInfo3(EvalContext *ctx) // This is used for $(sys.cf_agent), $(sys.cf_serverd) ... : char *components[COMPONENTS_SIZE] = { "cf-twin", "cf-agent", "cf-serverd", "cf-monitord", "cf-know", "cf-report", "cf-key", "cf-runagent", "cf-execd", "cf-hub", - "cf-promises", "cf-upgrade", "cf-net", "cf-check", "cf-keycrypt", + "cf-promises", "cf-upgrade", "cf-net", "cf-check", "cf-secret", NULL }; int have_component[COMPONENTS_SIZE]; diff --git a/tests/acceptance/27_cf-keycrypt/decrypt.cf b/tests/acceptance/27_cf-secret/decrypt.cf similarity index 79% rename from tests/acceptance/27_cf-keycrypt/decrypt.cf rename to tests/acceptance/27_cf-secret/decrypt.cf index 6f73819c0d..1dbca5ce79 100644 --- a/tests/acceptance/27_cf-keycrypt/decrypt.cf +++ b/tests/acceptance/27_cf-secret/decrypt.cf @@ -9,10 +9,10 @@ bundle agent test { meta: "description" - string => "Test that cf-keycrypt can still decrypt content encrypted at the time of initial implementation"; + string => "Test that cf-secret can still decrypt content encrypted at the time of initial implementation"; commands: - "$(sys.cf_keycrypt)" + "$(sys.cf_secret)" args => "decrypt -k $(this.promise_dirname)/testkey.priv -o $(G.testfile) $(this.promise_dirname)/encrypted"; } diff --git a/tests/acceptance/27_cf-keycrypt/encrypt-decrypt-args.cf b/tests/acceptance/27_cf-secret/encrypt-decrypt-args.cf similarity index 89% rename from tests/acceptance/27_cf-keycrypt/encrypt-decrypt-args.cf rename to tests/acceptance/27_cf-secret/encrypt-decrypt-args.cf index 66bd5b0d0a..d1979688cf 100644 --- a/tests/acceptance/27_cf-keycrypt/encrypt-decrypt-args.cf +++ b/tests/acceptance/27_cf-secret/encrypt-decrypt-args.cf @@ -19,7 +19,7 @@ bundle agent test { meta: "description" - string => "Test cf-keycrypt with different arguments/order"; + string => "Test cf-secret with different arguments/order"; vars: @@ -32,13 +32,13 @@ bundle agent test edit_line => insert_lines( "$(text)" ); commands: - "$(sys.cf_keycrypt)" + "$(sys.cf_secret)" args => "encrypt --key $(sys.workdir)/ppkeys/localhost.pub --output $(G.testfile).encrypted $(G.testfile).plain"; - "$(sys.cf_keycrypt)" + "$(sys.cf_secret)" args => "decrypt --key $(sys.workdir)/ppkeys/localhost.priv -o $(G.testfile).decrypted $(G.testfile).encrypted"; reports: "Binaries/folders:"; - "$(sys.cf_keycrypt)"; + "$(sys.cf_secret)"; "$(sys.cf_agent)"; "$(sys.workdir)"; } diff --git a/tests/acceptance/27_cf-keycrypt/encrypt-decrypt.cf b/tests/acceptance/27_cf-secret/encrypt-decrypt.cf similarity index 88% rename from tests/acceptance/27_cf-keycrypt/encrypt-decrypt.cf rename to tests/acceptance/27_cf-secret/encrypt-decrypt.cf index 29e217f3a3..25fb244d8d 100644 --- a/tests/acceptance/27_cf-keycrypt/encrypt-decrypt.cf +++ b/tests/acceptance/27_cf-secret/encrypt-decrypt.cf @@ -19,7 +19,7 @@ bundle agent test { meta: "description" - string => "Test that cf-keycrypt basic key based encryption and decryption work"; + string => "Test that cf-secret basic key based encryption and decryption work"; vars: @@ -33,14 +33,14 @@ bundle agent test edit_line => insert_lines( "$(text)" ); commands: - "$(sys.cf_keycrypt)" + "$(sys.cf_secret)" args => "encrypt -k $(sys.workdir)/ppkeys/localhost.pub -o $(G.testfile).encrypted $(G.testfile).plain"; - "$(sys.cf_keycrypt)" + "$(sys.cf_secret)" args => "decrypt -k $(sys.workdir)/ppkeys/localhost.priv -o $(G.testfile).decrypted $(G.testfile).encrypted"; reports: "Binaries/folders:"; - "$(sys.cf_keycrypt)"; + "$(sys.cf_secret)"; "$(sys.cf_agent)"; "$(sys.workdir)"; } diff --git a/tests/acceptance/27_cf-keycrypt/encrypt.x.cf b/tests/acceptance/27_cf-secret/encrypt.x.cf similarity index 83% rename from tests/acceptance/27_cf-keycrypt/encrypt.x.cf rename to tests/acceptance/27_cf-secret/encrypt.x.cf index fe6b017a9a..dca6dfe767 100644 --- a/tests/acceptance/27_cf-keycrypt/encrypt.x.cf +++ b/tests/acceptance/27_cf-secret/encrypt.x.cf @@ -9,12 +9,12 @@ bundle agent test { meta: "description" - string => "Test that cf-keycrypt encryption uses random padding"; + string => "Test that cf-secret encryption uses random padding"; commands: - "$(sys.cf_keycrypt)" + "$(sys.cf_secret)" args => "encrypt -k $(this.promise_dirname)/testkey.pub -o $(G.testfile) $(this.promise_dirname)/plaintext"; - "$(sys.cf_keycrypt)" + "$(sys.cf_secret)" args => "encrypt -k $(this.promise_dirname)/testkey.pub -o $(G.testfile).2 $(this.promise_dirname)/plaintext"; } diff --git a/tests/acceptance/27_cf-keycrypt/encrypted b/tests/acceptance/27_cf-secret/encrypted similarity index 100% rename from tests/acceptance/27_cf-keycrypt/encrypted rename to tests/acceptance/27_cf-secret/encrypted diff --git a/tests/acceptance/27_cf-keycrypt/plaintext b/tests/acceptance/27_cf-secret/plaintext similarity index 100% rename from tests/acceptance/27_cf-keycrypt/plaintext rename to tests/acceptance/27_cf-secret/plaintext diff --git a/tests/acceptance/27_cf-keycrypt/testkey.priv b/tests/acceptance/27_cf-secret/testkey.priv similarity index 100% rename from tests/acceptance/27_cf-keycrypt/testkey.priv rename to tests/acceptance/27_cf-secret/testkey.priv diff --git a/tests/acceptance/27_cf-keycrypt/testkey.pub b/tests/acceptance/27_cf-secret/testkey.pub similarity index 100% rename from tests/acceptance/27_cf-keycrypt/testkey.pub rename to tests/acceptance/27_cf-secret/testkey.pub diff --git a/tests/acceptance/testall b/tests/acceptance/testall index 807db54974..43c462724b 100755 --- a/tests/acceptance/testall +++ b/tests/acceptance/testall @@ -208,7 +208,7 @@ unix_seconds() { } usage() { - echo "testall [-h|--help] [-q|--quiet] [--gainroot=] [--agent=] [--cfpromises=] [--cfserverd=] [--cfexecd=] [--cfkey=] [--cfkeycrypt=] [--cfnet=] [--cfcheck=] [--bindir=] [--tests=...] [--gdb] [--printlog] [ ...]" + echo "testall [-h|--help] [-q|--quiet] [--gainroot=] [--agent=] [--cfpromises=] [--cfserverd=] [--cfexecd=] [--cfkey=] [--cfkeycrypt=] [--cfnet=] [--cfcheck=] [--bindir=] [--tests=...] [--gdb] [--printlog] [ ...]" echo echo "If no test is given, all standard tests are run:" echo " Tests with names of form .cf are expected to run successfully" @@ -238,7 +238,7 @@ usage() { echo " and defaults to $DEFCF_EXECD." echo " --cfkey provides a way to specify non-default cf-key location," echo " and defaults to $DEFCF_KEY." - echo " --cfkeycrypt provides a way to specify non-default cf-keycrypt location," + echo " --cfkeycrypt provides a way to specify non-default cf-secret location," echo " and defaults to $DEFCF_KEYCRYPT." echo " --cfnet provides a way to specify non-default cf-net location," echo " and defaults to $DEFCF_NET." @@ -949,7 +949,7 @@ find_default_binary DEFCF_PROMISES cf-promises find_default_binary DEFCF_SERVERD cf-serverd find_default_binary DEFCF_EXECD cf-execd find_default_binary DEFCF_KEY cf-key -find_default_binary DEFCF_KEYCRYPT cf-keycrypt +find_default_binary DEFCF_KEYCRYPT cf-secret find_default_binary DEFCF_NET cf-net find_default_binary DEFCF_CHECK cf-check find_default_binary DEFCF_RUNAGENT cf-runagent From bfd36ab4218b364493faabe321583b818265da51 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 10 Apr 2020 14:44:25 +0200 Subject: [PATCH 106/333] Add a new print-headers command to cf-secret Useful for debugging and potentially much more in the future. (cherry picked from commit fb5629ac7e25234ec1e1e503315ab1a57c6ee87f) --- cf-secret/cf-secret.c | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/cf-secret/cf-secret.c b/cf-secret/cf-secret.c index 6a330f8358..b2c855c108 100644 --- a/cf-secret/cf-secret.c +++ b/cf-secret/cf-secret.c @@ -125,6 +125,7 @@ static const Description COMMANDS[] = { {"encrypt", "Encrypt data for one or more hosts/keys", "cf-secret encrypt -k/-H KEY/HOST -o OUTPUT INPUT"}, {"decrypt", "Decrypt data", "cf-secret decrypt [-k/-H KEY/HOST] -o OUTPUT INPUT"}, + {"print-headers", "Print headers from an encrypted file", "cf-secret print-headers ENCRYPTED_FILE"}, {NULL, NULL, NULL} }; @@ -824,6 +825,7 @@ int main(int argc, char *argv[]) char *host_arg = NULL; bool encrypt = false; bool decrypt = false; + bool print_headers = false; size_t offset = 0; if (StringSafeEqual(argv[1], "encrypt")) @@ -836,6 +838,11 @@ int main(int argc, char *argv[]) offset++; decrypt = true; } + else if (StringSafeEqual(argv[1], "print-headers")) + { + print_headers = true; + offset++; + } int c = 0; while ((c = getopt_long(argc - offset, argv + offset, "hMedk:o:H:", OPTIONS, NULL)) != -1) @@ -881,9 +888,9 @@ int main(int argc, char *argv[]) } } - if (!(decrypt || encrypt)) + if (!(decrypt || encrypt || print_headers)) { - printf("Command required. Specify either 'encrypt' or 'decrypt'\n"); + printf("Command required. Specify either 'encrypt', 'decrypt' or 'print-headers'\n"); CFKeyCryptHelp(); exit(EXIT_FAILURE); } @@ -899,6 +906,26 @@ int main(int argc, char *argv[]) exit(EXIT_FAILURE); } + if (print_headers) + { + FILE *input_file = OpenInputOutput(input_path, "r"); + char key[MAX_HEADER_KEY_LEN + 1]; + char value[MAX_HEADER_VAL_LEN + 1]; + + while (ParseHeader(input_file, key, value)) + { + Log(LOG_LEVEL_DEBUG, "Parsed header '%s: %s'", key, value); + if (!CheckHeader(key, value)) + { + fclose(input_file); + exit(EXIT_FAILURE); + } + printf("%s: %s\n", key, value); + } + fclose(input_file); + exit(EXIT_SUCCESS); + } + if (decrypt && (host_arg == NULL) && (key_path_arg == NULL)) { /* Decryption requires a private key which is usually only available for From e0a035b3c90bf6c1b911e906c43bca6bf67df48e Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Mon, 1 Jun 2020 17:01:33 +0200 Subject: [PATCH 107/333] Properly detect end of the headers section in 'cf-secret print-headers' Attempt to read the next header after the last one fails because the ParseHeader() function just reads until it hits the maximum limit of characters to read or the key-value delimiter (':'). The check for the end of the headers section of the file needs to be done outside of this function, just like it is done in ParseHeaders() which is part of the decryption process. Ticket: CFE-3368 Changelog: None (cherry picked from commit 8899c2353ee598f9e06d6f97632d4dbc05b29b79) --- cf-secret/cf-secret.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/cf-secret/cf-secret.c b/cf-secret/cf-secret.c index b2c855c108..5ec6e77e99 100644 --- a/cf-secret/cf-secret.c +++ b/cf-secret/cf-secret.c @@ -912,7 +912,8 @@ int main(int argc, char *argv[]) char key[MAX_HEADER_KEY_LEN + 1]; char value[MAX_HEADER_VAL_LEN + 1]; - while (ParseHeader(input_file, key, value)) + bool done = false; + while (!done && ParseHeader(input_file, key, value)) { Log(LOG_LEVEL_DEBUG, "Parsed header '%s: %s'", key, value); if (!CheckHeader(key, value)) @@ -921,6 +922,17 @@ int main(int argc, char *argv[]) exit(EXIT_FAILURE); } printf("%s: %s\n", key, value); + + /* headers are supposed to be terminated by a blank line */ + int next = fgetc(input_file); + if (next == '\n') + { + done = true; + } + else + { + ungetc(next, input_file); + } } fclose(input_file); exit(EXIT_SUCCESS); From 9bd2dfb6bab8b5669e5db742aea64a7b293303c8 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 4 Jun 2020 11:49:10 +0200 Subject: [PATCH 108/333] Replace the last few remaining occurences of cf-keycrypt cf-keycrypt was renamed to cf-secret in cfengine/core@b93fce47cf. (cherry picked from commit 811fff1f1e0bc8cd37edbadbbbec26e02a69310b) --- cf-secret/cf-secret.c | 10 +++++----- tests/acceptance/testall | 18 +++++++++--------- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/cf-secret/cf-secret.c b/cf-secret/cf-secret.c index 5ec6e77e99..4eacb34703 100644 --- a/cf-secret/cf-secret.c +++ b/cf-secret/cf-secret.c @@ -73,7 +73,7 @@ typedef enum { /* see README.md for details about the format */ typedef enum { - CF_KEYCRYPT_FORMAT_V_1_0, + CF_SECRET_FORMAT_V_1_0, } CFKeyCryptFormatVersion; static const char passphrase[] = "Cfengine passphrase"; @@ -82,10 +82,10 @@ static const char passphrase[] = "Cfengine passphrase"; // DOCUMENTATION / GETOPT CONSTS: //******************************************************************* -static const char *const CF_KEYCRYPT_SHORT_DESCRIPTION = +static const char *const CF_SECRET_SHORT_DESCRIPTION = "cf-secret: Use CFEngine cryptographic keys to encrypt and decrypt files"; -static const char *const CF_KEYCRYPT_MANPAGE_LONG_DESCRIPTION = +static const char *const CF_SECRET_MANPAGE_LONG_DESCRIPTION = "cf-secret offers a simple way to encrypt or decrypt files using keys " "generated by cf-key. CFEngine uses asymmetric cryptography, and " "cf-secret allows you to encrypt a file using a public key file. " @@ -804,8 +804,8 @@ void CFKeyCryptMan() { Writer *out = FileWriter(stdout); ManPageWrite(out, "cf-secret", time(NULL), - CF_KEYCRYPT_SHORT_DESCRIPTION, - CF_KEYCRYPT_MANPAGE_LONG_DESCRIPTION, + CF_SECRET_SHORT_DESCRIPTION, + CF_SECRET_MANPAGE_LONG_DESCRIPTION, OPTIONS, HINTS, COMMANDS, true, true); FileWriterDetach(out); } diff --git a/tests/acceptance/testall b/tests/acceptance/testall index 43c462724b..907d3324ac 100755 --- a/tests/acceptance/testall +++ b/tests/acceptance/testall @@ -100,7 +100,7 @@ CF_PROMISES=${CF_PROMISES:-} ; export CF_PROMISES CF_SERVERD=${CF_SERVERD:-} ; export CF_SERVERD CF_EXECD=${CF_EXECD:-} ; export CF_EXECD CF_KEY=${CF_KEY:-} ; export CF_KEY -CF_KEYCRYPT=${CF_KEYCRYPT:-} ; export CF_KEYCRYPT +CF_SECRET=${CF_SECRET:-} ; export CF_SECRET CF_NET=${CF_NET:-} ; export CF_NET CF_CHECK=${CF_CHECK:-} ; export CF_CHECK CF_RUNAGENT=${CF_RUNAGENT:-} ; export CF_RUNAGENT @@ -239,7 +239,7 @@ usage() { echo " --cfkey provides a way to specify non-default cf-key location," echo " and defaults to $DEFCF_KEY." echo " --cfkeycrypt provides a way to specify non-default cf-secret location," - echo " and defaults to $DEFCF_KEYCRYPT." + echo " and defaults to $DEFCF_SECRET." echo " --cfnet provides a way to specify non-default cf-net location," echo " and defaults to $DEFCF_NET." echo " --cfcheck provides a way to specify non-default cf-check location," @@ -425,7 +425,7 @@ runtest() { $LN_CMD "$CF_SERVERD" "$WORKDIR/bin" $LN_CMD "$CF_EXECD" "$WORKDIR/bin" $LN_CMD "$CF_KEY" "$WORKDIR/bin" - $LN_CMD "$CF_KEYCRYPT" "$WORKDIR/bin" + $LN_CMD "$CF_SECRET" "$WORKDIR/bin" $LN_CMD "$CF_NET" "$WORKDIR/bin" $LN_CMD "$CF_CHECK" "$WORKDIR/bin" $LN_CMD "$CF_RUNAGENT" "$WORKDIR/bin" @@ -832,7 +832,7 @@ do --cfkey=*) CF_KEY=${1#--cfkey=};; --cfkeycrypt=*) - CF_KEYCRYPT=${1#--cfkeycrypt=};; + CF_SECRET=${1#--cfkeycrypt=};; --cfnet=*) CF_NET=${1#--cfnet=};; --cfcheck=*) @@ -920,7 +920,7 @@ do esac done -if [ -n "$AGENT" -o -n "$CF_PROMISES" -o -n "$CF_SERVERD" -o -n "$CF_EXECD" -o -n "$CF_KEY" -o -n "$CF_KEYCRYPT" -o -n "$CF_NET" -o -n "$CF_CHECK" -o -n "$CF_RUNAGENT" -o -n "$RPMVERCMP" ] +if [ -n "$AGENT" -o -n "$CF_PROMISES" -o -n "$CF_SERVERD" -o -n "$CF_EXECD" -o -n "$CF_KEY" -o -n "$CF_SECRET" -o -n "$CF_NET" -o -n "$CF_CHECK" -o -n "$CF_RUNAGENT" -o -n "$RPMVERCMP" ] then if [ -n "$BINDIR" ] then @@ -949,7 +949,7 @@ find_default_binary DEFCF_PROMISES cf-promises find_default_binary DEFCF_SERVERD cf-serverd find_default_binary DEFCF_EXECD cf-execd find_default_binary DEFCF_KEY cf-key -find_default_binary DEFCF_KEYCRYPT cf-secret +find_default_binary DEFCF_SECRET cf-secret find_default_binary DEFCF_NET cf-net find_default_binary DEFCF_CHECK cf-check find_default_binary DEFCF_RUNAGENT cf-runagent @@ -963,14 +963,14 @@ CF_PROMISES=${CF_PROMISES:-${DEFCF_PROMISES}} CF_SERVERD=${CF_SERVERD:-${DEFCF_SERVERD}} CF_EXECD=${CF_EXECD:-${DEFCF_EXECD}} CF_KEY=${CF_KEY:-${DEFCF_KEY}} -CF_KEYCRYPT=${CF_KEYCRYPT:-${DEFCF_KEYCRYPT}} +CF_SECRET=${CF_SECRET:-${DEFCF_SECRET}} CF_NET=${CF_NET:-${DEFCF_NET}} CF_CHECK=${CF_CHECK:-${DEFCF_CHECK}} CF_RUNAGENT=${CF_RUNAGENT:-${DEFCF_RUNAGENT}} RPMVERCMP=${RPMVERCMP:-${DEFRPMVERCMP}} LIBTOOL=${LIBTOOL:-${DEFLIBTOOL}} -if [ ! -x "$AGENT" -o ! -x "$CF_PROMISES" -o ! -x "$CF_SERVERD" -o ! -x "$CF_EXECD" -o ! -x "$CF_KEY" -o ! -x "$CF_KEYCRYPT" -o ! -x "$CF_NET" -o ! -x "$CF_CHECK" -o ! -x "$CF_RUNAGENT" -o ! -x "$RPMVERCMP" ] +if [ ! -x "$AGENT" -o ! -x "$CF_PROMISES" -o ! -x "$CF_SERVERD" -o ! -x "$CF_EXECD" -o ! -x "$CF_KEY" -o ! -x "$CF_SECRET" -o ! -x "$CF_NET" -o ! -x "$CF_CHECK" -o ! -x "$CF_RUNAGENT" -o ! -x "$RPMVERCMP" ] then echo "ERROR: can't find cf-agent or other binary;" \ " Are you sure you're running this from OBJDIR" \ @@ -980,7 +980,7 @@ then echo '$CF_SERVERD =' "$CF_SERVERD" echo '$CF_EXECD =' "$CF_EXECD" echo '$CF_KEY =' "$CF_KEY" - echo '$CF_KEYCRYPT =' "$CF_KEYCRYPT" + echo '$CF_SECRET =' "$CF_SECRET" echo '$CF_RUNAGENT =' "$CF_RUNAGENT" echo '$CF_NET =' "$CF_NET" echo '$CF_CHECK =' "$CF_CHECK" From b3d7c39ad9010a1072d221ff081d215fec1bc281 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Thu, 28 May 2020 11:30:27 -0500 Subject: [PATCH 109/333] LMDB files are now created with correct permissions Permissions will be 0600 as expected by MPF cfe_internal/enterprise/CFE_knowledge.cf Ticket: ENT-5986 Changelog: Title (cherry picked from commit 44130ecba727b943f514d91704f859afc671a34e) --- libpromises/dbm_lmdb.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libpromises/dbm_lmdb.c b/libpromises/dbm_lmdb.c index 4b1df3275d..1c35149e81 100644 --- a/libpromises/dbm_lmdb.c +++ b/libpromises/dbm_lmdb.c @@ -558,7 +558,7 @@ DBPriv *DBPrivOpenDB(const char *const dbpath, const dbid id) open_flags |= MDB_WRITEMAP; #endif - rc = LmdbEnvOpen(db->env, dbpath, open_flags, 0644); + rc = LmdbEnvOpen(db->env, dbpath, open_flags, CF_PERMS_DEFAULT); if (rc) { Log(LOG_LEVEL_ERR, "Could not open database %s: %s", From e821dceabfaf61c5d5e520e08c1c0f6ce4697190 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 27 May 2020 11:08:53 +0200 Subject: [PATCH 110/333] Add a new function for overwriting a value in local DB In some cases, it is necessary to read, compare and potentially overwrite a value in local DB in a single transaction to avoid race conditions. Ticket: CFE-3361 Changelog: None (cherry picked from commit 032bda408054defe6d90eb5c57be33bab68a257e) --- libpromises/Makefile.am | 2 +- libpromises/dbm_api.c | 7 +++++ libpromises/dbm_api.h | 3 +++ libpromises/dbm_api_types.h | 30 +++++++++++++++++++++ libpromises/dbm_lmdb.c | 53 +++++++++++++++++++++++++++++++++++++ libpromises/dbm_priv.h | 6 +++++ libpromises/dbm_quick.c | 47 ++++++++++++++++++++++++++++++++ libpromises/dbm_tokyocab.c | 31 ++++++++++++++++++++++ 8 files changed, 178 insertions(+), 1 deletion(-) create mode 100644 libpromises/dbm_api_types.h diff --git a/libpromises/Makefile.am b/libpromises/Makefile.am index 6fdded5a83..0f7894a2ca 100644 --- a/libpromises/Makefile.am +++ b/libpromises/Makefile.am @@ -70,7 +70,7 @@ libpromises_la_SOURCES = \ constants.c \ conversion.c conversion.h \ crypto.c crypto.h \ - dbm_api.c dbm_api.h dbm_priv.h \ + dbm_api.c dbm_api.h dbm_api_types.h dbm_priv.h \ dbm_migration.c dbm_migration.h \ dbm_migration_lastseen.c \ dbm_lmdb.c \ diff --git a/libpromises/dbm_api.c b/libpromises/dbm_api.c index 71f30e8f5a..a42a8ce8d0 100644 --- a/libpromises/dbm_api.c +++ b/libpromises/dbm_api.c @@ -563,6 +563,13 @@ bool WriteDB(DBHandle *handle, const char *key, const void *src, int srcSz) return DBPrivWrite(handle->priv, key, strlen(key) + 1, src, srcSz); } +bool OverwriteDB(DBHandle *handle, const char *key, const void *value, size_t value_size, + OverwriteCondition Condition, void *data) +{ + assert(handle != NULL); + return DBPrivOverwrite(handle->priv, key, strlen(key) + 1, value, value_size, Condition, data); +} + bool HasKeyDB(DBHandle *handle, const char *key, int key_size) { return DBPrivHasKey(handle->priv, key, key_size); diff --git a/libpromises/dbm_api.h b/libpromises/dbm_api.h index 56a5ba08a3..118b1c915c 100644 --- a/libpromises/dbm_api.h +++ b/libpromises/dbm_api.h @@ -29,6 +29,7 @@ #define EC_CORRUPTION_REPAIR_FAILED 121 #include +#include // Only append to the end, keep in sync with DB_PATHS_STATEDIR array typedef enum @@ -84,6 +85,8 @@ bool WriteComplexKeyDB(CF_DB *dbp, const char *key, int keySz, const void *src, bool DeleteComplexKeyDB(CF_DB *dbp, const char *key, int size); bool ReadDB(CF_DB *dbp, const char *key, void *dest, int destSz); bool WriteDB(CF_DB *dbp, const char *key, const void *src, int srcSz); +bool OverwriteDB(DBHandle *handle, const char *key, const void *value, size_t value_size, + OverwriteCondition Condition, void *data); bool DeleteDB(CF_DB *dbp, const char *key); void FreezeDB(DBHandle *handle); diff --git a/libpromises/dbm_api_types.h b/libpromises/dbm_api_types.h new file mode 100644 index 0000000000..01ed2756a4 --- /dev/null +++ b/libpromises/dbm_api_types.h @@ -0,0 +1,30 @@ +/* + Copyright 2020 Northern.tech AS + + This file is part of CFEngine 3 - written and maintained by Northern.tech AS. + + This program is free software; you can redistribute it and/or modify it + under the terms of the GNU General Public License as published by the + Free Software Foundation; version 3. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA + + To the extent this program is licensed as part of the Enterprise + versions of CFEngine, the applicable Commercial Open Source License + (COSL) may apply to this file if you as a licensee so wish it. See + included file COSL.txt. +*/ + +#ifndef CFENGINE_DBM_API_TYPES_H +#define CFENGINE_DBM_API_TYPES_H + +typedef bool (*OverwriteCondition) (void *value, size_t value_size, void *data); + +#endif /* CFENGINE_DBM_API_TYPES_H */ diff --git a/libpromises/dbm_lmdb.c b/libpromises/dbm_lmdb.c index 1c35149e81..f4d243011c 100644 --- a/libpromises/dbm_lmdb.c +++ b/libpromises/dbm_lmdb.c @@ -827,6 +827,59 @@ bool DBPrivWrite( return (rc == MDB_SUCCESS); } +bool DBPrivOverwrite(DBPriv *db, const char *key, int key_size, const void *value, size_t value_size, + OverwriteCondition Condition, void *data) +{ + assert(db != NULL); + assert(key_size >= 0); + DBTxn *txn; + int rc = GetWriteTransaction(db, &txn); + + if (rc != MDB_SUCCESS) + { + return false; + } + + assert(txn != NULL); + assert(!txn->cursor_open); + + MDB_val mkey, orig_data; + mkey.mv_data = (void *) key; + mkey.mv_size = key_size; + rc = mdb_get(txn->txn, db->dbi, &mkey, &orig_data); + CheckLMDBCorrupted(rc, db->env); + if ((rc != MDB_SUCCESS) && (rc != MDB_NOTFOUND)) + { + Log(LOG_LEVEL_ERR, "Could not read database entry from '%s': %s", + (char *) mdb_env_get_userctx(db->env), mdb_strerror(rc)); + AbortTransaction(db); + return false; + } + + void *cur_val = (rc == MDB_SUCCESS ? orig_data.mv_data : NULL); + size_t cur_val_size = (rc == MDB_SUCCESS ? orig_data.mv_size : 0); + if ((Condition != NULL) && !Condition(cur_val, cur_val_size, data)) + { + AbortTransaction(db); + return false; + } + + MDB_val new_data; + new_data.mv_data = (void *)value; + new_data.mv_size = value_size; + rc = mdb_put(txn->txn, db->dbi, &mkey, &new_data, 0); + CheckLMDBCorrupted(rc, db->env); + if (rc != MDB_SUCCESS) + { + Log(LOG_LEVEL_ERR, "Could not write database entry to '%s': %s", + (char *) mdb_env_get_userctx(db->env), mdb_strerror(rc)); + AbortTransaction(db); + return false; + } + DBPrivCommit(db); + return true; +} + bool DBPrivDelete(DBPriv *const db, const void *const key, const int key_size) { assert(key_size >= 0); diff --git a/libpromises/dbm_priv.h b/libpromises/dbm_priv.h index 1d1ada2dd7..604b0f5a2a 100644 --- a/libpromises/dbm_priv.h +++ b/libpromises/dbm_priv.h @@ -25,6 +25,7 @@ #ifndef CFENGINE_DBM_PRIV_H #define CFENGINE_DBM_PRIV_H +#include /* DBM implementation is supposed to define the following structures and * implement the following functions */ @@ -62,6 +63,11 @@ bool DBPrivRead(DBPriv *db, const void *key, int key_size, bool DBPrivWrite(DBPriv *db, const void *key, int key_size, const void *value, int value_size); +bool DBPrivOverwrite(DBPriv *handle, + const char *key, int key_size, + const void *value, size_t value_size, + OverwriteCondition Condition, void *data); + bool DBPrivDelete(DBPriv *db, const void *key, int key_size); diff --git a/libpromises/dbm_quick.c b/libpromises/dbm_quick.c index 1ca01a4066..57823dc729 100644 --- a/libpromises/dbm_quick.c +++ b/libpromises/dbm_quick.c @@ -249,6 +249,53 @@ bool DBPrivWrite(DBPriv *db, const void *key, int key_size, const void *value, i return true; } +bool DBPrivOverwrite(DBPriv *db, const void *key, int key_size, const void *value, int value_size, + OverwriteCondition *Condition, void *data) +{ + if (!Lock(db)) + { + return false; + } + + ssize_t cur_val_size = dpvsiz(db->depot, key, key_size); + bool exists = (cur_val_size != -1); + + void *cur_val = NULL; + if (exists) + { + assert(cur_val_size > 0); + cur_val = xmalloc((size_t) cur_val_size); + + if (dpgetwb(db->depot, key, key_size, 0, value_size, cur_val) == -1) + { + Log(LOG_LEVEL_DEBUG, "QDBM DBPrivRead: Could not read '%s', (dpgetwb: %s)", + (const char *)key, dperrmsg(dpecode)); + + Unlock(db); + return false; + } + } + if ((Condition != NULL) && !Condition(cur_val, cur_val_size, data)) + { + free(cur_val); + Unlock(db); + return false; + } + free(cur_val); + + if (!dpput(db->depot, key, key_size, value, value_size, DP_DOVER)) + { + char *db_name = dpname(db->depot); + Log(LOG_LEVEL_ERR, "Could not write key to DB '%s'. (dpput: %s)", + db_name, dperrmsg(dpecode)); + free(db_name); + Unlock(db); + return false; + } + Unlock(db); + return true; +} + bool DBPrivHasKey(DBPriv *db, const void *key, int key_size) { if (!Lock(db)) diff --git a/libpromises/dbm_tokyocab.c b/libpromises/dbm_tokyocab.c index 08c87d308d..81be2442a7 100644 --- a/libpromises/dbm_tokyocab.c +++ b/libpromises/dbm_tokyocab.c @@ -307,6 +307,37 @@ bool DBPrivWrite(DBPriv *db, const void *key, int key_size, const void *value, i return ret; } +bool DBPrivOverwrite(DBPriv *db, const void *key, int key_size, const void *value, int value_size, + OverwriteCondition *Condition, void *data) +{ + ssize_t cur_val_size = tchdbvsiz(db->hdb, key, key_size); + void *cur_val = NULL; + bool exists = (cur_val_size > 0); + + if (exists) + { + assert(cur_val_size > 0); + cur_val = xmalloc(cur_val_size); + if (tchdbget3(db->hdb, key, key_size, dest, dest_size) == -1) + { + /* If exists, we should never get the TCENOREC error. */ + assert(tchdbecode(db->hdb) != TCENOREC); + + Log(LOG_LEVEL_ERR, "Could not read key '%s': (tchdbget3: %s)", (const char *)key, ErrorMessage(db->hdb)); + free(cur_val); + return false; + } + } + if ((Condition != NULL) && !Condition(cur_val, cur_val_size, data)) + { + free(cur_val); + return false; + } + free(cur_val); + + return Write(db->hdb, key, key_size, value, value_size); +} + /* * This one has to be locked against cursor -- deleting entries might interrupt * iteration. From 58e3697a3394a78c155596b2ce632068b0f518c6 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 27 May 2020 11:21:51 +0200 Subject: [PATCH 111/333] Remove race condition in WaitForCriticalSection() We need to check the timestamp of the last lock and potentially overwrite it (acquire the lock) in a transaction, otherwise there is a race condition where multiple processes can overwrite each other's locks like this: procA: check lock => can acquire lock procB: check lock => can acquire lock procB: acquire lock procA: acquire lock resulting in two processes being in the critical section at the same time. Ticket: CFE-3361 Changelog: Fixed race condition when multiple agents are acquiring critical section locks simultaneously (cherry picked from commit 6cfd1194fea5c2efd8b24b58fdc72f6f3d22d8be) --- libpromises/locks.c | 89 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 79 insertions(+), 10 deletions(-) diff --git a/libpromises/locks.c b/libpromises/locks.c index fa7a905161..9905e6cbde 100644 --- a/libpromises/locks.c +++ b/libpromises/locks.c @@ -365,27 +365,96 @@ static int RemoveLock(const char *name) return 0; } +static bool NoOrObsoleteLock(LockData *entry, ARG_UNUSED size_t entry_size, size_t *max_old) +{ + assert((entry == NULL) || (entry_size == sizeof(LockData))); + + if (entry == NULL) + { + return true; + } + + time_t now = time(NULL); + if ((now - entry->time) <= *max_old) + { + Log(LOG_LEVEL_DEBUG, "Giving time to process '%d' (holding lock for %ld s)", entry->pid, (now - entry->time)); + } + return ((now - entry->time) > *max_old); +} + void WaitForCriticalSection(const char *section_id) { - time_t now = time(NULL), then = FindLockTime(section_id); + ThreadLock(cft_lock); + + CF_DB *dbp = OpenLock(); + if (dbp == NULL) + { + Log(LOG_LEVEL_CRIT, "Failed to open lock database when waiting for critical section"); + ThreadUnlock(cft_lock); + return; + } + + time_t started = time(NULL); + LockData entry = { 0 }; + entry.pid = getpid(); + entry.process_start_time = PROCESS_START_TIME_UNKNOWN; -/* Another agent has been waiting more than a minute, it means there - is likely crash detritus to clear up... After a minute we take our - chances ... */ +#ifdef LMDB + unsigned char ohash[LMDB_MAX_KEY_SIZE]; + HashLockKeyIfNecessary(section_id, ohash); + Log(LOG_LEVEL_DEBUG, "Hashed critical section lock '%s' to '%s'", section_id, ohash); + section_id = ohash; +#endif - while ((then != -1) && (now - then < 60)) + /* If another agent has been waiting more than a minute, it means there + is likely crash detritus to clear up... After a minute we take our + chances ... */ + size_t max_old = 60; + + Log(LOG_LEVEL_DEBUG, "Acquiring critical section lock '%s'", section_id); + bool got_lock = false; + while (!got_lock && ((time(NULL) - started) <= max_old)) { - sleep(1); - now = time(NULL); - then = FindLockTime(section_id); + entry.time = time(NULL); + got_lock = OverwriteDB(dbp, section_id, &entry, sizeof(entry), + (OverwriteCondition) NoOrObsoleteLock, &max_old); + if (!got_lock) + { + Log(LOG_LEVEL_DEBUG, "Waiting for critical section lock '%s'", section_id); + sleep(1); + } } - WriteLock(section_id); + /* If we still haven't gotten the lock, let's try the biggest hammer we + * have. */ + if (!got_lock) + { + Log(LOG_LEVEL_DEBUG, "Failed to wait for critical section lock '%s', force-writing new lock", section_id); + if (!WriteDB(dbp, section_id, &entry, sizeof(entry))) + { + Log(LOG_LEVEL_CRIT, "Failed to force-write critical section lock '%s'", section_id); + } + } + else + { + Log(LOG_LEVEL_DEBUG, "Acquired critical section lock '%s'", section_id); + } + + CloseLock(dbp); + ThreadUnlock(cft_lock); } void ReleaseCriticalSection(const char *section_id) { - RemoveLock(section_id); + Log(LOG_LEVEL_DEBUG, "Releasing critical section lock '%s'", section_id); + if (RemoveLock(section_id) == 0) + { + Log(LOG_LEVEL_DEBUG, "Released critical section lock '%s'", section_id); + } + else + { + Log(LOG_LEVEL_DEBUG, "Failed to release critical section lock '%s'", section_id); + } } static time_t FindLock(char *last) From e188588575601a0186c82a18b14389472eb7a286 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 29 May 2020 15:32:26 +0200 Subject: [PATCH 112/333] Add an easy way to debug the ifelapsed_and_expireafter/timed/defaults.cf test This test is very complex even though it only uses a couple lines of policy. It is also time-sensitive so an easy way to debug what's going on comes handy. Ticket: CFE-3361 Changelog: None (cherry picked from commit ae0e6940a9071765bf5d4d1f9844c9720e5f612b) --- .../ifelapsed_and_expireafter/timed/defaults.cf | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/defaults.cf b/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/defaults.cf index cc699ef600..fef37de68a 100644 --- a/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/defaults.cf +++ b/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/defaults.cf @@ -55,9 +55,19 @@ bundle agent test }; commands: - test_pass_1|test_pass_2|test_pass_3:: + !DEBUG:: "$(G.no_fds) $(sys.cf_agent) -Il -D $(cases) -f $(this.promise_filename).sub 1>>$(G.testfile).$(cases).output.log 2>&1 &" contain => in_shell; + + DEBUG.test_pass_1:: + "$(G.no_fds) $(sys.cf_agent) -l --debug -D $(cases) -f $(this.promise_filename).sub 1>>$(G.testfile).$(cases).1.output.log 2>&1 & echo $!" + contain => in_shell; + DEBUG.test_pass_2:: + "$(G.no_fds) $(sys.cf_agent) -l --debug -D $(cases) -f $(this.promise_filename).sub 1>>$(G.testfile).$(cases).2.output.log 2>&1 & echo $!" + contain => in_shell; + DEBUG.test_pass_3:: + "$(G.no_fds) $(sys.cf_agent) -l --debug -D $(cases) -f $(this.promise_filename).sub 1>>$(G.testfile).$(cases).3.output.log 2>&1 & echo $!" + contain => in_shell; } bundle agent check From f3b681c6a8f803227a06e31aa556ed07fd104d53 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 4 Jun 2020 17:37:29 +0200 Subject: [PATCH 113/333] Create a copy of mmap-ed LMDB data to ensure correct alignment mdb_get() sets up a pointer to the mmap-ed chunk of memory where proper alignment is not guaranteed. Trying to access the memory directly with multi-byte types can cause alignment failures. Let's create a local copy of the data to ensure correct alignment before we access it. Ticket: CFE-3361 Changelog: None (cherry picked from commit dc5d539fa97de147ee9a9768f074850b428c7d96) --- libpromises/dbm_lmdb.c | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-) diff --git a/libpromises/dbm_lmdb.c b/libpromises/dbm_lmdb.c index f4d243011c..adedbf2fa9 100644 --- a/libpromises/dbm_lmdb.c +++ b/libpromises/dbm_lmdb.c @@ -856,12 +856,32 @@ bool DBPrivOverwrite(DBPriv *db, const char *key, int key_size, const void *valu return false; } - void *cur_val = (rc == MDB_SUCCESS ? orig_data.mv_data : NULL); - size_t cur_val_size = (rc == MDB_SUCCESS ? orig_data.mv_size : 0); - if ((Condition != NULL) && !Condition(cur_val, cur_val_size, data)) + if (Condition != NULL) { - AbortTransaction(db); - return false; + if (rc == MDB_SUCCESS) + { + assert(orig_data.mv_size > 0); + + /* We have to copy the data because orig_data.mv_data is a pointer to + * the mmap()-ed area which can potentially have bad alignment causing + * a SIGBUS on some architectures. */ + unsigned char cur_val[orig_data.mv_size]; + memcpy(cur_val, orig_data.mv_data, orig_data.mv_size); + if (!Condition(cur_val, orig_data.mv_size, data)) + { + AbortTransaction(db); + return false; + } + } + else + { + assert(rc == MDB_NOTFOUND); + if (!Condition(NULL, 0, data)) + { + AbortTransaction(db); + return false; + } + } } MDB_val new_data; From 9f39d5e07583b33c26889703576298b2b8781da5 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 10 Jun 2020 13:45:47 +0200 Subject: [PATCH 114/333] Log a NOTICE-level message when force-writing critical section lock This is something that should ideally never happen and if it does, it should definitely be logged. At the same time, it is not exactly an error in the process that logs it because it is just dealing with some trash left behind by some other process. Still a message worth noticing, though. Ticket: CFE-3361 Changelog: None (cherry picked from commit 4c1d14bedfdbce128f7bf89261335e994a4debe0) --- libpromises/locks.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libpromises/locks.c b/libpromises/locks.c index 9905e6cbde..c6f71426b8 100644 --- a/libpromises/locks.c +++ b/libpromises/locks.c @@ -429,7 +429,7 @@ void WaitForCriticalSection(const char *section_id) * have. */ if (!got_lock) { - Log(LOG_LEVEL_DEBUG, "Failed to wait for critical section lock '%s', force-writing new lock", section_id); + Log(LOG_LEVEL_NOTICE, "Failed to wait for critical section lock '%s', force-writing new lock", section_id); if (!WriteDB(dbp, section_id, &entry, sizeof(entry))) { Log(LOG_LEVEL_CRIT, "Failed to force-write critical section lock '%s'", section_id); From 1f8caab225e6b2118e7a16bc2ca1e244132a321f Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Mon, 15 Jun 2020 13:14:58 -0500 Subject: [PATCH 115/333] Adjusted ifelse/ENT-4653 test to report properly Ticket: ENT-4653 Changelog: none (cherry picked from commit 501d2e72f7d63a4f6a930b92438cc2d4fef04739) --- .../01_vars/02_functions/ifelse_isvariable-ENT-4653.cf | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/tests/acceptance/01_vars/02_functions/ifelse_isvariable-ENT-4653.cf b/tests/acceptance/01_vars/02_functions/ifelse_isvariable-ENT-4653.cf index c7f569ec59..294d77d8a8 100644 --- a/tests/acceptance/01_vars/02_functions/ifelse_isvariable-ENT-4653.cf +++ b/tests/acceptance/01_vars/02_functions/ifelse_isvariable-ENT-4653.cf @@ -11,9 +11,6 @@ bundle agent test "description" -> { "ENT-4653" } string => "Test that ifelse can use the result of isvariable with 3 arguments even when they contain variables that don't resolve"; - "test_soft_fail" string => "any", - meta => { "ENT-4653" }; - vars: "lookup" string => "THIS_IS_NOT_A_DEFINED_VARIABLE"; @@ -27,12 +24,12 @@ bundle agent check { reports: - 'PASS' + '$(this.promise_filename) Pass' if => strcmp( "FALLBACK", $(test.test) ) ; - 'FAIL' + '$(this.promise_filename) FAIL' if => not( isvariable( "test.test" ) ); - 'FAIL' + '$(this.promise_filename) FAIL' if => not( strcmp( "FALLBACK", $(test.test) ) ); } From 79df1c77757c82c314f8f4458085549015f87d9e Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Thu, 18 Jun 2020 14:02:45 +0200 Subject: [PATCH 116/333] Renamed StringSafeEqual() to StringEqual() Backported commit: a60798f53c0297277fe38c6e6b44952f29529b33 Changelog: None Ticket: None Signed-off-by: Ole Herman Schumacher Elgesem --- cf-agent/cf-agent.c | 14 +++++++------- cf-agent/files_editline.c | 22 +++++++++++----------- cf-agent/nfs.c | 4 ++-- cf-agent/package_module.c | 24 ++++++++++++------------ cf-agent/retcode.c | 2 +- cf-agent/verify_packages.c | 8 ++++---- cf-check/cf-check.c | 16 ++++++++-------- cf-check/dump.c | 6 +++--- cf-check/validate.c | 4 ++-- cf-execd/cf-execd.c | 6 +++--- cf-secret/cf-secret.c | 22 +++++++++++----------- cf-testd/cf-testd.c | 4 ++-- libcfnet/conn_cache.c | 6 +++--- libcfnet/protocol.c | 4 ++-- libcfnet/protocol_version.c | 8 ++++---- libcfnet/tls_generic.c | 2 +- libenv/sysinfo.c | 2 +- libntech | 2 +- libpromises/class.c | 2 +- libpromises/dbm_api.c | 4 ++-- libpromises/eval_context.c | 10 +++++----- libpromises/evalfunction.c | 12 ++++++------ libpromises/generic_agent.c | 8 ++++---- libpromises/iteration.c | 2 +- libpromises/loading.c | 4 ++-- libpromises/mod_methods.c | 2 +- libpromises/policy.c | 2 +- libpromises/rlist.c | 2 +- libpromises/verify_vars.c | 2 +- tests/unit/logging_test.c | 2 +- 30 files changed, 104 insertions(+), 104 deletions(-) diff --git a/cf-agent/cf-agent.c b/cf-agent/cf-agent.c index 4974f08830..4ccb0d0297 100644 --- a/cf-agent/cf-agent.c +++ b/cf-agent/cf-agent.c @@ -595,7 +595,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) case 0: { const char *const option_name = OPTIONS[longopt_idx].name; - if (StringSafeEqual(option_name, "log-modules")) + if (StringEqual(option_name, "log-modules")) { bool ret = LogEnableModulesFromString(optarg); if (!ret) @@ -603,7 +603,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) DoCleanupAndExit(EXIT_FAILURE); } } - else if (StringSafeEqual(option_name, "show-evaluated-classes")) + else if (StringEqual(option_name, "show-evaluated-classes")) { if (optarg == NULL) { @@ -611,7 +611,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) } config->agent_specific.agent.show_evaluated_classes = xstrdup(optarg); } - else if (StringSafeEqual(option_name, "show-evaluated-vars")) + else if (StringEqual(option_name, "show-evaluated-vars")) { if (optarg == NULL) { @@ -619,21 +619,21 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) } config->agent_specific.agent.show_evaluated_variables = xstrdup(optarg); } - else if (StringSafeEqual(option_name, "skip-bootstrap-policy-run")) + else if (StringEqual(option_name, "skip-bootstrap-policy-run")) { config->agent_specific.agent.bootstrap_trigger_policy = false; } - else if (StringSafeEqual(option_name, "skip-db-check")) + else if (StringEqual(option_name, "skip-db-check")) { if (optarg == NULL) { PERFORM_DB_CHECK = false; // Skip (no arg), check = false } - else if (StringSafeEqual_IgnoreCase(optarg, "yes")) + else if (StringEqual_IgnoreCase(optarg, "yes")) { PERFORM_DB_CHECK = false; // Skip = yes, check = false } - else if (StringSafeEqual_IgnoreCase(optarg, "no")) + else if (StringEqual_IgnoreCase(optarg, "no")) { PERFORM_DB_CHECK = true; // Skip = no, check = true } diff --git a/cf-agent/files_editline.c b/cf-agent/files_editline.c index 0d3962f008..6bce616d63 100644 --- a/cf-agent/files_editline.c +++ b/cf-agent/files_editline.c @@ -890,7 +890,7 @@ static bool InsertMultipleLinesToRegion(EvalContext *ctx, Item **start, Item *be const Promise *pp, EditContext *edcontext, PromiseResult *result) { Item *ip, *prev = NULL; - int allow_multi_lines = StringSafeEqual(a->sourcetype, "preserve_all_lines"); + int allow_multi_lines = StringEqual(a->sourcetype, "preserve_all_lines"); // Insert at the start of the file @@ -962,7 +962,7 @@ static bool InsertMultipleLinesAtLocation(EvalContext *ctx, Item **start, Item * { const char *const type = a->sourcetype; - int isfileinsert = StringSafeEqual(type, "file") || StringSafeEqual(type, "file_preserve_block"); + int isfileinsert = StringEqual(type, "file") || StringEqual(type, "file_preserve_block"); if (isfileinsert) { @@ -1137,7 +1137,7 @@ static int ReplacePatterns(EvalContext *ctx, Item *file_start, Item *file_end, c Item *ip; int notfound = true, cutoff = 1, replaced = false; - if (StringSafeEqual(a->replace.occurrences, "first")) + if (StringEqual(a->replace.occurrences, "first")) { Log(LOG_LEVEL_WARNING, "Setting replace-occurrences policy to 'first' is not convergent"); once_only = true; @@ -1330,7 +1330,7 @@ static bool SanityCheckInsertions(const Attributes *a) Rlist *rp; InsertMatchType opt; int exact = false, ignore_something = false; - const bool preserve_block = StringSafeEqual(a->sourcetype, "preserve_block"); + const bool preserve_block = StringEqual(a->sourcetype, "preserve_block"); const LineSelect line_select = a->line_select; if (line_select.startwith_from_list) @@ -1635,7 +1635,7 @@ static bool InsertFileAtLocation(EvalContext *ctx, Item **start, Item *begin_ptr FILE *fin; bool retval = false; Item *loc = NULL; - const bool preserve_block = StringSafeEqual(a->sourcetype, "file_preserve_block"); + const bool preserve_block = StringEqual(a->sourcetype, "file_preserve_block"); if ((fin = safe_fopen(pp->promiser, "rt")) == NULL) { @@ -1732,7 +1732,7 @@ static bool InsertCompoundLineAtLocation(EvalContext *ctx, char *chunk, Item **s { bool retval = false; const char *const type = a->sourcetype; - const bool preserve_all_lines = StringSafeEqual(type, "preserve_all_lines"); + const bool preserve_all_lines = StringEqual(type, "preserve_all_lines"); const bool preserve_block = type && (preserve_all_lines || strcmp(type, "preserve_block") == 0 || strcmp(type, "file_preserve_block") == 0); if (!preserve_all_lines && MatchRegion(ctx, chunk, location, NULL, false)) @@ -1847,7 +1847,7 @@ static bool InsertLineAtLocation(EvalContext *ctx, char *newline, Item **start, /* Check line neighbourhood in whole file to avoid edge effects, iff we are not preseving block structure */ -{ int preserve_block = StringSafeEqual(a->sourcetype, "preserve_block"); +{ int preserve_block = StringEqual(a->sourcetype, "preserve_block"); if (!prev) /* Insert at first line */ { @@ -2226,7 +2226,7 @@ static bool DoEditColumn(Rlist **columns, const Attributes *a, EditContext *edco const char *const column_value = a->column.column_value; const char *const column_operation = a->column.column_operation; - if (StringSafeEqual(column_operation, "delete")) + if (StringEqual(column_operation, "delete")) { while ((found = RlistKeyIn(*columns, column_value))) { @@ -2239,7 +2239,7 @@ static bool DoEditColumn(Rlist **columns, const Attributes *a, EditContext *edco return retval; } - if (StringSafeEqual(column_operation, "set")) + if (StringEqual(column_operation, "set")) { int length = RlistLen(*columns); if (length == 1 && strcmp(RlistScalarValue(*columns), column_value) == 0) @@ -2261,7 +2261,7 @@ static bool DoEditColumn(Rlist **columns, const Attributes *a, EditContext *edco return true; } - if (StringSafeEqual(column_operation, "prepend")) + if (StringEqual(column_operation, "prepend")) { if (RlistPrependScalarIdemp(columns, column_value)) { @@ -2274,7 +2274,7 @@ static bool DoEditColumn(Rlist **columns, const Attributes *a, EditContext *edco } } - if (StringSafeEqual(column_operation, "alphanum")) + if (StringEqual(column_operation, "alphanum")) { if (RlistPrependScalarIdemp(columns, column_value)) { diff --git a/cf-agent/nfs.c b/cf-agent/nfs.c index a518606fb9..be153d33ea 100644 --- a/cf-agent/nfs.c +++ b/cf-agent/nfs.c @@ -644,11 +644,11 @@ PromiseResult VerifyMount(EvalContext *ctx, char *name, const Attributes *a, con PromiseResult result = PROMISE_RESULT_NOOP; if (!DONTDO) { - if (StringSafeEqual(a->mount.mount_type, "panfs")) + if (StringEqual(a->mount.mount_type, "panfs")) { snprintf(comm, CF_BUFSIZE, "%s -t panfs -o %s %s%s %s", CommandArg0(VMOUNTCOMM[VSYSTEMHARDCLASS]), opts, host, rmountpt, mountpt); } - else if (StringSafeEqual(a->mount.mount_type, "cifs")) + else if (StringEqual(a->mount.mount_type, "cifs")) { snprintf(comm, CF_BUFSIZE, "%s -t cifs -o %s %s%s %s", CommandArg0(VMOUNTCOMM[VSYSTEMHARDCLASS]), opts, host, rmountpt, mountpt); } diff --git a/cf-agent/package_module.c b/cf-agent/package_module.c index b3ce6bb22e..c3b07d3d78 100644 --- a/cf-agent/package_module.c +++ b/cf-agent/package_module.c @@ -305,11 +305,11 @@ static PackageInfo *ParseAndCheckPackageDataReply(const Rlist *data) if (StringStartsWith(line, "PackageType=")) { const char *type = line + strlen("PackageType="); - if (StringSafeEqual(type, "file")) + if (StringEqual(type, "file")) { package_data->type = PACKAGE_TYPE_FILE; } - else if (StringSafeEqual(type, "repo")) + else if (StringEqual(type, "repo")) { package_data->type = PACKAGE_TYPE_REPO; } @@ -482,7 +482,7 @@ static void GetPackageModuleExecInfo(const PackageModuleBody *package_module, ch char *package_module_path = NULL; - if (package_module->module_path != NULL && !StringSafeEqual(package_module->module_path, "")) + if (package_module->module_path != NULL && !StringEqual(package_module->module_path, "")) { package_module_path = xstrdup(package_module->module_path); } @@ -493,7 +493,7 @@ static void GetPackageModuleExecInfo(const PackageModuleBody *package_module, ch package_module->name); } - if (package_module->interpreter && !StringSafeEqual(package_module->interpreter, "")) + if (package_module->interpreter && !StringEqual(package_module->interpreter, "")) { *script_path = package_module_path; *script_path_quoted = Path_GetQuoted(*script_path); @@ -528,7 +528,7 @@ static int IsPackageInCache(EvalContext *ctx, /* Handle latest version in specific way for repo packages. * Please note that for file packages 'latest' version is not supported * and check against that is made in CheckPolicyAndPackageInfoMatch(). */ - if (version && StringSafeEqual(version, "latest")) + if (version && StringEqual(version, "latest")) { version = NULL; } @@ -1131,7 +1131,7 @@ PromiseResult RepoInstall(EvalContext *ctx, } const char *version = package_version; - if (StringSafeEqual(version, "latest")) + if (StringEqual(version, "latest")) { Log(LOG_LEVEL_DEBUG, "Clearing latest package version"); version = NULL; @@ -1154,7 +1154,7 @@ PromiseResult RepoInstall(EvalContext *ctx, /* We have 'latest' version in policy. */ - if (StringSafeEqual(package_version, "latest")) + if (StringEqual(package_version, "latest")) { /* This can return more than one latest version if we have packages * with different architectures installed. */ @@ -1182,7 +1182,7 @@ PromiseResult RepoInstall(EvalContext *ctx, * in updates available but we are interested only in updating * package with specific architecture. */ if (package_info->arch && - !StringSafeEqual(package_info->arch, update_package->arch)) + !StringEqual(package_info->arch, update_package->arch)) { Log(LOG_LEVEL_DEBUG, "Skipping update check of package '%s' as updates" @@ -1320,7 +1320,7 @@ static bool CheckPolicyAndPackageInfoMatch(const NewPackages *packages_policy, const PackageInfo *info) { if (packages_policy->package_version && - StringSafeEqual(packages_policy->package_version, "latest")) + StringEqual(packages_policy->package_version, "latest")) { Log(LOG_LEVEL_WARNING, "Unsupported 'latest' version for package " "promise of type file."); @@ -1329,7 +1329,7 @@ static bool CheckPolicyAndPackageInfoMatch(const NewPackages *packages_policy, /* Check if file we are having matches what we want in policy. */ if (info->arch && packages_policy->package_architecture && - !StringSafeEqual(info->arch, packages_policy->package_architecture)) + !StringEqual(info->arch, packages_policy->package_architecture)) { Log(LOG_LEVEL_WARNING, "Package arch and one specified in policy doesn't match: %s -> %s", @@ -1338,7 +1338,7 @@ static bool CheckPolicyAndPackageInfoMatch(const NewPackages *packages_policy, } if (info->version && packages_policy->package_version && - !StringSafeEqual(info->version, packages_policy->package_version)) + !StringEqual(info->version, packages_policy->package_version)) { Log(LOG_LEVEL_WARNING, @@ -1469,7 +1469,7 @@ PromiseResult HandleAbsentPromiseAction(EvalContext *ctx, { /* Check if we are not having 'latest' version. */ if (policy_data->package_version && - StringSafeEqual(policy_data->package_version, "latest")) + StringEqual(policy_data->package_version, "latest")) { Log(LOG_LEVEL_ERR, "Package version 'latest' not supported for" "absent package promise"); diff --git a/cf-agent/retcode.c b/cf-agent/retcode.c index 790c692a17..76c83822ed 100644 --- a/cf-agent/retcode.c +++ b/cf-agent/retcode.c @@ -48,7 +48,7 @@ bool VerifyCommandRetcode(EvalContext *ctx, int retcode, const Attributes *a, co // inform constraint is only for commands promises, // a->inform is actually false for other promise types, so // checking the promise type here is important: - if (StringSafeEqual("commands", pp->parent_promise_type->name) && (!a->inform)) + if (StringEqual("commands", pp->parent_promise_type->name) && (!a->inform)) { // for commands promises which don't make changes to the system, // you can use this to make the log messages verbose: diff --git a/cf-agent/verify_packages.c b/cf-agent/verify_packages.c index fd6c8727ef..f3d41c5297 100644 --- a/cf-agent/verify_packages.c +++ b/cf-agent/verify_packages.c @@ -2763,7 +2763,7 @@ static bool ExecuteSchedule(EvalContext *ctx, const PackageManager *schedule, Pa { bool ok = ExecPackageCommand(ctx, command_string, verify, true, &a, ppi, &result); - if (StringSafeEqual(pi->name, PACKAGE_IGNORED_CFE_INTERNAL)) + if (StringEqual(pi->name, PACKAGE_IGNORED_CFE_INTERNAL)) { Log(LOG_LEVEL_DEBUG, "ExecuteSchedule: Ignoring outcome for special package '%s'", pi->name); } @@ -2827,7 +2827,7 @@ static bool ExecuteSchedule(EvalContext *ctx, const PackageManager *schedule, Pa for (const PackageItem *pi = pm->pack_list; pi != NULL; pi = pi->next) { - if (StringSafeEqual(pi->name, PACKAGE_IGNORED_CFE_INTERNAL)) + if (StringEqual(pi->name, PACKAGE_IGNORED_CFE_INTERNAL)) { Log(LOG_LEVEL_DEBUG, "ExecuteSchedule: Ignoring outcome for special package '%s'", pi->name); } @@ -3009,7 +3009,7 @@ static bool ExecutePatch(EvalContext *ctx, const PackageManager *schedule, Packa { bool ok = ExecPackageCommand(ctx, command_string, false, true, &a, pp, &result); - if (StringSafeEqual(pi->name, PACKAGE_IGNORED_CFE_INTERNAL)) + if (StringEqual(pi->name, PACKAGE_IGNORED_CFE_INTERNAL)) { Log(LOG_LEVEL_DEBUG, "ExecutePatch: Ignoring outcome for special package '%s'", pi->name); } @@ -3052,7 +3052,7 @@ static bool ExecutePatch(EvalContext *ctx, const PackageManager *schedule, Packa for (const PackageItem *pi = pm->patch_list; pi != NULL; pi = pi->next) { - if (StringSafeEqual(pi->name, PACKAGE_IGNORED_CFE_INTERNAL)) + if (StringEqual(pi->name, PACKAGE_IGNORED_CFE_INTERNAL)) { Log(LOG_LEVEL_DEBUG, "ExecutePatch: Ignoring outcome for special package '%s'", pi->name); } diff --git a/cf-check/cf-check.c b/cf-check/cf-check.c index 548ba7fa8b..54c490c658 100644 --- a/cf-check/cf-check.c +++ b/cf-check/cf-check.c @@ -204,28 +204,28 @@ int main(int argc, const char *const *argv) int cmd_argc = argc - optind; const char *command = cmd_argv[0]; - if (StringSafeEqual_IgnoreCase(command, "lmdump")) + if (StringEqual_IgnoreCase(command, "lmdump")) { return lmdump_main(cmd_argc, cmd_argv); } - if (StringSafeEqual_IgnoreCase(command, "dump")) + if (StringEqual_IgnoreCase(command, "dump")) { return dump_main(cmd_argc, cmd_argv); } - if (StringSafeEqual_IgnoreCase(command, "diagnose")) + if (StringEqual_IgnoreCase(command, "diagnose")) { return diagnose_main(cmd_argc, cmd_argv); } - if (StringSafeEqual_IgnoreCase(command, "backup")) + if (StringEqual_IgnoreCase(command, "backup")) { return backup_main(cmd_argc, cmd_argv); } - if (StringSafeEqual_IgnoreCase(command, "repair") || - StringSafeEqual_IgnoreCase(command, "remediate")) + if (StringEqual_IgnoreCase(command, "repair") || + StringEqual_IgnoreCase(command, "remediate")) { return repair_main(cmd_argc, cmd_argv); } - if (StringSafeEqual_IgnoreCase(command, "help")) + if (StringEqual_IgnoreCase(command, "help")) { if (cmd_argc > 2) { @@ -243,7 +243,7 @@ int main(int argc, const char *const *argv) } return EXIT_SUCCESS; } - if (StringSafeEqual_IgnoreCase(command, "version")) + if (StringEqual_IgnoreCase(command, "version")) { print_version(); return EXIT_SUCCESS; diff --git a/cf-check/dump.c b/cf-check/dump.c index 611ef22626..bd3ffa2056 100644 --- a/cf-check/dump.c +++ b/cf-check/dump.c @@ -269,7 +269,7 @@ static void print_struct_or_string( } else if (StringEndsWith(file, "cf_observations.lmdb")) { - if (StringSafeEqual(key.mv_data, "DATABASE_AGE")) + if (StringEqual(key.mv_data, "DATABASE_AGE")) { assert(sizeof(double) == value.mv_size); const double *const age = value.mv_data; @@ -290,13 +290,13 @@ static void print_struct_or_string( } else if (StringEndsWith(file, "nova_agent_execution.lmdb")) { - if (StringSafeEqual(key.mv_data, "delta_gavr")) + if (StringEqual(key.mv_data, "delta_gavr")) { assert(sizeof(double) == value.mv_size); const double *const average = value.mv_data; printf("%f", *average); } - else if (StringSafeEqual(key.mv_data, "last_exec")) + else if (StringEqual(key.mv_data, "last_exec")) { assert(sizeof(time_t) == value.mv_size); const time_t *const last_exec = value.mv_data; diff --git a/cf-check/validate.c b/cf-check/validate.c index 9168ed3c6a..9ecd8f13ec 100644 --- a/cf-check/validate.c +++ b/cf-check/validate.c @@ -482,7 +482,7 @@ static void ValidateStateLastseen(ValidatorState *state) ValidationError( state, "Missing address entry for '%s'", address); } - else if (!StringSafeEqual(hostkey, lookup)) + else if (!StringEqual(hostkey, lookup)) { ValidationError( state, @@ -513,7 +513,7 @@ static void ValidateStateLastseen(ValidatorState *state) ValidationError( state, "Missing hostkey entry for '%s'", hostkey); } - else if (!StringSafeEqual(address, lookup)) + else if (!StringEqual(address, lookup)) { ValidationError( state, diff --git a/cf-execd/cf-execd.c b/cf-execd/cf-execd.c index 0d58208896..9b5253a319 100644 --- a/cf-execd/cf-execd.c +++ b/cf-execd/cf-execd.c @@ -325,17 +325,17 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) case 0: { const char *const option_name = OPTIONS[longopt_idx].name; - if (StringSafeEqual(option_name, "skip-db-check")) + if (StringEqual(option_name, "skip-db-check")) { if (optarg == NULL) { PERFORM_DB_CHECK = false; // Skip (no arg), check = false } - else if (StringSafeEqual_IgnoreCase(optarg, "yes")) + else if (StringEqual_IgnoreCase(optarg, "yes")) { PERFORM_DB_CHECK = false; // Skip = yes, check = false } - else if (StringSafeEqual_IgnoreCase(optarg, "no")) + else if (StringEqual_IgnoreCase(optarg, "no")) { PERFORM_DB_CHECK = true; // Skip = no, check = true } diff --git a/cf-secret/cf-secret.c b/cf-secret/cf-secret.c index 4eacb34703..72e159a5cd 100644 --- a/cf-secret/cf-secret.c +++ b/cf-secret/cf-secret.c @@ -172,7 +172,7 @@ static char *GetHostRSAKey(const char *host, HostRSAKeyType type) inet_ntop(res->ai_family, GetIPAddress((struct sockaddr *) res->ai_addr), ipaddress, sizeof(ipaddress)); - if (StringStartsWith(ipaddress, "127.") || StringSafeEqual(ipaddress, "::1")) + if (StringStartsWith(ipaddress, "127.") || StringEqual(ipaddress, "::1")) { Log(LOG_LEVEL_VERBOSE, "Using localhost%s key", key_ext); found = true; @@ -253,7 +253,7 @@ static FILE *OpenInputOutput(const char *path, const char *mode) { assert(path != NULL); assert(mode != NULL); - if (StringSafeEqual(path, "-")) + if (StringEqual(path, "-")) { if (*mode == 'r') { @@ -448,9 +448,9 @@ static bool RSAEncrypt(Seq *rsa_keys, const char *input_path, const char *output static inline bool CheckHeader(const char *key, const char *value) { - if (StringSafeEqual(key, "Version")) + if (StringEqual(key, "Version")) { - if (!StringSafeEqual(value, "1.0")) + if (!StringEqual(value, "1.0")) { Log(LOG_LEVEL_ERR, "Unsupported file format version: '%s'", value); return false; @@ -460,7 +460,7 @@ static inline bool CheckHeader(const char *key, const char *value) return true; } } - else if (StringSafeEqual(key, "Encrypted-for")) + else if (StringEqual(key, "Encrypted-for")) { /* TODO: do some verification that 'value' is valid hash digest? */ return true; @@ -565,14 +565,14 @@ static bool ParseHeaders(FILE *input_file, RSA *privkey, size_t *enc_key_pos, si return false; } - if (StringSafeEqual(key, "Version")) + if (StringEqual(key, "Version")) { version_specified = true; } - else if (StringSafeEqual(key, "Encrypted-for")) + else if (StringEqual(key, "Encrypted-for")) { Log(LOG_LEVEL_DEBUG, "Encrypted for '%s'", value); - if (StringSafeEqual(value, key_digest)) + if (StringEqual(value, key_digest)) { found_matching_digest = true; *enc_key_pos = n_enc_for_headers; @@ -828,17 +828,17 @@ int main(int argc, char *argv[]) bool print_headers = false; size_t offset = 0; - if (StringSafeEqual(argv[1], "encrypt")) + if (StringEqual(argv[1], "encrypt")) { encrypt = true; offset++; } - else if (StringSafeEqual(argv[1], "decrypt")) + else if (StringEqual(argv[1], "decrypt")) { offset++; decrypt = true; } - else if (StringSafeEqual(argv[1], "print-headers")) + else if (StringEqual(argv[1], "print-headers")) { print_headers = true; offset++; diff --git a/cf-testd/cf-testd.c b/cf-testd/cf-testd.c index 279a3c61ab..94564aa259 100644 --- a/cf-testd/cf-testd.c +++ b/cf-testd/cf-testd.c @@ -732,12 +732,12 @@ static char *IncrementIPaddress(const char *ip_str) int step = 1; char *last_dot = strrchr(ip_str, '.'); assert(last_dot != NULL); /* the doc comment says there must be dots! */ - if (StringSafeEqual(last_dot + 1, "255")) + if (StringEqual(last_dot + 1, "255")) { /* avoid the network address (ending with 0) */ step = 2; } - else if (StringSafeEqual(last_dot + 1, "254")) + else if (StringEqual(last_dot + 1, "254")) { /* avoid the broadcast address and the network address */ step = 3; diff --git a/libcfnet/conn_cache.c b/libcfnet/conn_cache.c index e5e1031119..a8be613483 100644 --- a/libcfnet/conn_cache.c +++ b/libcfnet/conn_cache.c @@ -32,7 +32,7 @@ #include /* ThreadLock */ #include /* Hostname2IPString */ #include /* CF_ASSERT */ -#include /* StringSafeEqual */ +#include /* StringEqual */ /** @@ -95,8 +95,8 @@ static bool ConnCacheEntryMatchesConnection(ConnCache_entry *entry, ConnectionFlags flags) { return ConnectionFlagsEqual(&flags, &entry->conn->flags) && - StringSafeEqual(port, entry->conn->this_port) && - StringSafeEqual(server, entry->conn->this_server); + StringEqual(port, entry->conn->this_port) && + StringEqual(server, entry->conn->this_server); } AgentConnection *ConnCache_FindIdleMarkBusy(const char *server, diff --git a/libcfnet/protocol.c b/libcfnet/protocol.c index 81e03fe270..898ac166a9 100644 --- a/libcfnet/protocol.c +++ b/libcfnet/protocol.c @@ -75,7 +75,7 @@ Seq *ProtocolOpenDir(AgentConnection *conn, const char *path) */ for (int i = 0; i < len && buf[i] != '\0'; i += strlen(buf + i) + 1) { - if (StringSafeEqualN(buf + i, CFD_TERMINATOR, + if (StringEqualN(buf + i, CFD_TERMINATOR, sizeof(CFD_TERMINATOR) - 1)) { more = 0; @@ -159,7 +159,7 @@ bool ProtocolGet(AgentConnection *conn, const char *remote_path, break; } - if (StringSafeEqualN(buf, cfchangedstr, sizeof(cfchangedstr) - 1)) + if (StringEqualN(buf, cfchangedstr, sizeof(cfchangedstr) - 1)) { Log(LOG_LEVEL_ERR, "Remote file %s:%s changed during file transfer", diff --git a/libcfnet/protocol_version.c b/libcfnet/protocol_version.c index 3e21907520..59a3cf4f37 100644 --- a/libcfnet/protocol_version.c +++ b/libcfnet/protocol_version.c @@ -17,19 +17,19 @@ ProtocolVersion ParseProtocolVersionNetwork(const char *const s) ProtocolVersion ParseProtocolVersionPolicy(const char *const s) { - if ((s == NULL) || StringSafeEqual(s, "0") || StringSafeEqual(s, "undefined")) + if ((s == NULL) || StringEqual(s, "0") || StringEqual(s, "undefined")) { return CF_PROTOCOL_UNDEFINED; } - if (StringSafeEqual(s, "1") || StringSafeEqual(s, "classic")) + if (StringEqual(s, "1") || StringEqual(s, "classic")) { return CF_PROTOCOL_CLASSIC; } - else if (StringSafeEqual(s, "2") || StringSafeEqual(s, "tls")) + else if (StringEqual(s, "2") || StringEqual(s, "tls")) { return CF_PROTOCOL_TLS; } - else if (StringSafeEqual(s, "latest")) + else if (StringEqual(s, "latest")) { return CF_PROTOCOL_LATEST; } diff --git a/libcfnet/tls_generic.c b/libcfnet/tls_generic.c index ae682fc20c..1a2459f3cb 100644 --- a/libcfnet/tls_generic.c +++ b/libcfnet/tls_generic.c @@ -901,7 +901,7 @@ void TLSSetDefaultOptions(SSL_CTX *ssl_ctx, const char *min_version) bool found = false; for (enum tls_version v = TLS_1_0; !found && v <= TLS_LAST; v++) { - if (StringSafeEqual(min_version, tls_version_strings[v])) + if (StringEqual(min_version, tls_version_strings[v])) { found = true; if (v < TLS_LOWEST_REQUIRED) diff --git a/libenv/sysinfo.c b/libenv/sysinfo.c index aa1bd79b60..cae60e41c7 100644 --- a/libenv/sysinfo.c +++ b/libenv/sysinfo.c @@ -1139,7 +1139,7 @@ static void OSReleaseParse(EvalContext *ctx, const char *file_path) CanonifyNameInPlace(os_release_id); const char *alias = NULL; - if (StringSafeEqual(os_release_id, "rhel")) + if (StringEqual(os_release_id, "rhel")) { alias = "redhat"; } diff --git a/libntech b/libntech index 0943bb572d..080b24b945 160000 --- a/libntech +++ b/libntech @@ -1 +1 @@ -Subproject commit 0943bb572d636b0fefb3378ae6f6692f990350dc +Subproject commit 080b24b945fb8052df12fa9396193e5557f5aadd diff --git a/libpromises/class.c b/libpromises/class.c index b24e1d33c9..6a969727c4 100644 --- a/libpromises/class.c +++ b/libpromises/class.c @@ -48,7 +48,7 @@ TYPED_MAP_DECLARE(Class, char *, Class *) TYPED_MAP_DEFINE(Class, char *, Class *, StringHash_untyped, - StringSafeEqual_untyped, + StringEqual_untyped, free, ClassDestroy_untyped) diff --git a/libpromises/dbm_api.c b/libpromises/dbm_api.c index a42a8ce8d0..c4deac8290 100644 --- a/libpromises/dbm_api.c +++ b/libpromises/dbm_api.c @@ -203,7 +203,7 @@ static bool IsSubHandle(DBHandle *handle, dbid id, const char *name) { char *sub_path = DBIdToSubPath(id, name); - bool result = StringSafeEqual(handle->filename, sub_path); + bool result = StringEqual(handle->filename, sub_path); free(sub_path); return result; } @@ -452,7 +452,7 @@ DBHandle *GetDBHandleFromFilename(const char *db_file_name) ThreadLock(&db_handles_lock); for(dbid id=0; id < dbid_max; id++) { - if (StringSafeEqual(db_handles[id].filename, db_file_name)) + if (StringEqual(db_handles[id].filename, db_file_name)) { ThreadUnlock(&db_handles_lock); return &(db_handles[id]); diff --git a/libpromises/eval_context.c b/libpromises/eval_context.c index 221ce5d93a..261da8ffee 100644 --- a/libpromises/eval_context.c +++ b/libpromises/eval_context.c @@ -101,7 +101,7 @@ TYPED_MAP_DECLARE(RemoteVarPromises, char *, Seq *) TYPED_MAP_DEFINE(RemoteVarPromises, char *, Seq *, StringHash_untyped, - StringSafeEqual_untyped, + StringEqual_untyped, free, SeqDestroy_untyped) @@ -613,7 +613,7 @@ static bool EvalWithTokenFromList(const char *expr, StringSet *token_set) bool EvalProcessResult(const char *process_result, StringSet *proc_attr) { assert(process_result != NULL); - if (StringSafeEqual(process_result, "")) + if (StringEqual(process_result, "")) { /* nothing to evaluate */ return false; @@ -1466,7 +1466,7 @@ void EvalContextStackPushPromiseFrame(EvalContext *ctx, const Promise *owner) for (size_t i = 0; i < SeqLength(owner->conlist); i++) { Constraint *cp = SeqAt(owner->conlist, i); - if (StringSafeEqual(cp->lval, "with")) + if (StringEqual(cp->lval, "with")) { Rval final = EvaluateFinalRval(ctx, PromiseGetPolicy(owner), NULL, "this", cp->rval, false, owner); @@ -2418,7 +2418,7 @@ const Bundle *EvalContextResolveBundleExpression(const EvalContext *ctx, const P const Bundle *curr_bp = SeqAt(policy->bundles, i); if ((strcmp(curr_bp->type, callee_type) != 0) || (strcmp(curr_bp->name, ref.name) != 0) || - !StringSafeEqual(curr_bp->ns, ref.ns)) + !StringEqual(curr_bp->ns, ref.ns)) { continue; } @@ -2439,7 +2439,7 @@ const Body *EvalContextFindFirstMatchingBody(const Policy *policy, const char *t const Body *curr_bp = SeqAt(policy->bodies, i); if ((strcmp(curr_bp->type, type) == 0) && (strcmp(curr_bp->name, name) == 0) && - StringSafeEqual(curr_bp->ns, namespace)) + StringEqual(curr_bp->ns, namespace)) { return curr_bp; } diff --git a/libpromises/evalfunction.c b/libpromises/evalfunction.c index dd4a043979..b05d34253c 100644 --- a/libpromises/evalfunction.c +++ b/libpromises/evalfunction.c @@ -1259,15 +1259,15 @@ static FnCallResult FnCallClassesMatching(EvalContext *ctx, ARG_UNUSED const Pol bool check_only = false; unsigned count = 0; - if (StringSafeEqual(fp->name, "classesmatching")) + if (StringEqual(fp->name, "classesmatching")) { // Expected / default case } - else if (StringSafeEqual(fp->name, "classmatch")) + else if (StringEqual(fp->name, "classmatch")) { check_only = true; } - else if (StringSafeEqual(fp->name, "countclassesmatching")) + else if (StringEqual(fp->name, "countclassesmatching")) { count_only = true; } @@ -4022,7 +4022,7 @@ static FnCallResult FnCallFileStat(ARG_UNUSED EvalContext *ctx, ARG_UNUSED const if (lstat(path, &statbuf) == -1) { - if (StringSafeEqual(fp->name, "filesize")) + if (StringEqual(fp->name, "filesize")) { return FnFailure(); } @@ -5148,7 +5148,7 @@ static FnCallResult FnCallFormat(EvalContext *ctx, ARG_UNUSED const Policy *poli if (SeqLength(s) >= 2) { const char *format_piece = BufferData(SeqAt(s, 1)); - bool percent = StringSafeEqualN(format_piece, "%%", 2); + bool percent = StringEqualN(format_piece, "%%", 2); char *data = NULL; if (percent) @@ -6665,7 +6665,7 @@ static FnCallResult FnCallReadData(ARG_UNUSED EvalContext *ctx, const char *input_path = RlistScalarValue(args); const char *const mode_string = RlistScalarValue(args->next); DataFileType requested_mode = DATAFILETYPE_UNKNOWN; - if (StringSafeEqual("auto", mode_string)) + if (StringEqual("auto", mode_string)) { requested_mode = GetDataFileTypeFromSuffix(input_path); Log(LOG_LEVEL_VERBOSE, diff --git a/libpromises/generic_agent.c b/libpromises/generic_agent.c index 8441bfb62c..5cdc34872c 100644 --- a/libpromises/generic_agent.c +++ b/libpromises/generic_agent.c @@ -251,10 +251,10 @@ static bool LoadAugmentsData(EvalContext *ctx, const char *filename, const JsonE const char *key; while ((key = JsonIteratorNextKey(&iter))) { - if (!(StringSafeEqual(key, "vars") || - StringSafeEqual(key, "classes") || - StringSafeEqual(key, "inputs") || - StringSafeEqual(key, "augments"))) + if (!(StringEqual(key, "vars") || + StringEqual(key, "classes") || + StringEqual(key, "inputs") || + StringEqual(key, "augments"))) { Log(LOG_LEVEL_VERBOSE, "Unknown augments key '%s' in file '%s', skipping it", key, filename); diff --git a/libpromises/iteration.c b/libpromises/iteration.c index a9ef013947..7e508d138c 100644 --- a/libpromises/iteration.c +++ b/libpromises/iteration.c @@ -1054,7 +1054,7 @@ bool PromiseIteratorNext(PromiseIterator *iterctx, EvalContext *evalctx) for (size_t i = 0; i < SeqLength(iterctx->pp->conlist); i++) { Constraint *cp = SeqAt(iterctx->pp->conlist, i); - if (StringSafeEqual(cp->lval, "with")) + if (StringEqual(cp->lval, "with")) { Rval final = EvaluateFinalRval(evalctx, PromiseGetPolicy(iterctx->pp), NULL, "this", cp->rval, false, iterctx->pp); diff --git a/libpromises/loading.c b/libpromises/loading.c index fe200b2bdf..72b335a861 100644 --- a/libpromises/loading.c +++ b/libpromises/loading.c @@ -281,10 +281,10 @@ static void RenameMainBundle(EvalContext *ctx, Policy *policy) for (int i = 0; i < length; ++i) { Bundle *const bundle = SeqAt(bundles, i); - if (StringSafeEqual(bundle->name, "__main__")) + if (StringEqual(bundle->name, "__main__")) { char *abspath = GetRealPath(bundle->source_path); - if (StringSafeEqual(abspath, entry_point)) + if (StringEqual(abspath, entry_point)) { Log(LOG_LEVEL_VERBOSE, "Redefining __main__ bundle from file %s to be main", diff --git a/libpromises/mod_methods.c b/libpromises/mod_methods.c index e67dd75b6b..554db19fe0 100644 --- a/libpromises/mod_methods.c +++ b/libpromises/mod_methods.c @@ -51,7 +51,7 @@ static bool MethodsParseTreeCheck(const Promise *pp, Seq *errors) const Constraint *cp = SeqAt(pp->conlist, i); // ensure: if call and callee are resolved, then they have matching arity - if (StringSafeEqual(cp->lval, "usebundle")) + if (StringEqual(cp->lval, "usebundle")) { if (cp->rval.type == RVAL_TYPE_FNCALL) { diff --git a/libpromises/policy.c b/libpromises/policy.c index 9182a5cf36..885e8c89b3 100644 --- a/libpromises/policy.c +++ b/libpromises/policy.c @@ -991,7 +991,7 @@ bool PolicyCheckDuplicateHandles(const Policy *policy, Seq *errors) { bool success = true; - Map *recorded = MapNew(StringHash_untyped, StringSafeEqual_untyped, NULL, NULL); + Map *recorded = MapNew(StringHash_untyped, StringEqual_untyped, NULL, NULL); for (size_t bpi = 0; bpi < SeqLength(policy->bundles); bpi++) { diff --git a/libpromises/rlist.c b/libpromises/rlist.c index d392e36d5b..66891cc054 100644 --- a/libpromises/rlist.c +++ b/libpromises/rlist.c @@ -300,7 +300,7 @@ bool RlistContainsString(const Rlist *list, const char *string) for (const Rlist *rp = list; rp != NULL; rp = rp->next) { if (rp->val.type == RVAL_TYPE_SCALAR && - StringSafeEqual(RlistScalarValue(rp), string)) + StringEqual(RlistScalarValue(rp), string)) { return true; } diff --git a/libpromises/verify_vars.c b/libpromises/verify_vars.c index 8812b091ec..d7f5112806 100644 --- a/libpromises/verify_vars.c +++ b/libpromises/verify_vars.c @@ -82,7 +82,7 @@ static bool IsLegalVariableName(EvalContext *ctx, const Promise *pp) char *prefix = xstrndup(var_name, dot - var_name); const Bundle *cur_bundle = PromiseGetBundle(pp); - if (!StringSafeEqual(prefix, cur_bundle->name)) + if (!StringEqual(prefix, cur_bundle->name)) { Log(LOG_LEVEL_VERBOSE, "Variable '%s' may be attempted to be injected into a remote bundle", diff --git a/tests/unit/logging_test.c b/tests/unit/logging_test.c index e625e2b8c9..cc342de44c 100644 --- a/tests/unit/logging_test.c +++ b/tests/unit/logging_test.c @@ -67,7 +67,7 @@ static void test_set_host(void) #define check_level(str, lvl) \ {\ assert_int_equal(LogLevelFromString(str), lvl);\ - assert_true(StringSafeEqual_IgnoreCase(str, LogLevelToString(lvl)));\ + assert_true(StringEqual_IgnoreCase(str, LogLevelToString(lvl)));\ } static void test_log_level(void) From 21255a20aef8270daa96e4204cd370233a968394 Mon Sep 17 00:00:00 2001 From: Lluis Campos Date: Fri, 5 Jun 2020 15:30:55 +0200 Subject: [PATCH 117/333] libntech: Bumped to latest; rename TO_STRING macro Latest libntech renamed TOSTRING to TO_STRING. Fix here accordingly and clean the wrongly used "- 1" from the log expression. Changelog: None Ticket: None Signed-off-by: Lluis Campos (cherry picked from commit ebfddd77cfdf5ce6a3a2290ee4a5f4c0dc5de506) --- cf-monitord/mon_io_linux.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cf-monitord/mon_io_linux.c b/cf-monitord/mon_io_linux.c index 423df446b8..92e8ff75c6 100644 --- a/cf-monitord/mon_io_linux.c +++ b/cf-monitord/mon_io_linux.c @@ -182,7 +182,7 @@ static void MonIoDiskstatsGatherData(double *cf_this) if (!strchr(buf, '\n')) { Log(LOG_LEVEL_ERR, - "/proc/diskstats format error: read overlong string (> " TOSTRING(CF_BUFSIZE - 1) " bytes)"); + "/proc/diskstats format error: read overlong string (> " TO_STRING(CF_BUFSIZE) " bytes)"); goto err; } From 3ca41e5da32e227551fab79a411a622975a0116f Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Fri, 19 Jun 2020 13:53:08 +0200 Subject: [PATCH 118/333] Fixed locking of promises using log_repaired / log_string with timestamps log_string in action bodies is no longer used in promise hashing. Promise hashes are used to track a promise between agent runs, so anything which changes every run (like time) should be excluded. Log messages very commonly have timestamps, and our default from standard library does. Changelog: Title Ticket: CFE-3376 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit a18ad4e92b50bc3d119f4cee3523dbb5f3e41694) --- libpromises/locks.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libpromises/locks.c b/libpromises/locks.c index c6f71426b8..01407eda22 100644 --- a/libpromises/locks.c +++ b/libpromises/locks.c @@ -648,7 +648,7 @@ void PromiseRuntimeHash(const Promise *pp, const char *salt, Rlist *rp; FnCall *fp; - char *noRvalHash[] = { "mtime", "atime", "ctime", "stime_range", "ttime_range", NULL }; + char *noRvalHash[] = { "mtime", "atime", "ctime", "stime_range", "ttime_range", "log_string", NULL }; int doHash; md = HashDigestFromId(type); From 15f276c27a9af5b82826c0ea54b1c9d3f1e79d41 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 15 Jul 2020 10:17:01 +0200 Subject: [PATCH 119/333] Wait for background processes before generating reports Reports generation only happens if there is exactly one 'cf-agent' process running (to avoid corruptions in the reporting data). If some promise in the policy uses an action policy with 'background => true', there might still be a forked child process evaluating the promise when the main process reaches the point when it wants to generate reports. In such cases, it is important that the main process waits for the forked child processes. Ticket: ENT-6042 Changelog: Promises with 'action => bg()' no longer break reporting data (cherry picked from commit 3d9abd5cb4a21a920d5c6c9c6e25946533e132e9) --- cf-agent/cf-agent.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/cf-agent/cf-agent.c b/cf-agent/cf-agent.c index 4ccb0d0297..7f7345f5d8 100644 --- a/cf-agent/cf-agent.c +++ b/cf-agent/cf-agent.c @@ -159,6 +159,7 @@ static bool HasAvahiSupport(void); static int AutomaticBootstrap(GenericAgentConfig *config); static void BannerStatus(PromiseResult status, char *type, char *name); static PromiseResult DefaultVarPromise(EvalContext *ctx, const Promise *pp); +static void WaitForBackgroundProcesses(); /*******************************************************************/ /* Command line options */ @@ -295,6 +296,11 @@ int main(int argc, char *argv[]) /* Update packages cache. */ UpdatePackagesCache(ctx, false); + /* Wait for background processes before generating reports because + * GenerateReports() does nothing if it detects multiple cf-agent processes + * running. */ + WaitForBackgroundProcesses(); + GenerateReports(config, ctx); PurgeLocks(); @@ -2127,3 +2133,25 @@ static int AutomaticBootstrap(ARG_UNUSED GenericAgentConfig *config) } #endif // Avahi + +static void WaitForBackgroundProcesses() +{ +#ifdef __MINGW32__ + /* no fork() on Windows */ + return; +#else + Log(LOG_LEVEL_VERBOSE, "Waiting for background processes"); + bool have_children = true; + while (have_children) + { + pid_t child = wait(NULL); + if (child >= 0) + { + Log(LOG_LEVEL_VERBOSE, "Background process %ju terminated", (uintmax_t) child); + } + have_children = !((child == -1) && (errno == ECHILD)); + } + Log(LOG_LEVEL_VERBOSE, "No more background processes to wait for"); + return; +#endif /* __MINGW32__ */ +} From 004a3fd500de6ea5bb9c6d695ed705ec178896a5 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 22 Jul 2020 12:38:10 +0200 Subject: [PATCH 120/333] Backgrounded commands are now correctly executed in the child process Changelog: Title Ticket: CFE-3379 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 66e0a82d962735024412eea58edbad89fe0f253a) --- cf-agent/verify_exec.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/cf-agent/verify_exec.c b/cf-agent/verify_exec.c index 9a8eaa6709..66a70e4ff2 100644 --- a/cf-agent/verify_exec.c +++ b/cf-agent/verify_exec.c @@ -212,7 +212,8 @@ static ActionResult RepairExec(EvalContext *ctx, const Attributes *a, char eventname[CF_BUFSIZE]; char cmdline[CF_BUFSIZE]; char comm[20]; - int outsourced, count = 0; + bool do_work_here; + int count = 0; #if !defined(__MINGW32__) mode_t maskval = 0; #endif @@ -294,18 +295,18 @@ static ActionResult RepairExec(EvalContext *ctx, const Attributes *a, if (a->transaction.background) { #ifdef __MINGW32__ - outsourced = true; + do_work_here = true; #else Log(LOG_LEVEL_VERBOSE, "Backgrounding job '%s'", cmdline); - outsourced = fork(); + do_work_here = (fork() == 0); // true for child, false for parent #endif } else { - outsourced = false; + do_work_here = false; } - if (outsourced || (!a->transaction.background)) // work done here: either by child or non-background parent + if (do_work_here || (!a->transaction.background)) // work done here: either by child or non-background parent { if (a->contain.timeout != CF_NOINT) { @@ -421,7 +422,7 @@ static ActionResult RepairExec(EvalContext *ctx, const Attributes *a, free(line); #ifdef __MINGW32__ - if (outsourced) // only get return value if we waited for command execution + if (do_work_here) // only get return value if we waited for command execution { cf_pclose_nowait(pfp); } @@ -467,7 +468,7 @@ static ActionResult RepairExec(EvalContext *ctx, const Attributes *a, snprintf(eventname, CF_BUFSIZE - 1, "Exec(%s)", cmdline); #ifndef __MINGW32__ - if ((a->transaction.background) && outsourced) + if ((a->transaction.background) && do_work_here) { Log(LOG_LEVEL_VERBOSE, "Backgrounded command '%s' is done - exiting", cmdline); From 329d1e0f30f393b577bb4481584fa9a00d5e8962 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 22 Jul 2020 13:32:43 +0200 Subject: [PATCH 121/333] Backgrounded child processes no longer run cleanup exit handlers I haven't seen this cause issues, but it's more correct. Changelog: None Ticket: CFE-3379 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 7fedde00fd5ea94a02eb75a829baec1a6c6783fd) --- cf-agent/verify_exec.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/cf-agent/verify_exec.c b/cf-agent/verify_exec.c index 66a70e4ff2..f345e3a5bb 100644 --- a/cf-agent/verify_exec.c +++ b/cf-agent/verify_exec.c @@ -468,13 +468,12 @@ static ActionResult RepairExec(EvalContext *ctx, const Attributes *a, snprintf(eventname, CF_BUFSIZE - 1, "Exec(%s)", cmdline); #ifndef __MINGW32__ - if ((a->transaction.background) && do_work_here) + if ((a->transaction.background) && do_work_here) // Child process { Log(LOG_LEVEL_VERBOSE, "Backgrounded command '%s' is done - exiting", cmdline); - /* exit() OK since this is a forked process and no functions are - registered for cleanup */ - exit(EXIT_SUCCESS); + // _exit() since this is the child and we don't want to run cleanup + _exit(EXIT_SUCCESS); } #endif /* !__MINGW32__ */ From 30af04e209eac397eafd0fda5888ed55649da143 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 22 Jul 2020 14:36:54 +0200 Subject: [PATCH 122/333] Cleaned up use of do_work_here variable Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit ea1629658cc7c104c2943f609ea61c6bf00e8319) --- cf-agent/verify_exec.c | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/cf-agent/verify_exec.c b/cf-agent/verify_exec.c index f345e3a5bb..76f18b99d7 100644 --- a/cf-agent/verify_exec.c +++ b/cf-agent/verify_exec.c @@ -212,7 +212,6 @@ static ActionResult RepairExec(EvalContext *ctx, const Attributes *a, char eventname[CF_BUFSIZE]; char cmdline[CF_BUFSIZE]; char comm[20]; - bool do_work_here; int count = 0; #if !defined(__MINGW32__) mode_t maskval = 0; @@ -292,21 +291,17 @@ static ActionResult RepairExec(EvalContext *ctx, const Attributes *a, CommandPrefix(cmdline, comm); + bool do_work_here = true; + if (a->transaction.background) { -#ifdef __MINGW32__ - do_work_here = true; -#else Log(LOG_LEVEL_VERBOSE, "Backgrounding job '%s'", cmdline); +#ifndef __MINGW32__ do_work_here = (fork() == 0); // true for child, false for parent #endif } - else - { - do_work_here = false; - } - if (do_work_here || (!a->transaction.background)) // work done here: either by child or non-background parent + if (do_work_here) // work done here: either by child or non-background parent { if (a->contain.timeout != CF_NOINT) { @@ -422,7 +417,7 @@ static ActionResult RepairExec(EvalContext *ctx, const Attributes *a, free(line); #ifdef __MINGW32__ - if (do_work_here) // only get return value if we waited for command execution + if (a->transaction.background) // only get return value if we waited for command execution { cf_pclose_nowait(pfp); } From dfd9a05c3ff8a1761a016065611bc0c305fdbbe0 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Fri, 24 Jul 2020 13:43:58 +0200 Subject: [PATCH 123/333] Stopped printing background log message on windows Just a small mistake from previous PR. Changelog: None Ticket: CFE-3379 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit c7ee3e9ddc5a14171be3ba73027296587239bd3f) --- cf-agent/verify_exec.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cf-agent/verify_exec.c b/cf-agent/verify_exec.c index 76f18b99d7..326e0d907c 100644 --- a/cf-agent/verify_exec.c +++ b/cf-agent/verify_exec.c @@ -293,13 +293,13 @@ static ActionResult RepairExec(EvalContext *ctx, const Attributes *a, bool do_work_here = true; +#ifndef __MINGW32__ if (a->transaction.background) { Log(LOG_LEVEL_VERBOSE, "Backgrounding job '%s'", cmdline); -#ifndef __MINGW32__ do_work_here = (fork() == 0); // true for child, false for parent -#endif } +#endif if (do_work_here) // work done here: either by child or non-background parent { From f0a52aa9b635cb2e55beaef86a6c650ecc9889f1 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Wed, 29 Jul 2020 09:45:16 -0500 Subject: [PATCH 124/333] Added example for storejson() Ticket: CFE-3388 Changelog: None (cherry picked from commit 27b8a80f204f1d58ff9a19a10ea0a2f383542cdc) --- examples/storejson.cf | 59 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 examples/storejson.cf diff --git a/examples/storejson.cf b/examples/storejson.cf new file mode 100644 index 0000000000..851f47d6d0 --- /dev/null +++ b/examples/storejson.cf @@ -0,0 +1,59 @@ +#+begin_src cfengine3 +bundle common globals +{ + vars: + "example_data" data => '{ "msg": "Hello from $(this.bundle)" }'; +} +bundle agent example_storejson +# @brief Example showing storejson +{ + vars: + "example_data" data => '{ "msg": "Hello from $(this.bundle)" }'; + + # Using storejson with data from remote bundle + + # "json_string_zero" -> { "CFEngine 3.16.0"} + # string => storejson( globals.example_data ) + # comment => "Unquoted with . (dot) present will cause the parser to error"; + + "json_string_one" string => storejson( @(globals.example_data) ); + "json_string_two" string => storejson( "globals.example_data" ); + + # Using storejson with data from this bundle + "json_string_three" string => storejson( @(example_storejson.example_data) ); + "json_string_four" string => storejson( "example_storejson.example_data"); + "json_string_five" string => storejson( example_data ); + "json_string_six" string => storejson( "$(this.bundle).example_data"); + "json_string_seven" string => storejson( @(example_data) ); + + reports: + "json_string_one and json_string_two are identical:$(const.n)$(json_string_one)" + if => strcmp( $(json_string_one), $(json_string_two) ); + + "json_string_{one,two,three,four,five,six,seven} are identical:$(const.n)$(json_string_three)" + if => and( + strcmp( $(json_string_three), $(json_string_four) ), + strcmp( $(json_string_four), $(json_string_five) ), + strcmp( $(json_string_five), $(json_string_six) ), + strcmp( $(json_string_six), $(json_string_seven) ) + ); +} + +bundle agent __main__ +{ + methods: "example_storejson"; +} +############################################################################### +#+end_src +#+begin_src example_output +#@ ``` +#@ R: json_string_one and json_string_two are identical: +#@ { +#@ "msg": "Hello from globals" +#@ } +#@ R: json_string_{one,two,three,four,five,six,seven} are identical: +#@ { +#@ "msg": "Hello from example_storejson" +#@ } +#@ ``` +#+end_src From f632e93c4e97a37cb51c7bded377920f5ac87272 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Mon, 3 Aug 2020 11:28:39 +0200 Subject: [PATCH 125/333] Rename 'no_files' to 'n_files' in FileChangesSetDirectoryList() So that it doesn't look like a boolean value "no files" and more like "number of files" instead. Ticket: CFE-3382 Changelog: None (cherry picked from commit 774ed0ad5669629049873859a2e88288107735d1) --- cf-agent/files_changes.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/cf-agent/files_changes.c b/cf-agent/files_changes.c index 944f196b11..a780ddd291 100644 --- a/cf-agent/files_changes.c +++ b/cf-agent/files_changes.c @@ -410,25 +410,25 @@ bool FileChangesGetDirectoryList(const char *path, Seq *files) static bool FileChangesSetDirectoryList(CF_DB *db, const char *path, const Seq *files) { int size = 0; - int no_files = SeqLength(files); + int n_files = SeqLength(files); char key[strlen(path) + 3]; xsnprintf(key, sizeof(key), "D_%s", path); - if (no_files == 0) + if (n_files == 0) { DeleteDB(db, key); return true; } - for (int c = 0; c < no_files; c++) + for (int c = 0; c < n_files; c++) { size += strlen(SeqAt(files, c)) + 1; } char raw_entries[size]; char *pos = raw_entries; - for (int c = 0; c < no_files; c++) + for (int c = 0; c < n_files; c++) { strcpy(pos, SeqAt(files, c)); pos += strlen(pos) + 1; From 6d7e363836e7da44abb767f8d23d9f2fdd9ca0f6 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Mon, 3 Aug 2020 12:00:43 +0200 Subject: [PATCH 126/333] Only write directory listing if it changes No need to write to the DB if there is no change and no need to spam the output. Ticket: CFE-3382 Changelog: Directory listings in files changes monitoring are now only updated when there is a change (cherry picked from commit 0c141029c93867d933c6f41243016d7af93f5404) Conflicts: cf-agent/files_changes.c --- cf-agent/files_changes.c | 49 ++++++++++++++++++++++++++++------------ 1 file changed, 34 insertions(+), 15 deletions(-) diff --git a/cf-agent/files_changes.c b/cf-agent/files_changes.c index a780ddd291..4d001b9615 100644 --- a/cf-agent/files_changes.c +++ b/cf-agent/files_changes.c @@ -61,7 +61,7 @@ typedef struct } ChecksumValue; static bool GetDirectoryListFromDatabase(CF_DB *db, const char * path, Seq *files); -static bool FileChangesSetDirectoryList(CF_DB *db, const char *path, const Seq *files); +static bool FileChangesSetDirectoryList(CF_DB *db, const char *path, const Seq *files, bool *change); /* * Key format: @@ -198,7 +198,8 @@ static void AddMigratedFileToDirectoryList(CF_DB *changes_db, const char *file, { SeqAppend(files, xstrdup(basefile)); SeqSort(files, (SeqItemComparator)strcmp, NULL); - if (!FileChangesSetDirectoryList(changes_db, dir, files)) + bool changes; + if (!FileChangesSetDirectoryList(changes_db, dir, files, &changes)) { Log(LOG_LEVEL_ERR, "%s: Not able to update directory index", common_msg); } @@ -407,8 +408,10 @@ bool FileChangesGetDirectoryList(const char *path, Seq *files) return result; } -static bool FileChangesSetDirectoryList(CF_DB *db, const char *path, const Seq *files) +static bool FileChangesSetDirectoryList(CF_DB *db, const char *path, const Seq *files, bool *change) { + assert(change != NULL); + int size = 0; int n_files = SeqLength(files); @@ -417,7 +420,7 @@ static bool FileChangesSetDirectoryList(CF_DB *db, const char *path, const Seq * if (n_files == 0) { - DeleteDB(db, key); + *change = DeleteDB(db, key); return true; } @@ -434,12 +437,25 @@ static bool FileChangesSetDirectoryList(CF_DB *db, const char *path, const Seq * pos += strlen(pos) + 1; } + if (HasKeyDB(db, key, sizeof(key))) + { + char old_entries[MAX(size, 2 * CF_BUFSIZE)]; + if (ReadDB(db, key, old_entries, sizeof(old_entries)) && + (memcmp(old_entries, raw_entries, size) == 0)) + { + Log(LOG_LEVEL_VERBOSE, "No changes in directory list"); + *change = false; + return true; + } + } + if (!WriteDB(db, key, raw_entries, size)) { Log(LOG_LEVEL_ERR, "Could not write to changes database"); return false; } + *change = true; return true; } @@ -618,18 +634,21 @@ void FileChangesCheckAndUpdateDirectory(const char *name, const Seq *file_set, c } } - PromiseResult newres; - if (!changed) - { - newres = PROMISE_RESULT_NOOP; - } - else if (update && FileChangesSetDirectoryList(db, name, disk_file_set)) - { - newres = PROMISE_RESULT_CHANGE; - } - else + bool dir_list_changes = false; + PromiseResult newres = PROMISE_RESULT_NOOP; + if (changed) { - newres = PROMISE_RESULT_FAIL; + if (update && FileChangesSetDirectoryList(db, name, disk_file_set, &dir_list_changes)) + { + if (dir_list_changes) + { + newres = PROMISE_RESULT_CHANGE; + } + } + else + { + newres = PROMISE_RESULT_FAIL; + } } *result = PromiseResultUpdate(*result, newres); From a022e31d3f4ff4f4e610efaa6a02915f9737ee94 Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Wed, 22 Jul 2020 14:12:45 +0200 Subject: [PATCH 127/333] ENT-5335 add workarounds for tests failing on SUSE. Issue is that `usermod` on suse doesn't like editing secondary groups together with any other attributes. In real life, this isn't a big issue - the promise will be completely repaired on second run. However, this test failed. This commit changes this behavior to expect secondary group to *not* be set after first promise run. When this get fixed, this commit should be reverted :) (cherry picked from commit 3d36de3949c0e4a5a1e7e7d3b3fabc1d6b822801) --- .../unsafe/10_modify_user_with_many_attributes.cf | 8 +++++++- .../unsafe/10_modify_user_with_many_attributes_warn.cf | 8 +++++++- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/tests/acceptance/17_users/unsafe/10_modify_user_with_many_attributes.cf b/tests/acceptance/17_users/unsafe/10_modify_user_with_many_attributes.cf index c5d90b48b7..f38842a3c9 100644 --- a/tests/acceptance/17_users/unsafe/10_modify_user_with_many_attributes.cf +++ b/tests/acceptance/17_users/unsafe/10_modify_user_with_many_attributes.cf @@ -123,7 +123,13 @@ bundle agent check "not_sgroup_success", "!not_sgroup_failure", }; windows:: "unix_ok" expression => "any"; - any:: + suse:: + "ok" -> "CFE-3386" + and => { "pgroup_success", "!pgroup_failure", "!sgroup_success", "sgroup_failure", + "hash_success", "!hash_failure", + "home_success", "!home_failure", "desc_success", "!desc_failure", + "unix_ok" }; + !suse:: "ok" and => { "pgroup_success", "!pgroup_failure", "sgroup_success", "!sgroup_failure", "hash_success", "!hash_failure", "home_success", "!home_failure", "desc_success", "!desc_failure", diff --git a/tests/acceptance/17_users/unsafe/10_modify_user_with_many_attributes_warn.cf b/tests/acceptance/17_users/unsafe/10_modify_user_with_many_attributes_warn.cf index 3f342e0bb1..f7dda05d15 100644 --- a/tests/acceptance/17_users/unsafe/10_modify_user_with_many_attributes_warn.cf +++ b/tests/acceptance/17_users/unsafe/10_modify_user_with_many_attributes_warn.cf @@ -131,7 +131,13 @@ bundle agent check "not_sgroup_success", "!not_sgroup_failure", }; windows:: "unix_ok" expression => "any"; - any:: + suse:: + "ok" -> "CFE-3386" + and => { "pgroup_success", "!pgroup_failure", "!sgroup_success", "sgroup_failure", + "hash_success", "!hash_failure", + "home_success", "!home_failure", "desc_success", "!desc_failure", + "unix_ok" }; + !suse:: "ok" and => { "pgroup_success", "!pgroup_failure", "sgroup_success", "!sgroup_failure", "hash_success", "!hash_failure", "home_success", "!home_failure", "desc_success", "!desc_failure", From c604c74c50d9d059c825c97f56d006333004d617 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 19 Aug 2020 14:26:40 +0200 Subject: [PATCH 128/333] Make the opposite() helper function inline It is trivial and should be inlined. Ticket: CFE-2434 Changelog: None (cherry picked from commit 9221b17995e8c94e9b8bab0bdd0920c8d359b261) --- libpromises/expand.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libpromises/expand.c b/libpromises/expand.c index b1eb00600d..99ac2c5def 100644 --- a/libpromises/expand.c +++ b/libpromises/expand.c @@ -1101,7 +1101,7 @@ bool IsExpandable(const char *str) /*********************************************************************/ -static char opposite(char c) +static inline char opposite(char c) { switch (c) { From 469ba964534d6f577142f868031e3eb6f21c8efb Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 19 Aug 2020 14:28:35 +0200 Subject: [PATCH 129/333] Add unit test for IsNakedVar() This function can play a significant role in policy function calls and 'methods' promises, among the other things. We should have tests for it. Also use a proper character literal as a value for a 'char' variable in the IsNakedVar() function. Ticket: CFE-2434 Changelog: None (cherry picked from commit 51f4fb20681d51b59999d30d7ff8c74416244959) --- libpromises/expand.c | 2 +- tests/unit/expand_test.c | 14 ++++++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/libpromises/expand.c b/libpromises/expand.c index 99ac2c5def..24f6f8f040 100644 --- a/libpromises/expand.c +++ b/libpromises/expand.c @@ -1122,7 +1122,7 @@ static inline char opposite(char c) bool IsNakedVar(const char *str, char vtype) { size_t len = strlen(str); - char last = len > 0 ? str[len-1] : 0; + char last = len > 0 ? str[len-1] : '\0'; if (len < 3 || str[0] != vtype diff --git a/tests/unit/expand_test.c b/tests/unit/expand_test.c index 08e1f5aa3b..35ba2d3825 100644 --- a/tests/unit/expand_test.c +++ b/tests/unit/expand_test.c @@ -62,6 +62,19 @@ static void test_extract_reference(void) test_extract_reference_("$(x${$(y)}) $(y) ${x${z}}", true, "$(x${$(y)})", "x${$(y)}"); } +static void test_isnakedvar() +{ + assert_true(IsNakedVar("$(whatever)", '$')); + assert_true(IsNakedVar("${whatever}", '$')); + assert_true(IsNakedVar("$(blah$(blue))", '$')); + + assert_false(IsNakedVar("$(blah)blue", '$')); + assert_false(IsNakedVar("blah$(blue)", '$')); + assert_false(IsNakedVar("$(blah)$(blue)", '$')); + assert_false(IsNakedVar("$(blah}", '$')); +} + + #if 0 static void test_map_iterators_from_rval_empty(void **state) { @@ -564,6 +577,7 @@ int main() { unit_test(test_extract_scalar_prefix), unit_test(test_extract_reference), + unit_test(test_isnakedvar), #if 0 unit_test_setup_teardown(test_map_iterators_from_rval_empty, test_setup, test_teardown), unit_test_setup_teardown(test_map_iterators_from_rval_literal, test_setup, test_teardown), From cbde9ea43fb7f6834ad25c38e9a3daf15ddda67a Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 19 Aug 2020 15:42:47 +0200 Subject: [PATCH 130/333] Expand variables specifying data/list names in @() references If there is a function or 'methods' promise argument in the '@($(container_name).field)' form, we need to expand the nested variable before working with the data/list argument. Changelog: Variables specifying data/list names in @() references are now expanded Ticket: CFE-2434 (cherry picked from commit 4474edc45cd2cc14932a95459fdf881d7609958f) Conflicts: libpromises/expand.c --- libpromises/expand.c | 16 ++++++++++++++++ .../04_containers/pass_variable_container.cf | 10 ++++------ .../pass_variable_container.cf.expected | 2 +- 3 files changed, 21 insertions(+), 7 deletions(-) diff --git a/libpromises/expand.c b/libpromises/expand.c index 24f6f8f040..90c841e430 100644 --- a/libpromises/expand.c +++ b/libpromises/expand.c @@ -345,10 +345,26 @@ Rval ExpandPrivateRval(EvalContext *ctx, return returnval; } +static inline bool StartsWithVariableDataListReference(const char *str) +{ + return ((str[0] == '@') && + ((str[1] == '{') || (str[1] == '(')) && + (str[2] == '$') && + ((str[3] == '{') || (str[3] == '('))); +} + static Rval ExpandListEntry(EvalContext *ctx, const char *ns, const char *scope, int expandnaked, Rval entry) { + /* If rval is something like '@($(container_name).field)', we need to expand + * the nested variable first. */ + if (entry.type == RVAL_TYPE_SCALAR && + StartsWithVariableDataListReference(entry.item)) + { + entry = ExpandPrivateRval(ctx, ns, scope, entry.item, entry.type); + } + if (entry.type == RVAL_TYPE_SCALAR && IsNakedVar(entry.item, '@')) { diff --git a/tests/acceptance/01_vars/04_containers/pass_variable_container.cf b/tests/acceptance/01_vars/04_containers/pass_variable_container.cf index e7e7029325..86a0655397 100644 --- a/tests/acceptance/01_vars/04_containers/pass_variable_container.cf +++ b/tests/acceptance/01_vars/04_containers/pass_variable_container.cf @@ -35,17 +35,15 @@ bundle agent test { meta: - "description" string => "Test that variable data containers can be passed."; - "test_soft_fail" - string => "any", - meta => { "CFE-2434" }; + "description" -> { "CFE-2434" } + string => "Test that variable data containers can be passed."; vars: "found" slist => bundlesmatching(".*", "find"); # "ns_bundle" string => "default:init"; # This is the same content as the # "found" variable, so we just use that instead. - "bundle" string => "init"; + "bundle_name" string => "init"; methods: # These work. @@ -54,7 +52,7 @@ bundle agent test # These do not work. (Note: Quotes are required for this, ideally they # wouldn't be but thats a seperate issue.) - "" usebundle => report_data("Pass by variable name", "@($(bundle).d)"); + "" usebundle => report_data("Pass by variable name:", "@($(bundle_name).d)"); "" usebundle => report_data("Pass by variable name including namespace:", "@($(found).d)"); } diff --git a/tests/acceptance/01_vars/04_containers/pass_variable_container.cf.expected b/tests/acceptance/01_vars/04_containers/pass_variable_container.cf.expected index fde9e8fc63..49ddbaf5aa 100644 --- a/tests/acceptance/01_vars/04_containers/pass_variable_container.cf.expected +++ b/tests/acceptance/01_vars/04_containers/pass_variable_container.cf.expected @@ -3,6 +3,6 @@ Pass by name: auto_master=/etc/auto.master Pass by namespace:name: hosts=/etc/hosts Pass by namespace:name: auto_master=/etc/auto.master Pass by variable name: hosts=/etc/hosts -Pass by variable:name: auto_master=/etc/auto.master +Pass by variable name: auto_master=/etc/auto.master Pass by variable name including namespace: hosts=/etc/hosts Pass by variable name including namespace: auto_master=/etc/auto.master From b6d9bc498d1cf053a874fd99fc38de396185ca1f Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 20 Aug 2020 10:37:06 +0200 Subject: [PATCH 131/333] Also expand variable data/list names with prefix Cases like '@(prefix$(var))' or '@(nspace:$(var))'. Ticket: CFE-2434 Changelog: None (cherry picked from commit 7c6fb23c21fa64f7d69ff49710fbb0fa3ab6777f) --- libpromises/expand.c | 62 ++++++++++++++++--- .../04_containers/pass_variable_container.cf | 4 +- .../pass_variable_container.cf.expected | 4 ++ 3 files changed, 62 insertions(+), 8 deletions(-) diff --git a/libpromises/expand.c b/libpromises/expand.c index 90c841e430..e1eaf6f3f4 100644 --- a/libpromises/expand.c +++ b/libpromises/expand.c @@ -43,7 +43,6 @@ #include #include - /** * VARIABLES AND PROMISE EXPANSION * @@ -105,6 +104,7 @@ * */ +static inline char opposite(char c); static void PutHandleVariable(EvalContext *ctx, const Promise *pp) { @@ -345,12 +345,60 @@ Rval ExpandPrivateRval(EvalContext *ctx, return returnval; } -static inline bool StartsWithVariableDataListReference(const char *str) +/** + * Detects a variable expansion inside of a data/list reference, for example + * "@(${container_name})" or "@(prefix${container_name})" or + * "@(nspace:${container_name})". + * + * @note This function doesn't have to be bullet-proof, it only needs to + * properly detect valid cases. The rest is left to the parser and code + * expanding variables. + */ +static inline bool ContainsVariableDataListReference(const char *str) { - return ((str[0] == '@') && - ((str[1] == '{') || (str[1] == '(')) && - (str[2] == '$') && - ((str[3] == '{') || (str[3] == '('))); + assert(str != NULL); + + size_t len = strlen(str); + + /* at least '@($(X))' is needed */ + if (len < 7) + { + return false; + } + + if (!((str[0] == '@') && + ((str[1] == '{') || (str[1] == '(')))) + { + return false; + } + + /* Check if, after '@(', there are only characters allowed in data/list + * names or ':' to separate namespace from the name followed by "$(" or "${" + * with a matching close bracket somewhere. */ + for (size_t i = 2; i < len; i++) + { + if (!(isalnum((int) str[i]) || + (str[i] == '_') || (str[i] == ':') || + (str[i] == '$'))) + { + return false; + } + + if (str[i] == '$') + { + if (((i + 1) < len) && ((str[i + 1] == '{') || (str[i + 1] == '('))) + { + int close_bracket = (int) opposite(str[i+1]); + return (strchr(str + i + 2, close_bracket) != NULL); + } + else + { + return false; + } + } + } + + return false; } static Rval ExpandListEntry(EvalContext *ctx, @@ -360,7 +408,7 @@ static Rval ExpandListEntry(EvalContext *ctx, /* If rval is something like '@($(container_name).field)', we need to expand * the nested variable first. */ if (entry.type == RVAL_TYPE_SCALAR && - StartsWithVariableDataListReference(entry.item)) + ContainsVariableDataListReference(entry.item)) { entry = ExpandPrivateRval(ctx, ns, scope, entry.item, entry.type); } diff --git a/tests/acceptance/01_vars/04_containers/pass_variable_container.cf b/tests/acceptance/01_vars/04_containers/pass_variable_container.cf index 86a0655397..0490b27417 100644 --- a/tests/acceptance/01_vars/04_containers/pass_variable_container.cf +++ b/tests/acceptance/01_vars/04_containers/pass_variable_container.cf @@ -44,6 +44,7 @@ bundle agent test # "found" variable, so we just use that instead. "bundle_name" string => "init"; + "bundle_name_suffix" string => "it"; methods: # These work. @@ -54,7 +55,8 @@ bundle agent test # wouldn't be but thats a seperate issue.) "" usebundle => report_data("Pass by variable name:", "@($(bundle_name).d)"); "" usebundle => report_data("Pass by variable name including namespace:", "@($(found).d)"); - + "" usebundle => report_data("Pass by variable name with prefix:", "@(in$(bundle_name_suffix).d)"); + "" usebundle => report_data("Pass by variable name with namespace:", "@(default:$(bundle_name).d)"); } bundle agent check diff --git a/tests/acceptance/01_vars/04_containers/pass_variable_container.cf.expected b/tests/acceptance/01_vars/04_containers/pass_variable_container.cf.expected index 49ddbaf5aa..53d03e88c4 100644 --- a/tests/acceptance/01_vars/04_containers/pass_variable_container.cf.expected +++ b/tests/acceptance/01_vars/04_containers/pass_variable_container.cf.expected @@ -6,3 +6,7 @@ Pass by variable name: hosts=/etc/hosts Pass by variable name: auto_master=/etc/auto.master Pass by variable name including namespace: hosts=/etc/hosts Pass by variable name including namespace: auto_master=/etc/auto.master +Pass by variable name with prefix: hosts=/etc/hosts +Pass by variable name with prefix: auto_master=/etc/auto.master +Pass by variable name with namespace: hosts=/etc/hosts +Pass by variable name with namespace: auto_master=/etc/auto.master From 7f131dce1c7319fd22bed7cfce18ad6ff96e785d Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Sat, 15 Aug 2020 19:49:46 +0200 Subject: [PATCH 132/333] ENT-6098 Fix how we check for `--cols` argument to `ps`. Issue was that on newer suse's (12 and 15), `ps` menions this argument only in `ps --help output` only. (cherry picked from commit 93cb76fb876a1baf7d857db2855c5322a6e7430b) --- misc/init.d/cfengine3.in | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/misc/init.d/cfengine3.in b/misc/init.d/cfengine3.in index 577a85b3c5..2965d261bb 100644 --- a/misc/init.d/cfengine3.in +++ b/misc/init.d/cfengine3.in @@ -150,10 +150,11 @@ fi CURRENT_PS_UID=__none__ INSIDE_CONTAINER=-1 -if ps --help 2>/dev/null | egrep -e '--cols\b' > /dev/null; then +if ps --help 2>/dev/null | egrep -e '--cols\b' > /dev/null || ps --help output 2>/dev/null | egrep -e '--cols\b'; then # There is a bug in SUSE which means that ps output will be truncated even # when piped to grep, if the terminal size is small. However using --cols - # will override it. + # will override it. NOTE: On Suse 12 and 15, `ps` mentions `--cols` only + # in `ps --help output` only. PS_OPTIONS="--cols 200" else PS_OPTIONS= From 47433c14ae00707a314b2c82c05903326c131c10 Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Sun, 16 Aug 2020 15:22:32 +0200 Subject: [PATCH 133/333] ENT-6098 limit SUSE workaround on failing tests to suse 11 only Because the issue we're working around is fixed in SUSE 12 and later (cherry picked from commit e5c077d7294d09b3d4cb29023aeff87495071394) --- .../17_users/unsafe/10_modify_user_with_many_attributes.cf | 4 ++-- .../unsafe/10_modify_user_with_many_attributes_warn.cf | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/acceptance/17_users/unsafe/10_modify_user_with_many_attributes.cf b/tests/acceptance/17_users/unsafe/10_modify_user_with_many_attributes.cf index f38842a3c9..655c26c84a 100644 --- a/tests/acceptance/17_users/unsafe/10_modify_user_with_many_attributes.cf +++ b/tests/acceptance/17_users/unsafe/10_modify_user_with_many_attributes.cf @@ -123,13 +123,13 @@ bundle agent check "not_sgroup_success", "!not_sgroup_failure", }; windows:: "unix_ok" expression => "any"; - suse:: + sles_11:: "ok" -> "CFE-3386" and => { "pgroup_success", "!pgroup_failure", "!sgroup_success", "sgroup_failure", "hash_success", "!hash_failure", "home_success", "!home_failure", "desc_success", "!desc_failure", "unix_ok" }; - !suse:: + !sles_11:: "ok" and => { "pgroup_success", "!pgroup_failure", "sgroup_success", "!sgroup_failure", "hash_success", "!hash_failure", "home_success", "!home_failure", "desc_success", "!desc_failure", diff --git a/tests/acceptance/17_users/unsafe/10_modify_user_with_many_attributes_warn.cf b/tests/acceptance/17_users/unsafe/10_modify_user_with_many_attributes_warn.cf index f7dda05d15..5366a0953d 100644 --- a/tests/acceptance/17_users/unsafe/10_modify_user_with_many_attributes_warn.cf +++ b/tests/acceptance/17_users/unsafe/10_modify_user_with_many_attributes_warn.cf @@ -131,13 +131,13 @@ bundle agent check "not_sgroup_success", "!not_sgroup_failure", }; windows:: "unix_ok" expression => "any"; - suse:: + sles_11:: "ok" -> "CFE-3386" and => { "pgroup_success", "!pgroup_failure", "!sgroup_success", "sgroup_failure", "hash_success", "!hash_failure", "home_success", "!home_failure", "desc_success", "!desc_failure", "unix_ok" }; - !suse:: + !sles_11:: "ok" and => { "pgroup_success", "!pgroup_failure", "sgroup_success", "!sgroup_failure", "hash_success", "!hash_failure", "home_success", "!home_failure", "desc_success", "!desc_failure", From ebf1d4863429dc0c051b34ad0765f7ce0cc06bab Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Mon, 24 Aug 2020 11:57:55 +0200 Subject: [PATCH 134/333] Only store path to the 'netstat' binary in VNETSTAT in cf-monitord/mon_network.c The only place where this string is used needs the path to the binary with no extra options. Also use a smaller buffer for the path to the binary. Ticket: CFE-2945 Changelog: None (cherry picked from commit 70767135ba03e779dc68a817481c3224e4cd10a6) --- cf-monitord/mon_network.c | 41 ++++++++++++++++++++------------------- 1 file changed, 21 insertions(+), 20 deletions(-) diff --git a/cf-monitord/mon_network.c b/cf-monitord/mon_network.c index 1e194ff8cd..48bee4e8f1 100644 --- a/cf-monitord/mon_network.c +++ b/cf-monitord/mon_network.c @@ -79,26 +79,26 @@ static const Sock ECGSOCKS[] = /* extended to map old to new using enum */ static const char *const VNETSTAT[] = { [PLATFORM_CONTEXT_UNKNOWN] = "-", - [PLATFORM_CONTEXT_OPENVZ] = "/bin/netstat -rn", /* virt_host_vz_vzps */ - [PLATFORM_CONTEXT_HP] = "/usr/bin/netstat -rn", /* hpux */ - [PLATFORM_CONTEXT_AIX] = "/usr/bin/netstat -rn", /* aix */ - [PLATFORM_CONTEXT_LINUX] = "/bin/netstat -rn", /* linux */ - [PLATFORM_CONTEXT_BUSYBOX] = "/bin/netstat -rn", /* linux */ - [PLATFORM_CONTEXT_SOLARIS] = "/usr/bin/netstat -rn", /* solaris */ - [PLATFORM_CONTEXT_SUN_SOLARIS] = "/usr/bin/netstat -rn", /* solaris */ - [PLATFORM_CONTEXT_FREEBSD] = "/usr/bin/netstat -rn", /* freebsd */ - [PLATFORM_CONTEXT_NETBSD] = "/usr/bin/netstat -rn", /* netbsd */ - [PLATFORM_CONTEXT_CRAYOS] = "/usr/ucb/netstat -rn", /* cray */ + [PLATFORM_CONTEXT_OPENVZ] = "/bin/netstat", /* virt_host_vz_vzps */ + [PLATFORM_CONTEXT_HP] = "/usr/bin/netstat", /* hpux */ + [PLATFORM_CONTEXT_AIX] = "/usr/bin/netstat", /* aix */ + [PLATFORM_CONTEXT_LINUX] = "/bin/netstat", /* linux */ + [PLATFORM_CONTEXT_BUSYBOX] = "/bin/netstat", /* linux */ + [PLATFORM_CONTEXT_SOLARIS] = "/usr/bin/netstat", /* solaris */ + [PLATFORM_CONTEXT_SUN_SOLARIS] = "/usr/bin/netstat", /* solaris */ + [PLATFORM_CONTEXT_FREEBSD] = "/usr/bin/netstat", /* freebsd */ + [PLATFORM_CONTEXT_NETBSD] = "/usr/bin/netstat", /* netbsd */ + [PLATFORM_CONTEXT_CRAYOS] = "/usr/ucb/netstat", /* cray */ [PLATFORM_CONTEXT_WINDOWS_NT] = "/cygdrive/c/WINNT/System32/netstat", /* CygWin */ - [PLATFORM_CONTEXT_SYSTEMV] = "/usr/bin/netstat -rn", /* Unixware */ - [PLATFORM_CONTEXT_OPENBSD] = "/usr/bin/netstat -rn", /* openbsd */ - [PLATFORM_CONTEXT_CFSCO] = "/usr/bin/netstat -rn", /* sco */ - [PLATFORM_CONTEXT_DARWIN] = "/usr/sbin/netstat -rn", /* darwin */ - [PLATFORM_CONTEXT_QNX] = "/usr/bin/netstat -rn", /* qnx */ - [PLATFORM_CONTEXT_DRAGONFLY] = "/usr/bin/netstat -rn", /* dragonfly */ + [PLATFORM_CONTEXT_SYSTEMV] = "/usr/bin/netstat", /* Unixware */ + [PLATFORM_CONTEXT_OPENBSD] = "/usr/bin/netstat", /* openbsd */ + [PLATFORM_CONTEXT_CFSCO] = "/usr/bin/netstat", /* sco */ + [PLATFORM_CONTEXT_DARWIN] = "/usr/sbin/netstat", /* darwin */ + [PLATFORM_CONTEXT_QNX] = "/usr/bin/netstat", /* qnx */ + [PLATFORM_CONTEXT_DRAGONFLY] = "/usr/bin/netstat", /* dragonfly */ [PLATFORM_CONTEXT_MINGW] = "mingw-invalid", /* mingw */ [PLATFORM_CONTEXT_VMWARE] = "/usr/bin/netstat", /* vmware */ - [PLATFORM_CONTEXT_ANDROID] = "/system/xbin/netstat -rn", /* android */ + [PLATFORM_CONTEXT_ANDROID] = "/system/xbin/netstat", /* android */ }; /* Implementation */ @@ -186,7 +186,8 @@ static void SetNetworkEntropyClasses(const char *service, const char *direction, void MonNetworkGatherData(double *cf_this) { FILE *pp; - char local[CF_BUFSIZE], remote[CF_BUFSIZE], comm[CF_BUFSIZE]; + char local[CF_BUFSIZE], remote[CF_BUFSIZE]; + char comm[PATH_MAX + 4] = {0}; /* path to the binary + " -an" */ Item *in[ATTR], *out[ATTR]; char *sp; int i; @@ -207,7 +208,7 @@ void MonNetworkGatherData(double *cf_this) DeleteItemList(MON_UDP6); MON_UDP4 = MON_UDP6 = MON_TCP4 = MON_TCP6 = NULL; - sscanf(VNETSTAT[VSYSTEMHARDCLASS], "%s", comm); + strncpy(comm, VNETSTAT[VSYSTEMHARDCLASS], (sizeof(comm) - 1)); if (!FileCanOpen(comm, "r")) { @@ -217,7 +218,7 @@ void MonNetworkGatherData(double *cf_this) return; } - strcat(comm, " -an"); + strncat(comm, " -an", sizeof(comm) - 1); if ((pp = cf_popen(comm, "r", true)) == NULL) { From 485bfb726a68df747e45f67914ebd5f4b695e7a1 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Mon, 24 Aug 2020 16:17:37 +0200 Subject: [PATCH 135/333] Use smaller buffers for IP:PORT specs in MonNetworkGatherData() No need to waste space. Also let the compiler generate the code to zero the buffers. Ticket: CFE-2945 Changelog: None (cherry picked from commit 0a64e87735bcf66222bd06b1078b4de2223bfef8) --- cf-monitord/mon_network.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/cf-monitord/mon_network.c b/cf-monitord/mon_network.c index 48bee4e8f1..9b77fbfa6a 100644 --- a/cf-monitord/mon_network.c +++ b/cf-monitord/mon_network.c @@ -186,7 +186,6 @@ static void SetNetworkEntropyClasses(const char *service, const char *direction, void MonNetworkGatherData(double *cf_this) { FILE *pp; - char local[CF_BUFSIZE], remote[CF_BUFSIZE]; char comm[PATH_MAX + 4] = {0}; /* path to the binary + " -an" */ Item *in[ATTR], *out[ATTR]; char *sp; @@ -230,11 +229,10 @@ void MonNetworkGatherData(double *cf_this) size_t vbuff_size = CF_BUFSIZE; char *vbuff = xmalloc(vbuff_size); - for (;;) { - memset(local, 0, CF_BUFSIZE); - memset(remote, 0, CF_BUFSIZE); + char local[CF_MAX_IP_LEN + CF_MAX_PORT_LEN] = {0}; + char remote[CF_MAX_IP_LEN + CF_MAX_PORT_LEN] = {0}; size_t res = CfReadLine(&vbuff, &vbuff_size, pp); if (res == -1) From 4b580f949640f9fef55b8224fddbf5289cefa788 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Tue, 25 Aug 2020 10:42:05 +0200 Subject: [PATCH 136/333] Split saving network data from MonNetworkGatherData() The function was too long and saving network data has nothing to do with parsing the data. Ticket: CFE-2945 Changelog: None (cherry picked from commit 3d2d91db41ae6f8fabffff378b889e3a1a2cc20d) --- cf-monitord/mon_network.c | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/cf-monitord/mon_network.c b/cf-monitord/mon_network.c index 9b77fbfa6a..13627dde6c 100644 --- a/cf-monitord/mon_network.c +++ b/cf-monitord/mon_network.c @@ -183,6 +183,8 @@ static void SetNetworkEntropyClasses(const char *service, const char *direction, /******************************************************************************/ +static void SaveNetworkData(Item * const *in, Item * const *out); + void MonNetworkGatherData(double *cf_this) { FILE *pp; @@ -399,16 +401,22 @@ void MonNetworkGatherData(double *cf_this) } } } + free(vbuff); cf_pclose(pp); -/* Now save the state for ShowState() - the state is not smaller than the last or at least 40 minutes - older. This mirrors the persistence of the maxima classes */ + /* Now save the state for ShowState() + the state is not smaller than the last or at least 40 minutes + older. This mirrors the persistence of the maxima classes */ + SaveNetworkData(in, out); +} +static void SaveNetworkData(Item * const *in, Item * const *out) +{ const char* const statedir = GetStateDir(); - for (i = 0; i < ATTR; i++) + char vbuff[CF_BUFSIZE]; + for (size_t i = 0; i < ATTR; i++) { struct stat statbuf; time_t now = time(NULL); @@ -436,7 +444,7 @@ void MonNetworkGatherData(double *cf_this) Log(LOG_LEVEL_DEBUG, "Saved in netstat data in '%s'", vbuff); } - for (i = 0; i < ATTR; i++) + for (size_t i = 0; i < ATTR; i++) { struct stat statbuf; time_t now = time(NULL); @@ -462,6 +470,4 @@ void MonNetworkGatherData(double *cf_this) Log(LOG_LEVEL_DEBUG, "Saved out netstat data in '%s'", vbuff); DeleteItemList(out[i]); } - - free(vbuff); } From 50e13246fd35360de78630e9e2241306aeb85e58 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Tue, 25 Aug 2020 10:47:51 +0200 Subject: [PATCH 137/333] Split parsing 'netstat' data from from MonNetworkGatherData() To better structure the code and clearly separate data parsing from further processing the data. Ticket: CFE-2945 Changelog: None (cherry picked from commit c6477664399cd796f1d7ca52cc4f3380ab5018c4) --- cf-monitord/mon_network.c | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-) diff --git a/cf-monitord/mon_network.c b/cf-monitord/mon_network.c index 13627dde6c..1d5e4c178a 100644 --- a/cf-monitord/mon_network.c +++ b/cf-monitord/mon_network.c @@ -184,6 +184,7 @@ static void SetNetworkEntropyClasses(const char *service, const char *direction, /******************************************************************************/ static void SaveNetworkData(Item * const *in, Item * const *out); +static void GetNetworkDataFromNetstat(FILE *fp, double *cf_this, Item **in, Item **out); void MonNetworkGatherData(double *cf_this) { @@ -192,8 +193,6 @@ void MonNetworkGatherData(double *cf_this) Item *in[ATTR], *out[ATTR]; char *sp; int i; - enum cf_netstat_type { cfn_new, cfn_old } type = cfn_new; - enum cf_packet_type { cfn_udp4, cfn_udp6, cfn_tcp4, cfn_tcp6} packet = cfn_tcp4; for (i = 0; i < ATTR; i++) { @@ -228,6 +227,19 @@ void MonNetworkGatherData(double *cf_this) comm); return; } + GetNetworkDataFromNetstat(pp, cf_this, in, out); + cf_pclose(pp); + + /* Now save the state for ShowState() + the state is not smaller than the last or at least 40 minutes + older. This mirrors the persistence of the maxima classes */ + SaveNetworkData(in, out); +} + +static void GetNetworkDataFromNetstat(FILE *fp, double *cf_this, Item **in, Item **out) +{ + enum cf_netstat_type { cfn_new, cfn_old } type = cfn_new; + enum cf_packet_type { cfn_udp4, cfn_udp6, cfn_tcp4, cfn_tcp6} packet = cfn_tcp4; size_t vbuff_size = CF_BUFSIZE; char *vbuff = xmalloc(vbuff_size); @@ -236,16 +248,14 @@ void MonNetworkGatherData(double *cf_this) char local[CF_MAX_IP_LEN + CF_MAX_PORT_LEN] = {0}; char remote[CF_MAX_IP_LEN + CF_MAX_PORT_LEN] = {0}; - size_t res = CfReadLine(&vbuff, &vbuff_size, pp); + size_t res = CfReadLine(&vbuff, &vbuff_size, fp); if (res == -1) { - if (!feof(pp)) + if (!feof(fp)) { Log(LOG_LEVEL_DEBUG, - "Error occured while reading '%s' " - "(CfReadLine/getline returned -1 but no EOF found)", - comm); - cf_pclose(pp); + "Error occured while reading data from 'netstat' " + "(CfReadLine/getline returned -1 but no EOF found)"); free(vbuff); return; } @@ -336,6 +346,7 @@ void MonNetworkGatherData(double *cf_this) // Extract the port number from the end of the string + char *sp; for (sp = local + strlen(local); (*sp != '.') && (*sp != ':') && (sp > local); sp--) { } @@ -384,7 +395,7 @@ void MonNetworkGatherData(double *cf_this) // Now look for the specific vital signs to count frequencies - for (i = 0; i < ATTR; i++) + for (size_t i = 0; i < ATTR; i++) { if (strcmp(localport, ECGSOCKS[i].portnr) == 0) { @@ -402,13 +413,6 @@ void MonNetworkGatherData(double *cf_this) } } free(vbuff); - - cf_pclose(pp); - - /* Now save the state for ShowState() - the state is not smaller than the last or at least 40 minutes - older. This mirrors the persistence of the maxima classes */ - SaveNetworkData(in, out); } static void SaveNetworkData(Item * const *in, Item * const *out) From 496ca251e4d7eed0c1b6516fde5cfbaac7f3cc7a Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Tue, 25 Aug 2020 10:58:19 +0200 Subject: [PATCH 138/333] Refactor MonNetworkGatherData() Declare variables near the place where they are being used for the first time and move resetting network data lists into a separate (inline) function. Ticket: CFE-2945 Changelog: None (cherry picked from commit 14c25a560c772f0ba603b47f6c61839c169e82be) --- cf-monitord/mon_network.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/cf-monitord/mon_network.c b/cf-monitord/mon_network.c index 1d5e4c178a..03c700493f 100644 --- a/cf-monitord/mon_network.c +++ b/cf-monitord/mon_network.c @@ -186,19 +186,8 @@ static void SetNetworkEntropyClasses(const char *service, const char *direction, static void SaveNetworkData(Item * const *in, Item * const *out); static void GetNetworkDataFromNetstat(FILE *fp, double *cf_this, Item **in, Item **out); -void MonNetworkGatherData(double *cf_this) +static inline void ResetNetworkData() { - FILE *pp; - char comm[PATH_MAX + 4] = {0}; /* path to the binary + " -an" */ - Item *in[ATTR], *out[ATTR]; - char *sp; - int i; - - for (i = 0; i < ATTR; i++) - { - in[i] = out[i] = NULL; - } - DeleteItemList(ALL_INCOMING); ALL_INCOMING = NULL; @@ -207,7 +196,13 @@ void MonNetworkGatherData(double *cf_this) DeleteItemList(MON_UDP4); DeleteItemList(MON_UDP6); MON_UDP4 = MON_UDP6 = MON_TCP4 = MON_TCP6 = NULL; +} + +void MonNetworkGatherData(double *cf_this) +{ + ResetNetworkData(); + char comm[PATH_MAX + 4] = {0}; /* path to the binary + " -an" */ strncpy(comm, VNETSTAT[VSYSTEMHARDCLASS], (sizeof(comm) - 1)); if (!FileCanOpen(comm, "r")) @@ -220,6 +215,7 @@ void MonNetworkGatherData(double *cf_this) strncat(comm, " -an", sizeof(comm) - 1); + FILE *pp; if ((pp = cf_popen(comm, "r", true)) == NULL) { Log(LOG_LEVEL_VERBOSE, @@ -227,6 +223,9 @@ void MonNetworkGatherData(double *cf_this) comm); return; } + + Item *in[ATTR] = {0}; + Item *out[ATTR] = {0}; GetNetworkDataFromNetstat(pp, cf_this, in, out); cf_pclose(pp); From 7919d87d92725f9a00c6780e73e382f6cf987509 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 26 Aug 2020 22:08:49 +0200 Subject: [PATCH 139/333] Add code for parsing /proc/net/{tcp,tcp6,udp,upd6} So that we can use the data instead of running 'netstat'. Ticket: CFE-2945 Changelog: None (cherry picked from commit e62a24501535f37559850b686682266ce7849440) --- .gitignore | 1 + cf-monitord/Makefile.am | 13 ++- cf-monitord/get_socket_info.c | 189 +++++++++++++++++++++++++++++++++ cf-monitord/proc_net_parsing.c | 153 ++++++++++++++++++++++++++ cf-monitord/proc_net_parsing.h | 67 ++++++++++++ libntech | 2 +- 6 files changed, 423 insertions(+), 2 deletions(-) create mode 100644 cf-monitord/get_socket_info.c create mode 100644 cf-monitord/proc_net_parsing.c create mode 100644 cf-monitord/proc_net_parsing.h diff --git a/.gitignore b/.gitignore index 4a16d90378..5a0e26c6b1 100644 --- a/.gitignore +++ b/.gitignore @@ -86,6 +86,7 @@ stamp-h1 /cf-key/cf-key.exe /cf-monitord/cf-monitord /cf-monitord/cf-monitord.exe +/cf-monitord/get_socket_info /cf-promises/cf-promises /cf-promises/cf-promises.exe /cf-runagent/cf-runagent diff --git a/cf-monitord/Makefile.am b/cf-monitord/Makefile.am index f459aff6cc..10edfb4a29 100644 --- a/cf-monitord/Makefile.am +++ b/cf-monitord/Makefile.am @@ -57,7 +57,7 @@ libcf_monitord_la_SOURCES = \ cf-monitord.c if LINUX -libcf_monitord_la_SOURCES += mon_io_linux.c mon_mem_linux.c +libcf_monitord_la_SOURCES += mon_io_linux.c mon_mem_linux.c proc_net_parsing.c proc_net_parsing.h endif if SOLARIS @@ -71,6 +71,17 @@ libcf_monitord_la_SOURCES += mon_mem_stub.c endif endif +TESTS = + +if LINUX +TESTS += get_socket_info +noinst_PROGRAMS = get_socket_info +get_socket_info_SOURCES = get_socket_info.c +get_socket_info_LDADD = ${top_srcdir}/libntech/libutils/libutils.la \ + $(PCRE_LIBS) \ + $(LIBYAML_LIBS) \ + libcf-monitord.la +endif if !BUILTIN_EXTENSIONS bin_PROGRAMS = cf-monitord diff --git a/cf-monitord/get_socket_info.c b/cf-monitord/get_socket_info.c new file mode 100644 index 0000000000..df461b304f --- /dev/null +++ b/cf-monitord/get_socket_info.c @@ -0,0 +1,189 @@ +/* + Copyright 2020 Northern.tech AS + + This file is part of CFEngine 3 - written and maintained by Northern.tech AS. + + This program is free software; you can redistribute it and/or modify it + under the terms of the GNU General Public License as published by the + Free Software Foundation; version 3. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA + + To the extent this program is licensed as part of the Enterprise + versions of CFEngine, the applicable Commercial Open Source License + (COSL) may apply to this file if you as a licensee so wish it. See + included file COSL.txt. +*/ + +#include +#include + +#include +#include +#include +#include + +#include + +int main() +{ + FILE *fp = fopen("/proc/net/tcp", "r"); + + size_t buff_size = 256; + char *buff = xmalloc(buff_size); + + /* Read the header */ + ssize_t ret = CfReadLine(&buff, &buff_size, fp); + if (ret == -1) + { + Log(LOG_LEVEL_ERR, "Failed to read data from /proc/net/tcp"); + free(buff); + return 1; + } + + /* Read real data */ + Seq *lines = SeqNew(64, free); + ret = CfReadLines(&buff, &buff_size, fp, lines); + if (ret == -1) + { + Log(LOG_LEVEL_ERR, "Failed to read data from /proc/net/tcp"); + SeqDestroy(lines); + free(buff); + return 1; + } + + /* else */ + char local_addr[INET_ADDRSTRLEN]; + char remote_addr[INET_ADDRSTRLEN]; + uint32_t l_port, r_port; + SocketState state; + for (size_t i = 0; i < ret; i++) + { + char *line = SeqAt(lines,i); + if (ParseIPv4SocketInfo(line, local_addr, &l_port, remote_addr, &r_port, &state)) + { + printf("%s:%d -> %s:%d%s\n", + local_addr, l_port, + remote_addr, r_port, + (state == SOCK_STATE_LISTEN) ? " [LISTEN]" : ""); + } + } + SeqClear(lines); + + char local_addr6[INET6_ADDRSTRLEN]; + char remote_addr6[INET6_ADDRSTRLEN]; + fp = fopen("/proc/net/tcp6", "r"); + if (fp != NULL) + { + ret = CfReadLine(&buff, &buff_size, fp); + if (ret == -1) + { + Log(LOG_LEVEL_ERR, "Failed to read data from /proc/net/tcp6"); + SeqDestroy(lines); + free(buff); + return 1; + } + + /* Read real data */ + ret = CfReadLines(&buff, &buff_size, fp, lines); + if (ret == -1) + { + Log(LOG_LEVEL_ERR, "Failed to read data from /proc/net/tcp6"); + SeqDestroy(lines); + free(buff); + return 1; + } + + /* else */ + for (size_t i = 0; i < ret; i++) + { + char *line = SeqAt(lines,i); + if (ParseIPv6SocketInfo(line, local_addr6, &l_port, remote_addr6, &r_port, &state)) + { + printf("%s:%d -> %s:%d%s\n", + local_addr6, l_port, + remote_addr6, r_port, + (state == SOCK_STATE_LISTEN) ? " [LISTEN]" : ""); + } + } + SeqClear(lines); + } + + fp = fopen("/proc/net/udp", "r"); + ret = CfReadLine(&buff, &buff_size, fp); + if (ret == -1) + { + Log(LOG_LEVEL_ERR, "Failed to read data from /proc/net/udp"); + SeqDestroy(lines); + free(buff); + return 1; + } + + /* Read real data */ + ret = CfReadLines(&buff, &buff_size, fp, lines); + if (ret == -1) + { + Log(LOG_LEVEL_ERR, "Failed to read data from /proc/net/udp"); + SeqDestroy(lines); + free(buff); + return 1; + } + + /* else */ + for (size_t i = 0; i < ret; i++) + { + char *line = SeqAt(lines,i); + if (ParseIPv4SocketInfo(line, local_addr, &l_port, remote_addr, &r_port, &state)) + { + printf("%s:%d -> %s:%d [udp]\n", + local_addr, l_port, + remote_addr, r_port); + } + } + SeqClear(lines); + + fp = fopen("/proc/net/udp6", "r"); + if (fp != NULL) + { + ret = CfReadLine(&buff, &buff_size, fp); + if (ret == -1) + { + Log(LOG_LEVEL_ERR, "Failed to read data from /proc/net/udp6"); + SeqDestroy(lines); + free(buff); + return 1; + } + + /* Read real data */ + ret = CfReadLines(&buff, &buff_size, fp, lines); + if (ret == -1) + { + Log(LOG_LEVEL_ERR, "Failed to read data from /proc/net/udp6"); + SeqDestroy(lines); + free(buff); + return 1; + } + + /* else */ + for (size_t i = 0; i < ret; i++) + { + char *line = SeqAt(lines,i); + if (ParseIPv6SocketInfo(line, local_addr6, &l_port, remote_addr6, &r_port, &state)) + { + printf("%s:%d -> %s:%d [udp6]\n", + local_addr6, l_port, + remote_addr6, r_port); + } + } + } + SeqDestroy(lines); + free(buff); + return 0; +} diff --git a/cf-monitord/proc_net_parsing.c b/cf-monitord/proc_net_parsing.c new file mode 100644 index 0000000000..f8c99c3c8c --- /dev/null +++ b/cf-monitord/proc_net_parsing.c @@ -0,0 +1,153 @@ +/* + Copyright 2020 Northern.tech AS + + This file is part of CFEngine 3 - written and maintained by Northern.tech AS. + + This program is free software; you can redistribute it and/or modify it + under the terms of the GNU General Public License as published by the + Free Software Foundation; version 3. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA + + To the extent this program is licensed as part of the Enterprise + versions of CFEngine, the applicable Commercial Open Source License + (COSL) may apply to this file if you as a licensee so wish it. See + included file COSL.txt. +*/ + +#include +#include + +#include +#include + +/* Inspired by https://git.kernel.org/pub/scm/network/iproute2/iproute2.git/tree/misc/ss.c#n2404 */ +static bool SplitProcNetLine(const char *line, char **local, char **remote, char **state) +{ + /* Example data: + * 2: B400A8C0:9DAE 38420D1F:01BB 01 00000000:00000000 ... (IPv4) + * 0: 00000000000000000000000001000000:0277 00000000000000000000000000000000:0000 0A 00000000:00000000 ... (IPv6) + */ + char *p; + + if ((p = strchr(line, ':')) == NULL) + { + return false; + } + + *local = p + 2; + if ((p = strchr(*local, ':')) == NULL) + { + return false; + } + + *remote = p + 6; + if ((p = strchr(*remote, ':')) == NULL) + { + return false; + } + + *state = p + 6; + return true; +} + +bool ParseIPv4SocketInfo(const char *line, + char local_addr[INET_ADDRSTRLEN], uint32_t *local_port, + char remote_addr[INET_ADDRSTRLEN], uint32_t *remote_port, + SocketState *state) +{ + char *local, *remote, *state_str; + if (!SplitProcNetLine(line, &local, &remote, &state_str)) + { + return false; + } + + /* else */ + struct in_addr l_addr; + if (sscanf(local, "%x:%x", &(l_addr.s_addr), local_port) != 2) + { + Log(LOG_LEVEL_ERR,"Failed to parse local addr:port from line '%s'\n", line); + return false; + } + struct in_addr r_addr; + if (sscanf(remote, "%x:%x", &(r_addr.s_addr), remote_port) != 2) + { + Log(LOG_LEVEL_ERR, "Failed to parse remote addr:port from line '%s'\n", line); + return false; + } + + if (sscanf(state_str, "%x", state) != 1) + { + Log(LOG_LEVEL_ERR, "Failed to parse state from line '%s'\n", line); + return false; + } + + if (inet_ntop(AF_INET, &l_addr, local_addr, INET_ADDRSTRLEN) == NULL) + { + Log(LOG_LEVEL_ERR, "Failed to get ASCII representation of local address"); + return false; + } + if (inet_ntop(AF_INET, &r_addr, remote_addr, INET_ADDRSTRLEN) == NULL) + { + Log(LOG_LEVEL_ERR, "Failed to get ASCII representation of remote address"); + return false; + } + return true; +} + +bool ParseIPv6SocketInfo(const char *line, + char local_addr[INET6_ADDRSTRLEN], uint32_t *local_port, + char remote_addr[INET6_ADDRSTRLEN], uint32_t *remote_port, + SocketState *state) +{ + char *local, *remote, *state_str; + if (!SplitProcNetLine(line, &local, &remote, &state_str)) + { + return false; + } + + /* else */ + struct in6_addr l_addr; + uint32_t *addr_data = (uint32_t*) l_addr.s6_addr; + if (sscanf(local, "%08x%08x%08x%08x:%x", + addr_data, addr_data + 1, addr_data + 2, addr_data + 3, + local_port) != 5) + { + Log(LOG_LEVEL_ERR,"Failed to parse local addr:port from line '%s'\n", line); + return false; + } + struct in6_addr r_addr; + addr_data = (uint32_t*) r_addr.s6_addr; + if (sscanf(remote, "%08x%08x%08x%08x:%x", + addr_data, addr_data + 1, addr_data + 2, addr_data + 3, + remote_port) != 5) + { + Log(LOG_LEVEL_ERR, "Failed to parse remote addr:port from line '%s'\n", line); + return false; + } + + if (sscanf(state_str, "%x", state) != 1) + { + Log(LOG_LEVEL_ERR, "Failed to parse state from line '%s'\n", line); + return false; + } + + if (inet_ntop(AF_INET6, &l_addr, local_addr, INET6_ADDRSTRLEN) == NULL) + { + Log(LOG_LEVEL_ERR, "Failed to get ASCII representation of local address"); + return false; + } + if (inet_ntop(AF_INET6, &r_addr, remote_addr, INET6_ADDRSTRLEN) == NULL) + { + Log(LOG_LEVEL_ERR, "Failed to get ASCII representation of remote address"); + return false; + } + return true; +} diff --git a/cf-monitord/proc_net_parsing.h b/cf-monitord/proc_net_parsing.h new file mode 100644 index 0000000000..0b58c7e593 --- /dev/null +++ b/cf-monitord/proc_net_parsing.h @@ -0,0 +1,67 @@ +/* + Copyright 2020 Northern.tech AS + + This file is part of CFEngine 3 - written and maintained by Northern.tech AS. + + This program is free software; you can redistribute it and/or modify it + under the terms of the GNU General Public License as published by the + Free Software Foundation; version 3. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA + + To the extent this program is licensed as part of the Enterprise + versions of CFEngine, the applicable Commercial Open Source License + (COSL) may apply to this file if you as a licensee so wish it. See + included file COSL.txt. +*/ + +#ifndef _CFE_PROC_NET_PARSING_H_ +#define _CFE_PROC_NET_PARSING_H_ + +#include + +/* Taken from https://git.kernel.org/pub/scm/network/iproute2/iproute2.git/tree/misc/ss.c#n221 */ +typedef enum { + SOCK_STATE_UNKNOWN = 0, + SOCK_STATE_ESTAB, + SOCK_STATE_SYN_SENT, + SOCK_STATE_SYN_RECV, + SOCK_STATE_FIN_WAIT1, + SOCK_STATE_FIN_WAIT2, + SOCK_STATE_TIME_WAIT, + SOCK_STATE_UNCONN, + SOCK_STATE_CLOSE_WAIT, + SOCK_STATE_LAST_ACK, + SOCK_STATE_LISTEN, + SOCK_STATE_CLOSING, +} SocketState; + +#ifdef __linux__ +bool ParseIPv4SocketInfo(const char *line, + char local_addr[INET_ADDRSTRLEN], uint32_t *local_port, + char remote_addr[INET_ADDRSTRLEN], uint32_t *remote_port, + SocketState *state); + +bool ParseIPv6SocketInfo(const char *line, + char local_addr[INET6_ADDRSTRLEN], uint32_t *local_port, + char remote_addr[INET6_ADDRSTRLEN], uint32_t *remote_port, + SocketState *state); +#else /* __linux__ */ +bool ParseIPv4SocketInfo(const char *line, + char local_addr[INET_ADDRSTRLEN], uint32_t *local_port, + char remote_addr[INET_ADDRSTRLEN], uint32_t *remote_port, + SocketState *state) __attribute__((error ("Only supported on Linux"))); +bool ParseIPv6SocketInfo(const char *line, + char local_addr[INET6_ADDRSTRLEN], uint32_t *local_port, + char remote_addr[INET6_ADDRSTRLEN], uint32_t *remote_port, + SocketState *state) __attribute__((error ("Only supported on Linux"))); +#endif /* __linux__ */ + +#endif /* _CFE_PROC_NET_PARSING_H_ */ diff --git a/libntech b/libntech index 080b24b945..ec0065fbac 160000 --- a/libntech +++ b/libntech @@ -1 +1 @@ -Subproject commit 080b24b945fb8052df12fa9396193e5557f5aadd +Subproject commit ec0065fbacfc8391458fd73c9676ec137e2b6e7b From 8bc4b3263a5a9bdc522bb42cf384e7f2ad081538 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 26 Aug 2020 22:10:39 +0200 Subject: [PATCH 140/333] Use data from /proc/net/{tcp,tcp6,udp,udp6} if available On Linux systems, we can use data from these files instead of running 'netstat' (or 'ss') and parsing its output. Using data from the files under /proc is more efficient and doesn't require any tools. Ticket: CFE-2945 Changelog: cf-monitord now uses /proc/net/* files to get network information if possible (cherry picked from commit 7cb783c0cfc7af16fec0b191bf1b93d0cb41a85c) --- cf-monitord/mon_network.c | 408 ++++++++++++++++++++++++++++++++++---- 1 file changed, 366 insertions(+), 42 deletions(-) diff --git a/cf-monitord/mon_network.c b/cf-monitord/mon_network.c index 03c700493f..e546c0c904 100644 --- a/cf-monitord/mon_network.c +++ b/cf-monitord/mon_network.c @@ -34,17 +34,30 @@ #include #include +#ifdef __linux__ +#include +#endif + /* Globals */ Item *ALL_INCOMING = NULL; Item *MON_UDP4 = NULL, *MON_UDP6 = NULL, *MON_TCP4 = NULL, *MON_TCP6 = NULL; +typedef enum { + cfn_udp4 = 0, + cfn_udp6, + cfn_tcp4, + cfn_tcp6, + cfn_unknown, +} SocketType; + /*******************************************************************/ /* Anomaly */ /*******************************************************************/ typedef struct { + uint32_t port; char *portnr; char *name; enum observables in; @@ -53,26 +66,26 @@ typedef struct static const Sock ECGSOCKS[] = /* extended to map old to new using enum */ { - {"137", "netbiosns", ob_netbiosns_in, ob_netbiosns_out}, - {"138", "netbiosdgm", ob_netbiosdgm_in, ob_netbiosdgm_out}, - {"139", "netbiosssn", ob_netbiosssn_in, ob_netbiosssn_out}, - {"445", "microsoft_ds", ob_microsoft_ds_in, ob_microsoft_ds_out}, - {"5308", "cfengine", ob_cfengine_in, ob_cfengine_out}, - {"2049", "nfsd", ob_nfsd_in, ob_nfsd_out}, - {"25", "smtp", ob_smtp_in, ob_smtp_out}, - {"80", "www", ob_www_in, ob_www_out}, - {"8080", "www-alt", ob_www_alt_in, ob_www_alt_out}, - {"21", "ftp", ob_ftp_in, ob_ftp_out}, - {"22", "ssh", ob_ssh_in, ob_ssh_out}, - {"443", "wwws", ob_wwws_in, ob_wwws_out}, - {"143", "imap", ob_imap_in, ob_imap_out}, - {"993", "imaps", ob_imaps_in, ob_imaps_out}, - {"389", "ldap", ob_ldap_in, ob_ldap_out}, - {"636", "ldaps", ob_ldaps_in, ob_ldaps_out}, - {"27017", "mongo", ob_mongo_in, ob_mongo_out}, - {"3306", "mysql", ob_mysql_in, ob_mysql_out}, - {"5432", "postgresql", ob_postgresql_in, ob_postgresql_out}, - {"631", "ipp", ob_ipp_in, ob_ipp_out}, + {137, "137", "netbiosns", ob_netbiosns_in, ob_netbiosns_out}, + {138, "138", "netbiosdgm", ob_netbiosdgm_in, ob_netbiosdgm_out}, + {139, "139", "netbiosssn", ob_netbiosssn_in, ob_netbiosssn_out}, + {445, "445", "microsoft_ds", ob_microsoft_ds_in, ob_microsoft_ds_out}, + {5308, "5308", "cfengine", ob_cfengine_in, ob_cfengine_out}, + {2049, "2049", "nfsd", ob_nfsd_in, ob_nfsd_out}, + {25, "25", "smtp", ob_smtp_in, ob_smtp_out}, + {80, "80", "www", ob_www_in, ob_www_out}, + {8080, "8080", "www-alt", ob_www_alt_in, ob_www_alt_out}, + {21, "21", "ftp", ob_ftp_in, ob_ftp_out}, + {22, "22", "ssh", ob_ssh_in, ob_ssh_out}, + {433, "443", "wwws", ob_wwws_in, ob_wwws_out}, + {143, "143", "imap", ob_imap_in, ob_imap_out}, + {993, "993", "imaps", ob_imaps_in, ob_imaps_out}, + {389, "389", "ldap", ob_ldap_in, ob_ldap_out}, + {636, "636", "ldaps", ob_ldaps_in, ob_ldaps_out}, + {27017, "27017", "mongo", ob_mongo_in, ob_mongo_out}, + {3306, "3306", "mysql", ob_mysql_in, ob_mysql_out}, + {5432, "5432", "postgresql", ob_postgresql_in, ob_postgresql_out}, + {631, "631", "ipp", ob_ipp_in, ob_ipp_out}, }; #define ATTR (sizeof(ECGSOCKS) / sizeof(ECGSOCKS[0])) @@ -185,6 +198,9 @@ static void SetNetworkEntropyClasses(const char *service, const char *direction, static void SaveNetworkData(Item * const *in, Item * const *out); static void GetNetworkDataFromNetstat(FILE *fp, double *cf_this, Item **in, Item **out); +#ifdef __linux__ +static bool GetNetworkDataFromProcNet(double *cf_this, Item **in, Item **out); +#endif static inline void ResetNetworkData() { @@ -202,32 +218,40 @@ void MonNetworkGatherData(double *cf_this) { ResetNetworkData(); - char comm[PATH_MAX + 4] = {0}; /* path to the binary + " -an" */ - strncpy(comm, VNETSTAT[VSYSTEMHARDCLASS], (sizeof(comm) - 1)); + Item *in[ATTR] = {0}; + Item *out[ATTR] = {0}; - if (!FileCanOpen(comm, "r")) +#ifdef __linux__ + /* On Linux, prefer parsing data from /proc/net with our custom code (more + * efficient), but fall back to netstat if proc parsing fails. */ + if ((access("/proc/net/tcp", R_OK) != 0) || !GetNetworkDataFromProcNet(cf_this, in, out)) +#endif { - Log(LOG_LEVEL_VERBOSE, - "Cannot open '%s', aborting gathering of network data (monitoring)", - comm); - return; - } + char comm[PATH_MAX + 4] = {0}; /* path to the binary + " -an" */ + strncpy(comm, VNETSTAT[VSYSTEMHARDCLASS], (sizeof(comm) - 1)); - strncat(comm, " -an", sizeof(comm) - 1); + if (!FileCanOpen(comm, "r")) + { + Log(LOG_LEVEL_VERBOSE, + "Cannot open '%s', aborting gathering of network data (monitoring)", + comm); + return; + } - FILE *pp; - if ((pp = cf_popen(comm, "r", true)) == NULL) - { - Log(LOG_LEVEL_VERBOSE, - "Opening '%s' failed, aborting gathering of network data (monitoring)", - comm); - return; - } + strncat(comm, " -an", sizeof(comm) - 1); - Item *in[ATTR] = {0}; - Item *out[ATTR] = {0}; - GetNetworkDataFromNetstat(pp, cf_this, in, out); - cf_pclose(pp); + FILE *pp; + if ((pp = cf_popen(comm, "r", true)) == NULL) + { + Log(LOG_LEVEL_VERBOSE, + "Opening '%s' failed, aborting gathering of network data (monitoring)", + comm); + return; + } + + GetNetworkDataFromNetstat(pp, cf_this, in, out); + cf_pclose(pp); + } /* Now save the state for ShowState() the state is not smaller than the last or at least 40 minutes @@ -235,10 +259,310 @@ void MonNetworkGatherData(double *cf_this) SaveNetworkData(in, out); } +#ifdef __linux__ +static inline void SaveSocketInfo(const char *local_addr, + uint32_t local_port, + uint32_t remote_port, + SocketState state, + SocketType type, + const char *socket_info, + double *cf_this, + Item **in, Item **out) +{ + Log(LOG_LEVEL_DEBUG, "Saving socket info '%s:%d:%d [%d, %d]", + local_addr, local_port, remote_port, state, type); + + if (state == SOCK_STATE_LISTEN) + { + char port_str[CF_MAX_PORT_LEN]; + snprintf(port_str, sizeof(port_str), "%d", local_port); + + IdempPrependItem(&ALL_INCOMING, port_str, NULL); + + switch (type) + { + case cfn_tcp4: + IdempPrependItem(&MON_TCP4, port_str, local_addr); + break; + case cfn_tcp6: + IdempPrependItem(&MON_TCP6, port_str, local_addr); + break; + case cfn_udp4: + IdempPrependItem(&MON_UDP4, port_str, local_addr); + break; + case cfn_udp6: + IdempPrependItem(&MON_UDP6, port_str, local_addr); + break; + default: + debug_abort_if_reached(); + break; + } + } + for (size_t i = 0; i < ATTR; i++) + { + if (local_port == ECGSOCKS[i].port) + { + cf_this[ECGSOCKS[i].in]++; + AppendItem(&in[i], socket_info, ""); + + } + + if (remote_port == ECGSOCKS[i].port) + { + cf_this[ECGSOCKS[i].out]++; + AppendItem(&out[i], socket_info, ""); + + } + } +} + +static inline bool GetNetworkDataFromProcNetTCP(char local_addr[INET_ADDRSTRLEN], + char remote_addr[INET_ADDRSTRLEN], + char **buff, size_t *buff_size, Seq *lines, + double *cf_this, Item **in, Item **out) +{ + FILE *fp = fopen("/proc/net/tcp", "r"); + if (fp == NULL) + { + Log(LOG_LEVEL_ERR, "Failed to open /proc/net/tcp for reading"); + return false; + } + /* Read the header */ + ssize_t ret = CfReadLine(buff, buff_size, fp); + if (ret == -1) + { + Log(LOG_LEVEL_ERR, "Failed to read data from /proc/net/tcp"); + fclose(fp); + return false; + } + + /* Read real data */ + ret = CfReadLines(buff, buff_size, fp, lines); + if (ret == -1) + { + Log(LOG_LEVEL_ERR, "Failed to read data from /proc/net/tcp"); + fclose(fp); + return false; + } + Log(LOG_LEVEL_VERBOSE, "Read %zu lines from /proc/net/tcp", ret); + + uint32_t l_port, r_port; + SocketState state; + for (size_t i = 0; i < ret; i++) + { + char *line = SeqAt(lines,i); + if (ParseIPv4SocketInfo(line, local_addr, &l_port, remote_addr, &r_port, &state)) + { + SaveSocketInfo(local_addr, l_port, r_port, + state, cfn_tcp4, + line, cf_this, in, out); + } + } + fclose(fp); + SeqClear(lines); + return true; +} + +static inline bool GetNetworkDataFromProcNetTCP6(char local_addr6[INET6_ADDRSTRLEN], + char remote_addr6[INET6_ADDRSTRLEN], + char **buff, size_t *buff_size, Seq *lines, + double *cf_this, Item **in, Item **out) +{ + FILE *fp = fopen("/proc/net/tcp6", "r"); + if (fp == NULL) + { + /* IPv6 may be completely disabled in kernel so this should be handled gracefully. */ + Log(LOG_LEVEL_VERBOSE, "Failed to read data from /proc/net/tcp6"); + return true; + } + ssize_t ret = CfReadLine(buff, buff_size, fp); + if (ret == -1) + { + Log(LOG_LEVEL_ERR, "Failed to read data from /proc/net/tcp6"); + fclose(fp); + return false; + } + + /* Read real data */ + ret = CfReadLines(buff, buff_size, fp, lines); + if (ret == -1) + { + Log(LOG_LEVEL_ERR, "Failed to read data from /proc/net/tcp6"); + fclose(fp); + return false; + } + Log(LOG_LEVEL_VERBOSE, "Read %zu lines from /proc/net/tcp6", ret); + + uint32_t l_port, r_port; + SocketState state; + for (size_t i = 0; i < ret; i++) + { + char *line = SeqAt(lines,i); + if (ParseIPv6SocketInfo(line, local_addr6, &l_port, remote_addr6, &r_port, &state)) + { + SaveSocketInfo(local_addr6, l_port, r_port, + state, cfn_tcp6, + line, cf_this, in, out); + } + } + fclose(fp); + SeqClear(lines); + return true; +} + +static inline bool GetNetworkDataFromProcNetUDP(char local_addr[INET_ADDRSTRLEN], + char remote_addr[INET_ADDRSTRLEN], + char **buff, size_t *buff_size, Seq *lines, + double *cf_this, Item **in, Item **out) +{ + FILE *fp = fopen("/proc/net/udp", "r"); + if (fp == NULL) + { + Log(LOG_LEVEL_ERR, "Failed to open /proc/net/udp for reading"); + return false; + } + /* Read the header */ + ssize_t ret = CfReadLine(buff, buff_size, fp); + if (ret == -1) + { + Log(LOG_LEVEL_ERR, "Failed to read data from /proc/net/udp"); + fclose(fp); + return false; + } + + /* Read real data */ + ret = CfReadLines(buff, buff_size, fp, lines); + if (ret == -1) + { + Log(LOG_LEVEL_ERR, "Failed to read data from /proc/net/udp"); + fclose(fp); + return false; + } + Log(LOG_LEVEL_VERBOSE, "Read %zu lines from /proc/net/udp", ret); + + uint32_t l_port, r_port; + SocketState state; + for (size_t i = 0; i < ret; i++) + { + char *line = SeqAt(lines,i); + if (ParseIPv4SocketInfo(line, local_addr, &l_port, remote_addr, &r_port, &state)) + { + SaveSocketInfo(local_addr, l_port, r_port, + state, cfn_udp4, + line, cf_this, in, out); + } + } + fclose(fp); + SeqClear(lines); + return true; +} + +static inline bool GetNetworkDataFromProcNetUDP6(char local_addr6[INET6_ADDRSTRLEN], + char remote_addr6[INET6_ADDRSTRLEN], + char **buff, size_t *buff_size, Seq *lines, + double *cf_this, Item **in, Item **out) +{ + FILE *fp = fopen("/proc/net/udp6", "r"); + if (fp == NULL) + { + /* IPv6 may be completely disabled in kernel so this should be handled gracefully. */ + Log(LOG_LEVEL_VERBOSE, "Failed to read data from /proc/net/udp6"); + return true; + } + ssize_t ret = CfReadLine(buff, buff_size, fp); + if (ret == -1) + { + Log(LOG_LEVEL_ERR, "Failed to read data from /proc/net/udp6"); + fclose(fp); + return false; + } + + /* Read real data */ + ret = CfReadLines(buff, buff_size, fp, lines); + if (ret == -1) + { + Log(LOG_LEVEL_ERR, "Failed to read data from /proc/net/udp6"); + fclose(fp); + return false; + } + Log(LOG_LEVEL_VERBOSE, "Read %zu lines from /proc/net/udp6", ret); + + uint32_t l_port, r_port; + SocketState state; + for (size_t i = 0; i < ret; i++) + { + char *line = SeqAt(lines,i); + if (ParseIPv6SocketInfo(line, local_addr6, &l_port, remote_addr6, &r_port, &state)) + { + SaveSocketInfo(local_addr6, l_port, r_port, + state, cfn_udp6, + line, cf_this, in, out); + } + } + fclose(fp); + SeqClear(lines); + return true; +} + +static bool GetNetworkDataFromProcNet(double *cf_this, Item **in, Item **out) +{ + bool result = true; + Seq *lines = SeqNew(64, free); + + size_t buff_size = 256; + char *buff = xmalloc(buff_size); + + { + char local_addr[INET_ADDRSTRLEN]; + char remote_addr[INET_ADDRSTRLEN]; + if (!GetNetworkDataFromProcNetTCP(local_addr, remote_addr, + &buff, &buff_size, lines, + cf_this, in, out)) + { + SeqDestroy(lines); + free(buff); + return false; + } + SeqClear(lines); + if (!GetNetworkDataFromProcNetUDP(local_addr, remote_addr, + &buff, &buff_size, lines, + cf_this, in, out)) + { + SeqDestroy(lines); + free(buff); + return false; + } + SeqClear(lines); + } + + { + char local_addr6[INET6_ADDRSTRLEN]; + char remote_addr6[INET6_ADDRSTRLEN]; + if (!GetNetworkDataFromProcNetTCP6(local_addr6, remote_addr6, + &buff, &buff_size, lines, + cf_this, in, out)) + { + Log(LOG_LEVEL_VERBOSE, "Failed to get IPv6 TCP sockets information"); + } + SeqClear(lines); + if (!GetNetworkDataFromProcNetUDP6(local_addr6, remote_addr6, + &buff, &buff_size, lines, + cf_this, in, out)) + { + Log(LOG_LEVEL_VERBOSE, "Failed to get IPv6 UDP sockets information"); + } + } + + SeqDestroy(lines); + free(buff); + return result; +} +#endif /* __linux__ */ + static void GetNetworkDataFromNetstat(FILE *fp, double *cf_this, Item **in, Item **out) { enum cf_netstat_type { cfn_new, cfn_old } type = cfn_new; - enum cf_packet_type { cfn_udp4, cfn_udp6, cfn_tcp4, cfn_tcp6} packet = cfn_tcp4; + SocketType packet = cfn_tcp4; size_t vbuff_size = CF_BUFSIZE; char *vbuff = xmalloc(vbuff_size); From 25bb7ecd2b84a20ac3ed00d77412ddb8f24ee685 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Tue, 1 Sep 2020 16:04:51 +0200 Subject: [PATCH 141/333] Added valgrind test using Ubuntu 20 container Adapted from the Travis test. Community only for now, enterprise will require some extra work. Conflicts: Related to static-check which is missing on 3.15.x Changelog: None Ticket: ENT-5553 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 58ce156b766c5492ff8b4763c9b096abe543962c) --- configure.ac | 3 +- tests/Makefile.am | 2 +- tests/valgrind-check/Containerfile | 8 ++ tests/valgrind-check/Makefile.am | 25 ++++ tests/valgrind-check/run.sh | 17 +++ tests/valgrind-check/run_checks.sh | 14 ++ tests/valgrind-check/valgrind.sh | 200 +++++++++++++++++++++++++++++ 7 files changed, 267 insertions(+), 2 deletions(-) create mode 100644 tests/valgrind-check/Containerfile create mode 100644 tests/valgrind-check/Makefile.am create mode 100644 tests/valgrind-check/run.sh create mode 100755 tests/valgrind-check/run_checks.sh create mode 100644 tests/valgrind-check/valgrind.sh diff --git a/configure.ac b/configure.ac index c6bd2edb0e..98f207f252 100644 --- a/configure.ac +++ b/configure.ac @@ -1782,7 +1782,8 @@ AC_CONFIG_FILES([Makefile tests/acceptance/Makefile tests/acceptance/25_cf-execd/Makefile tests/unit/Makefile - tests/load/Makefile]) + tests/load/Makefile + tests/valgrind-check/Makefile]) # Run autoconf/configure in libutils, generating necessary makefiles: AC_CONFIG_SUBDIRS([libntech]) diff --git a/tests/Makefile.am b/tests/Makefile.am index b0b72ac2b0..c04aa28805 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -21,4 +21,4 @@ # (COSL) may apply to this file if you as a licensee so wish it. See # included file COSL.txt. # -SUBDIRS = unit load acceptance +SUBDIRS = unit load acceptance valgrind-check diff --git a/tests/valgrind-check/Containerfile b/tests/valgrind-check/Containerfile new file mode 100644 index 0000000000..287d2dee4c --- /dev/null +++ b/tests/valgrind-check/Containerfile @@ -0,0 +1,8 @@ +FROM ubuntu:20.04 AS build +RUN DEBIAN_FRONTEND=noninteractive apt-get update -y +RUN DEBIAN_FRONTEND=noninteractive apt-get install -y libssl-dev libxml2-dev libpam0g-dev liblmdb-dev libacl1-dev libpcre3 libpcre3-dev +RUN DEBIAN_FRONTEND=noninteractive apt-get install -y python git flex bison byacc automake make autoconf libtool valgrind +COPY masterfiles masterfiles +COPY core core +WORKDIR core +CMD bash tests/valgrind-check/run_checks.sh diff --git a/tests/valgrind-check/Makefile.am b/tests/valgrind-check/Makefile.am new file mode 100644 index 0000000000..463634bef3 --- /dev/null +++ b/tests/valgrind-check/Makefile.am @@ -0,0 +1,25 @@ +# +# Copyright 2020 Northern.tech AS +# +# This file is part of CFEngine 3 - written and maintained by Northern.tech AS. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation; version 3. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA +# +# To the extent this program is licensed as part of the Enterprise +# versions of CFEngine, the applicable Commercial Open Source License +# (COSL) may apply to this file if you as a licensee so wish it. See +# included file COSL.txt. +# + +DISTFILES = run_checks.sh run.sh valgrind.sh Makefile.in Makefile.am Containerfile diff --git a/tests/valgrind-check/run.sh b/tests/valgrind-check/run.sh new file mode 100644 index 0000000000..ea2ee561d5 --- /dev/null +++ b/tests/valgrind-check/run.sh @@ -0,0 +1,17 @@ +#!/bin/bash + +set -e +trap "echo FAILURE" ERR + +set -x + +cd ../../../ + +if which podman ; then + CLI="sudo podman --cgroup-manager=cgroupfs" +else + CLI="docker" +fi + +$CLI build --tag ubuntu:mycfecontainer -f ./core/tests/valgrind-check/Containerfile . +$CLI run --rm ubuntu:mycfecontainer diff --git a/tests/valgrind-check/run_checks.sh b/tests/valgrind-check/run_checks.sh new file mode 100755 index 0000000000..e8a186fa2b --- /dev/null +++ b/tests/valgrind-check/run_checks.sh @@ -0,0 +1,14 @@ +#!/bin/bash + +set -x + +function check_with_valgrind() { + bash tests/valgrind-check/valgrind.sh +} + +cd "$(dirname $0)"/../../ + +failure=0 +check_with_valgrind || { echo "FAIL: valgrind check failed"; failure=1; } + +exit $failure diff --git a/tests/valgrind-check/valgrind.sh b/tests/valgrind-check/valgrind.sh new file mode 100644 index 0000000000..bc29c7e236 --- /dev/null +++ b/tests/valgrind-check/valgrind.sh @@ -0,0 +1,200 @@ +#!/bin/bash + +function print_ps { + set +e + echo "CFEngine processes:" + ps aux | grep [c]f- + + echo "Valgrind processes:" + ps aux | grep [v]algrind + set -e +} + +function stop_daemons { + echo "Stopping cfengine daemons" + pkill -f cf-serverd || echo "Did not kill cf-serverd" + pkill -f cf-execd || echo "Did not kill cf-execd" + pkill -f cf-monitord || echo "Did not kill cf-monitord" +} + +function no_errors { + set +e + grep -i "error" $1 + grep -q -i "error" $1 && exit 1 + set -e +} + +function check_daemon_output { + echo "Examining $1:" + no_errors $1 +} + +function check_valgrind_output { + set -e + if [ ! -f "$1" ]; then + echo "$1 does not exists!" + exit 1 + fi + echo "Looking for problems in $1:" + grep -i "ERROR SUMMARY: 0 errors" "$1" + cat $1 | sed -e "/ 0 errors/d" -e "/and suppressed error/d" -e "/a memory error detector/d" > filtered.txt + no_errors filtered.txt + set +e + grep -i "at 0x" filtered.txt + grep -i -q "at 0x" filtered.txt && exit 1 + grep -i "by 0x" filtered.txt + grep -i -q "by 0x" filtered.txt && exit 1 + grep -i "Failed to connect" filtered.txt + grep -i -q "Failed to connect" filtered.txt && exit 1 + set -e +} + +function check_masterfiles_and_inputs { + set -e + echo "Comparing promises.cf from inputs and masterfiles:" + diff -u /var/cfengine/inputs/promises.cf /var/cfengine/masterfiles/promises.cf +} + +set -e +set -x + +# Assume we are in core directory +if [ -f ./configure ] ; then + ./configure -C --enable-debug +else + ./autogen.sh -C --enable-debug +fi +make +make install + +cd ../masterfiles +if [ -f ./configure ] ; then + ./configure -C --enable-debug +else + ./autogen.sh -C --enable-debug +fi +make +make install + +/var/cfengine/bin/cf-agent --version + +VG_OPTS="--leak-check=full --track-origins=yes --error-exitcode=1" +BOOTSTRAP_IP="127.0.0.1" + +valgrind $VG_OPTS /var/cfengine/bin/cf-key 2>&1 | tee cf-key.txt +check_valgrind_output cf-key.txt +valgrind $VG_OPTS /var/cfengine/bin/cf-agent -B $BOOTSTRAP_IP 2>&1 | tee bootstrap.txt +check_valgrind_output bootstrap.txt + +echo "Running cf-check diagnose --validate on all databases:" +valgrind $VG_OPTS /var/cfengine/bin/cf-check diagnose --validate 2>&1 | tee cf_check_validate_all_1.txt +check_valgrind_output cf_check_validate_all_1.txt + +check_masterfiles_and_inputs + +print_ps + +echo "Stopping before relaunch under valgrind" +stop_daemons +sleep 10 +print_ps + +echo "Starting cf-serverd with valgrind in background:" +valgrind $VG_OPTS --log-file=serverd.txt /var/cfengine/bin/cf-serverd --no-fork 2>&1 > serverd_output.txt & +server_pid="$!" +sleep 20 + +echo "Starting cf-execd with valgrind in background:" +valgrind $VG_OPTS --log-file=execd.txt /var/cfengine/bin/cf-execd --no-fork 2>&1 > execd_output.txt & +exec_pid="$!" +sleep 10 + +print_ps + +echo "Running cf-net:" +valgrind $VG_OPTS /var/cfengine/bin/cf-net GET /var/cfengine/masterfiles/promises.cf 2>&1 | tee get.txt +check_valgrind_output get.txt + +echo "Checking promises.cf diff (from cf-net GET):" +diff -u ./promises.cf /var/cfengine/masterfiles/promises.cf + +echo "Running update.cf:" +valgrind $VG_OPTS /var/cfengine/bin/cf-agent -K -f update.cf 2>&1 | tee update.txt +check_valgrind_output update.txt +check_masterfiles_and_inputs +echo "Running update.cf without local copy:" +valgrind $VG_OPTS /var/cfengine/bin/cf-agent -K -f update.cf -D mpf_skip_local_copy_optimization 2>&1 | tee update2.txt +check_valgrind_output update2.txt +check_masterfiles_and_inputs +echo "Running promises.cf:" +valgrind $VG_OPTS /var/cfengine/bin/cf-agent -K -f promises.cf 2>&1 | tee promises.txt +check_valgrind_output promises.txt + +# Dump all databases, use grep to filter the JSON lines +# (optional whitespace then double quote or curly brackets). +# Some of the databases have strings containing "error" +# which check_valgrind_output greps for. +echo "Running cf-check dump:" +valgrind $VG_OPTS /var/cfengine/bin/cf-check dump 2>&1 | grep -E '\s*[{}"]' --invert-match | tee cf_check_dump.txt +check_valgrind_output cf_check_dump.txt + +echo "Running cf-check diagnose on all databases" +valgrind $VG_OPTS /var/cfengine/bin/cf-check diagnose 2>&1 | tee cf_check_diagnose.txt +check_valgrind_output cf_check_diagnose.txt + +echo "Running cf-check diagnose --validate on all databases:" +valgrind $VG_OPTS /var/cfengine/bin/cf-check diagnose --validate 2>&1 | tee cf_check_validate_all_2.txt +check_valgrind_output cf_check_validate_all_2.txt + +echo "Checking that bootstrap ID doesn't change" +/var/cfengine/bin/cf-agent --show-evaluated-vars | grep bootstrap_id > id_a +/var/cfengine/bin/cf-agent -K --show-evaluated-vars | grep bootstrap_id > id_b +cat id_a +diff id_a id_b + +echo "Checking that bootstrap ID has expected length" +[ `cat id_a | awk '{print $2}' | wc -c` -eq 41 ] + +print_ps + +echo "cf-execd outputs:" +tail execd_output.txt +tail execd.txt + +echo "cf-serverd outputs:" +tail serverd_output.txt +tail serverd.txt + +echo "Checking that serverd and execd PIDs are still correct/alive:" +ps -p $exec_pid +ps -p $server_pid + +echo "Killing valgrind cf-execd" +kill $exec_pid +echo "Killing valgrind cf-serverd" +kill $server_pid + +wait $exec_pid +wait $server_pid + +echo "Output from cf-execd in valgrind:" +cat execd.txt +check_valgrind_output execd.txt +check_daemon_output execd_output.txt + +echo "Output from cf-serverd in valgrind:" +cat serverd.txt +check_valgrind_output serverd.txt +check_daemon_output serverd_output.txt + +stop_daemons + +echo "Done killing" +sleep 10 +print_ps + +echo "Check that bootstrap was successful" +grep "This host assumes the role of policy server" bootstrap.txt +grep "completed successfully!" bootstrap.txt + +echo "valgrind_health_check successful! (valgrind.sh)" From 686449629c15fc6aa1bf10e51e6e2fbb4d2af0f5 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Fri, 3 Apr 2020 14:58:32 -0500 Subject: [PATCH 142/333] Changed bootstrap to loopback to a warning instead of exit The exit blocked effective use in a development environment without a stable non-loopback interface, such as a phone (android/termux). Ticket: CFE-3304 Changelog: title (cherry picked from commit 5b00565454195b0b79f420c5abeca764b1476ff0) --- cf-agent/cf-agent.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/cf-agent/cf-agent.c b/cf-agent/cf-agent.c index 7f7345f5d8..6f7d8490bc 100644 --- a/cf-agent/cf-agent.c +++ b/cf-agent/cf-agent.c @@ -371,8 +371,7 @@ static void ConfigureBootstrap(GenericAgentConfig *config, const char *argument) if(IsLoopbackAddress(argument)) { - Log(LOG_LEVEL_ERR, "Cannot bootstrap to a loopback address"); - DoCleanupAndExit(EXIT_FAILURE); + Log(LOG_LEVEL_WARNING, "Bootstrapping to loopback interface (localhost), other hosts will not be able to bootstrap to this server"); } // temporary assure that network functions are working From 8cdf618192b2671b07da9862bd9e16b912ff3f21 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Tue, 1 Sep 2020 20:33:09 +0200 Subject: [PATCH 143/333] Expand all variables in data/list references Variables in data/list references should be expanded, not only if they are in place of the data/list name but also in field name or index. Ticket: CFE-3299 Changelog: Expand variables in data/list references (cherry picked from commit ef2dafa7774baabeee209d36ed40d14c783cdee4) --- libpromises/expand.c | 36 +++--- .../pass_variable_container_index.cf | 110 ++++++++++++++++++ .../pass_variable_container_index.cf.expected | 24 ++++ 3 files changed, 157 insertions(+), 13 deletions(-) create mode 100644 tests/acceptance/01_vars/04_containers/pass_variable_container_index.cf create mode 100644 tests/acceptance/01_vars/04_containers/pass_variable_container_index.cf.expected diff --git a/libpromises/expand.c b/libpromises/expand.c index e1eaf6f3f4..0827f76dff 100644 --- a/libpromises/expand.c +++ b/libpromises/expand.c @@ -348,13 +348,13 @@ Rval ExpandPrivateRval(EvalContext *ctx, /** * Detects a variable expansion inside of a data/list reference, for example * "@(${container_name})" or "@(prefix${container_name})" or - * "@(nspace:${container_name})". + * "@(nspace:${container_name})" or "@(container_name[${field}])". * * @note This function doesn't have to be bullet-proof, it only needs to * properly detect valid cases. The rest is left to the parser and code * expanding variables. */ -static inline bool ContainsVariableDataListReference(const char *str) +static inline bool VariableDataOrListReference(const char *str) { assert(str != NULL); @@ -372,14 +372,16 @@ static inline bool ContainsVariableDataListReference(const char *str) return false; } - /* Check if, after '@(', there are only characters allowed in data/list - * names or ':' to separate namespace from the name followed by "$(" or "${" - * with a matching close bracket somewhere. */ + /* Check if, after '@(', there are only + * - characters allowed in data/list names or + * - ':' to separate namespace from the name or + * - '.' to separate bundle and variable name or, + * - '[' for data/list field/index specification, + * followed by "$(" or "${" with a matching close bracket somewhere. */ for (size_t i = 2; i < len; i++) { - if (!(isalnum((int) str[i]) || - (str[i] == '_') || (str[i] == ':') || - (str[i] == '$'))) + if (!(isalnum((int) str[i]) || (str[i] == '_') || + (str[i] == ':') || (str[i] == '$') || (str[i] == '.') || (str[i] == '['))) { return false; } @@ -405,12 +407,14 @@ static Rval ExpandListEntry(EvalContext *ctx, const char *ns, const char *scope, int expandnaked, Rval entry) { + Rval expanded_data_list = {0}; /* If rval is something like '@($(container_name).field)', we need to expand * the nested variable first. */ if (entry.type == RVAL_TYPE_SCALAR && - ContainsVariableDataListReference(entry.item)) + VariableDataOrListReference(entry.item)) { entry = ExpandPrivateRval(ctx, ns, scope, entry.item, entry.type); + expanded_data_list = entry; } if (entry.type == RVAL_TYPE_SCALAR && @@ -439,18 +443,24 @@ static Rval ExpandListEntry(EvalContext *ctx, if (value_type != CF_DATA_TYPE_NONE) /* variable found? */ { - return ExpandPrivateRval(ctx, ns, scope, value, - DataTypeToRvalType(value_type)); + Rval ret = ExpandPrivateRval(ctx, ns, scope, value, + DataTypeToRvalType(value_type)); + RvalDestroy(expanded_data_list); + return ret; } } } else { - return RvalNew(entry.item, RVAL_TYPE_SCALAR); + Rval ret = RvalNew(entry.item, RVAL_TYPE_SCALAR); + RvalDestroy(expanded_data_list); + return ret; } } - return ExpandPrivateRval(ctx, ns, scope, entry.item, entry.type); + Rval ret = ExpandPrivateRval(ctx, ns, scope, entry.item, entry.type); + RvalDestroy(expanded_data_list); + return ret; } Rlist *ExpandList(EvalContext *ctx, diff --git a/tests/acceptance/01_vars/04_containers/pass_variable_container_index.cf b/tests/acceptance/01_vars/04_containers/pass_variable_container_index.cf new file mode 100644 index 0000000000..cd9f642688 --- /dev/null +++ b/tests/acceptance/01_vars/04_containers/pass_variable_container_index.cf @@ -0,0 +1,110 @@ +####################################################### +# +# Test datacontainers can be passed with a variable name. +# +####################################################### + +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +####################################################### + +bundle agent init +{ + meta: + "tags" slist => { "find" }; +} + +bundle agent test + +{ + meta: + "description" -> { "CFE-3299" } + string => "Test that lists and data containers with variable index can be passed."; + + vars: + "data" + data => parsejson('{ "key1" : { "list" : [ "a", "b", "c" ], "scalar" : "value1" }, + "key2" : { "list" : [ "x", "y", "z" ], "scalar" : "value2" } }') ; + + "array[key1][list]" slist => { "a", "b", "c" } ; + "array[key1][scalar]" string => "value1" ; + "array[key2][list]" slist => { "x", "y", "z" } ; + "array[key2][scalar]" string => "value2" ; + + "datakey" slist => getindices("data") ; + "arraykey" slist => getindices("array") ; + + methods: + "test_data" + usebundle => launcher_data("test.data[${datakey}][list]") ; + + "test_array" + usebundle => launcher_array("test.array[${arraykey}][list]") ; + + "test_executor_data" + usebundle => executor("@{test.data[${datakey}][list]}") ; + + "test_executor_array" + usebundle => executor("@{test.data[${arraykey}][list]}") ; +} + +bundle agent check +{ + methods: + "" usebundle => dcs_check_diff( $(G.testfile), + "$(this.promise_filename).expected", + $(this.promise_filename)); +} + +bundle agent launcher_data(list_name) +{ + vars: + any:: + "expanded_list" slist => getvalues("${list_name}") ; + "callers" string => join(":", callstack_promisers()) ; + + methods: + any:: + # If we don't add something variable in the following promise, and the promise is kept, + # CFEngine will think that the promise is already kept and refuse to keep it more than + # once. Inserting a variable handle will make CFEngine understand that it's not the + # same promise we are asking it to keep over and over again, and execute it again every + # time we pass it a different list. + "run_list" + usebundle => executor(@{launcher_data.expanded_list}), + handle => "run_list_${list_name}" ; +} + +bundle agent launcher_array(list_name) +{ + vars: + any:: + "expanded_list" slist => { "@{${list_name}}"} ; + "callers" string => join(":", callstack_promisers()) ; + + methods: + any:: + # If we don't add something variable in the following promise, and the promise is kept, + # CFEngine will think that the promise is already kept and refuse to keep it more than + # once. Inserting a variable handle will make CFEngine understand that it's not the + # same promise we are asking it to keep over and over again, and execute it again every + # time we pass it a different list. + "run_list" + usebundle => executor(@{launcher_array.expanded_list}), + handle => "run_list_${list_name}" ; +} + +bundle agent executor(list) +{ + vars: + "callers" string => join(":", callstack_promisers()) ; + + reports: + "[${callers}:${this.bundle}] ${list}" + report_to_file => "$(G.testfile)"; +} diff --git a/tests/acceptance/01_vars/04_containers/pass_variable_container_index.cf.expected b/tests/acceptance/01_vars/04_containers/pass_variable_container_index.cf.expected new file mode 100644 index 0000000000..153d590163 --- /dev/null +++ b/tests/acceptance/01_vars/04_containers/pass_variable_container_index.cf.expected @@ -0,0 +1,24 @@ +[any:any:test_data:run_list:executor] a +[any:any:test_data:run_list:executor] b +[any:any:test_data:run_list:executor] c +[any:any:test_data:run_list:executor] x +[any:any:test_data:run_list:executor] y +[any:any:test_data:run_list:executor] z +[any:any:test_array:run_list:executor] x +[any:any:test_array:run_list:executor] y +[any:any:test_array:run_list:executor] z +[any:any:test_array:run_list:executor] a +[any:any:test_array:run_list:executor] b +[any:any:test_array:run_list:executor] c +[any:any:test_executor_data:executor] a +[any:any:test_executor_data:executor] b +[any:any:test_executor_data:executor] c +[any:any:test_executor_data:executor] x +[any:any:test_executor_data:executor] y +[any:any:test_executor_data:executor] z +[any:any:test_executor_array:executor] x +[any:any:test_executor_array:executor] y +[any:any:test_executor_array:executor] z +[any:any:test_executor_array:executor] a +[any:any:test_executor_array:executor] b +[any:any:test_executor_array:executor] c From 21d0cd640762994c907eaf379e72f682534754d9 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 2 Sep 2020 12:39:13 +0200 Subject: [PATCH 144/333] Remove an outdated comment from the pass_variable_container.cf test Things now work as expected. (cherry picked from commit d4a9994231ca69500b67708026d1471f4b0f87fc) --- .../01_vars/04_containers/pass_variable_container.cf | 4 ---- 1 file changed, 4 deletions(-) diff --git a/tests/acceptance/01_vars/04_containers/pass_variable_container.cf b/tests/acceptance/01_vars/04_containers/pass_variable_container.cf index 0490b27417..fd55182721 100644 --- a/tests/acceptance/01_vars/04_containers/pass_variable_container.cf +++ b/tests/acceptance/01_vars/04_containers/pass_variable_container.cf @@ -47,12 +47,8 @@ bundle agent test "bundle_name_suffix" string => "it"; methods: - # These work. "" usebundle => report_data("Pass by name:", @(init.d)); "" usebundle => report_data("Pass by namespace:name:", @(default:init.d)); - - # These do not work. (Note: Quotes are required for this, ideally they - # wouldn't be but thats a seperate issue.) "" usebundle => report_data("Pass by variable name:", "@($(bundle_name).d)"); "" usebundle => report_data("Pass by variable name including namespace:", "@($(found).d)"); "" usebundle => report_data("Pass by variable name with prefix:", "@(in$(bundle_name_suffix).d)"); From 3f7d52bfdf1469781c3b09f32d2a48c8a40d52d9 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Mon, 7 Sep 2020 11:27:37 +0200 Subject: [PATCH 145/333] Allow spaces inside square brackets in class expressions Ticket: CFE-3320 Changelog: Spaces inside square brackets (slist/data index) are now allowed in class expressions (cherry picked from commit 53e21fdc713783dcc1690e5cbfbf54e9e2fb388e) --- libpromises/string_expressions.c | 29 ++++++++- .../no_error_when_key_contains_space.cf | 61 +++++++++++++++++++ 2 files changed, 87 insertions(+), 3 deletions(-) create mode 100644 tests/acceptance/00_basics/01_compiler/no_error_when_key_contains_space.cf diff --git a/libpromises/string_expressions.c b/libpromises/string_expressions.c index d3933a100b..540f43a518 100644 --- a/libpromises/string_expressions.c +++ b/libpromises/string_expressions.c @@ -135,8 +135,10 @@ static StringParseResult ParseVarRef(const char *expr, int start, int end) /* */ -static bool ValidTokenCharacter(char c) +static inline bool ValidTokenCharacter(char c, bool *inside_index) { + assert(inside_index != NULL); + if (c >= 'a' && c <= 'z') { return true; @@ -152,7 +154,27 @@ static bool ValidTokenCharacter(char c) return true; } - if (c == '_' || c == '[' || c == ']' || c == ':') + if (c == '_' || c == ':') + { + return true; + } + + if (c == '[') + { + *inside_index = true; + return true; + } + + if (c == ']') + { + if (*inside_index) + { + *inside_index = false; + } + return true; + } + + if ((c == ' ') && *inside_index) { return true; } @@ -164,7 +186,8 @@ static StringParseResult ParseToken(const char *expr, int start, int end) { int endlit = start; - while (endlit < end && ValidTokenCharacter(expr[endlit])) + bool inside_index = false; + while (endlit < end && ValidTokenCharacter(expr[endlit], &inside_index)) { endlit++; } diff --git a/tests/acceptance/00_basics/01_compiler/no_error_when_key_contains_space.cf b/tests/acceptance/00_basics/01_compiler/no_error_when_key_contains_space.cf new file mode 100644 index 0000000000..8cba164686 --- /dev/null +++ b/tests/acceptance/00_basics/01_compiler/no_error_when_key_contains_space.cf @@ -0,0 +1,61 @@ +# This test should be renamed from .x.cf to .cf when the bug is fixed. + +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +####################################################### + +bundle agent test +{ + meta: + "description" -> { "CFE-3320" } + string => "Test that data container keys can contain spaces"; + + vars: + "d" data => '{ + "thing s": [ + { + "Title": "ExpectedPick", + "classexpr": "$(sys.class)" + } + ] + }'; + + "selected" + string => "$(d[thing s][0][Title])", + if => "$(d[thing s][0][classexpr])"; + # Note: wrapping with concat or classify prevents the error + # if => concat("$(d[thing s][$(di)][classexpr])"); + # if => classify("$(d[thing s][$(di)][classexpr])"); + + "sanity_check" + string => "You are sane", + if => "$(sys.class)"; + + reports: + EXTRA|DEBUG:: + "See iteration/expansion: $(d[thing s][ExpectedPick][Title]) has classexpr $(d[thing s][ExpectedPick][classexpr])"; + "See sanity_check: $(sanity_check) because $(sys.class) is a defined class"; + + "Picked: $(d[thing s][ExpectedPick][Title]) has classexpr $(d[thing s][ExpectedPick][classexpr])" + if => "$(d[thing s][ExpectedPick][classexpr])"; +} + +####################################################### + +bundle agent check +{ + vars: + "expected_selection" string => "ExpectedPick"; + + methods: + "Pass/Fail" + usebundle => dcs_check_strcmp($(expected_selection), $(test.selected), + $(this.promise_filename), "no"); + +} + From 67d6c8fa8f3c23196fb16ebab257874b24ce802f Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 23 Sep 2020 14:11:10 +0200 Subject: [PATCH 146/333] Add SELinux rules for accessing /proc/xen These rules should not be needed because only processes forked from the agent should access the folder, but pre-evaluation causes all our processes accessing the folder. Ticket: CFE-3240 Changelog: AVCs are no longer produced for CFEngine processes accessing /proc/net (cherry picked from commit a4f3aae826bc584eec799fc8804dd5988b463c97) --- misc/selinux/cfengine-enterprise.te | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index cb53019a07..3af8f738dd 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -25,6 +25,7 @@ require { type cfengine_execd_exec_t; type ping_exec_t; type proc_net_t; + type proc_xen_t; type cfengine_serverd_exec_t; type smtp_port_t; type rpm_exec_t; @@ -163,6 +164,9 @@ allow cfengine_execd_t cfengine_var_lib_t:file execute; # allow cf-execd to execute cf-promises allow cfengine_execd_t cfengine_var_lib_t:file execute_no_trans; +# TODO: this should not be needed +allow cfengine_execd_t proc_xen_t:dir search; + allow cfengine_execd_t cfengine_log_t:file { read unlink write }; allow cfengine_execd_t cfengine_log_t:lnk_file { create getattr read unlink }; allow cfengine_execd_t cfengine_monitord_exec_t:file getattr; @@ -227,6 +231,8 @@ allow cfengine_monitord_t useradd_exec_t:file getattr; allow cfengine_monitord_t var_t:dir read; allow cfengine_monitord_t semanage_exec_t:file getattr; +# TODO: this should not be needed +allow cfengine_monitord_t proc_xen_t:dir search; #============= cfengine_serverd_t ============== # allow cf-serverd to run cf-agent and make sure the forked process run in the @@ -245,6 +251,9 @@ allow cfengine_serverd_t cfengine_var_lib_t:file execute_no_trans; # allow cf-serverd to connect in case of call-collect allow cfengine_serverd_t unreserved_port_t:tcp_socket name_connect; +# TODO: this should not be needed +allow cfengine_serverd_t proc_xen_t:dir search; + allow cfengine_serverd_t cfengine_execd_exec_t:file getattr; allow cfengine_serverd_t cfengine_monitord_exec_t:file getattr; From a483d7cf2c95c186df4f2677aa9ff4e670175dd9 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 17 Sep 2020 17:03:07 +0200 Subject: [PATCH 147/333] Add SELinux rules for cf-hub Ticket: ENT-6103 Changelog: None (cherry picked from commit e0ca93c7c2d751bbee183187ff47806676bdef42) --- misc/selinux/cfengine-enterprise.fc | 1 + misc/selinux/cfengine-enterprise.te | 95 +++++++++++++++++++++++++++++ 2 files changed, 96 insertions(+) diff --git a/misc/selinux/cfengine-enterprise.fc b/misc/selinux/cfengine-enterprise.fc index d79f97bbb5..a9d84b323c 100644 --- a/misc/selinux/cfengine-enterprise.fc +++ b/misc/selinux/cfengine-enterprise.fc @@ -2,3 +2,4 @@ /var/cfengine/bin/cf-serverd -- gen_context(system_u:object_r:cfengine_serverd_exec_t,s0) /var/cfengine/bin/cf-monitord -- gen_context(system_u:object_r:cfengine_monitord_exec_t,s0) /var/cfengine/bin/cf-agent -- gen_context(system_u:object_r:cfengine_agent_exec_t,s0) +/var/cfengine/bin/cf-hub -- gen_context(system_u:object_r:cfengine_hub_exec_t,s0) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 3af8f738dd..8b6c89f293 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -171,6 +171,7 @@ allow cfengine_execd_t cfengine_log_t:file { read unlink write }; allow cfengine_execd_t cfengine_log_t:lnk_file { create getattr read unlink }; allow cfengine_execd_t cfengine_monitord_exec_t:file getattr; allow cfengine_execd_t cfengine_serverd_exec_t:file getattr; +allow cfengine_execd_t cfengine_hub_exec_t:file getattr; allow cfengine_execd_t self:capability sys_ptrace; @@ -213,6 +214,7 @@ allow cfengine_monitord_t cfengine_var_lib_t:file execute_no_trans; allow cfengine_monitord_t cfengine_execd_exec_t:file getattr; allow cfengine_monitord_t cfengine_serverd_exec_t:file getattr; allow cfengine_monitord_t cfengine_agent_exec_t:file getattr; +allow cfengine_monitord_t cfengine_hub_exec_t:file getattr; allow cfengine_monitord_t self:capability { dac_override dac_read_search sys_ptrace }; @@ -256,6 +258,7 @@ allow cfengine_serverd_t proc_xen_t:dir search; allow cfengine_serverd_t cfengine_execd_exec_t:file getattr; allow cfengine_serverd_t cfengine_monitord_exec_t:file getattr; +allow cfengine_serverd_t cfengine_hub_exec_t:file getattr; allow cfengine_serverd_t crontab_exec_t:file getattr; allow cfengine_serverd_t dmidecode_exec_t:file getattr; @@ -278,3 +281,95 @@ allow cfengine_serverd_t user_cron_spool_t:dir getattr; allow cfengine_serverd_t useradd_exec_t:file getattr; allow cfengine_serverd_t var_t:dir read; allow cfengine_serverd_t semanage_exec_t:file getattr; + + +#============= cfengine_hub_t ============== +type cfengine_hub_t; +typeattribute cfengine_hub_t domain; +role system_r types cfengine_hub_t; + +# /var/cfengine/bin/cf-hub has the 'cfengine_hub_exec_t' context which is an +# entrypoint for the 'cfengine_hub_t' domain +type cfengine_hub_exec_t; +typeattribute cfengine_hub_exec_t entry_type; +typeattribute cfengine_hub_exec_t exec_type; +typeattribute cfengine_hub_exec_t file_type, non_security_file_type, non_auth_file_type; +role object_r types cfengine_hub_exec_t; + +type_transition init_t cfengine_hub_exec_t:process cfengine_hub_t; +allow init_t cfengine_hub_t:process transition; +allow init_t cfengine_hub_exec_t:file { execute open read }; + +allow cfengine_hub_t cfengine_hub_exec_t:file entrypoint; +allow cfengine_hub_t cfengine_hub_exec_t:file { ioctl read getattr lock map execute open }; + +# allow cf-hub to use/execute libpromises.so +allow cfengine_hub_t cfengine_var_lib_t:file map; +allow cfengine_hub_t cfengine_var_lib_t:file execute; +allow cfengine_hub_t cfengine_var_lib_t:file { getattr open read }; + +allow cfengine_hub_t cfengine_agent_exec_t:file getattr; +allow cfengine_hub_t cfengine_execd_exec_t:file getattr; +allow cfengine_hub_t cfengine_monitord_exec_t:file getattr; +allow cfengine_hub_t cfengine_serverd_exec_t:file getattr; + +allow cfengine_hub_t unreserved_port_t:tcp_socket name_connect; + +allow cfengine_hub_t cfengine_log_t:dir getattr; +allow cfengine_hub_t cfengine_var_lib_t:dir { add_name getattr open read search write remove_name }; +allow cfengine_hub_t cfengine_var_lib_t:file { create ioctl lock write }; +allow cfengine_hub_t cfengine_var_lib_t:lnk_file { getattr read }; +allow cfengine_hub_t cfengine_var_lib_t:sock_file { create unlink }; + +allow cfengine_hub_t bin_t:file map; +allow cfengine_hub_t bin_t:file { execute execute_no_trans }; +allow cfengine_hub_t cert_t:dir search; +allow cfengine_hub_t cert_t:file { getattr open read }; +allow cfengine_hub_t crontab_exec_t:file getattr; +allow cfengine_hub_t devlog_t:lnk_file read; +allow cfengine_hub_t devlog_t:sock_file write; +allow cfengine_hub_t dmidecode_exec_t:file getattr; +allow cfengine_hub_t fs_t:filesystem getattr; +allow cfengine_hub_t groupadd_exec_t:file getattr; +allow cfengine_hub_t hostname_exec_t:file getattr; +allow cfengine_hub_t init_exec_t:file getattr; +allow cfengine_hub_t init_t:dir { getattr open read search }; +allow cfengine_hub_t init_t:file { getattr open read }; +allow cfengine_hub_t init_t:unix_stream_socket ioctl; +allow cfengine_hub_t init_var_run_t:dir search; +allow cfengine_hub_t journalctl_exec_t:file getattr; +allow cfengine_hub_t kernel_t:unix_dgram_socket sendto; +allow cfengine_hub_t net_conf_t:file { getattr open read }; +allow cfengine_hub_t passwd_file_t:file { getattr open read }; +allow cfengine_hub_t ping_exec_t:file getattr; +allow cfengine_hub_t proc_net_t:file { getattr open read }; +allow cfengine_hub_t proc_t:dir read; +allow cfengine_hub_t rpm_exec_t:file getattr; +allow cfengine_hub_t self:capability dac_override; +allow cfengine_hub_t self:tcp_socket { connect create getopt setopt }; +allow cfengine_hub_t self:udp_socket { connect create getattr ioctl setopt }; +allow cfengine_hub_t self:netlink_route_socket { create getopt setopt bind getattr }; +allow cfengine_hub_t self:unix_dgram_socket { create connect }; +allow cfengine_hub_t semanage_exec_t:file getattr; +allow cfengine_hub_t shadow_t:file getattr; +allow cfengine_hub_t sssd_public_t:dir search; +allow cfengine_hub_t sssd_public_t:file map; +allow cfengine_hub_t sssd_public_t:file { getattr open read }; +allow cfengine_hub_t sssd_t:unix_stream_socket connectto; +allow cfengine_hub_t sssd_var_lib_t:dir search; +allow cfengine_hub_t sssd_var_lib_t:sock_file write; +allow cfengine_hub_t sysctl_net_t:dir search; +allow cfengine_hub_t sysfs_t:dir read; +allow cfengine_hub_t sysfs_t:file { getattr open read }; +allow cfengine_hub_t syslogd_var_run_t:dir search; +allow cfengine_hub_t systemd_systemctl_exec_t:file getattr; +allow cfengine_hub_t tmp_t:sock_file write; +allow cfengine_hub_t user_cron_spool_t:dir getattr; +allow cfengine_hub_t useradd_exec_t:file getattr; +allow cfengine_hub_t var_t:dir read; + +# TODO: these should not be needed +allow cfengine_hub_t ifconfig_exec_t:file { execute execute_no_trans open read getattr map }; +allow cfengine_hub_t shell_exec_t:file map; +allow cfengine_hub_t shell_exec_t:file { execute execute_no_trans }; +allow cfengine_hub_t proc_xen_t:dir search; From a5072b51b3ad72c9f33f9a6ef12c537b5189a525 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Mon, 21 Sep 2020 15:10:54 +0200 Subject: [PATCH 148/333] Add SELinux rules for our PostgreSQL files and processes Ticket: ENT-6103 Changelog: None (cherry picked from commit 84fd038cb98e9fcd79c45249b697d8244b68c566) --- misc/selinux/cfengine-enterprise.fc | 1 + misc/selinux/cfengine-enterprise.te | 62 +++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+) diff --git a/misc/selinux/cfengine-enterprise.fc b/misc/selinux/cfengine-enterprise.fc index a9d84b323c..c0296eb638 100644 --- a/misc/selinux/cfengine-enterprise.fc +++ b/misc/selinux/cfengine-enterprise.fc @@ -3,3 +3,4 @@ /var/cfengine/bin/cf-monitord -- gen_context(system_u:object_r:cfengine_monitord_exec_t,s0) /var/cfengine/bin/cf-agent -- gen_context(system_u:object_r:cfengine_agent_exec_t,s0) /var/cfengine/bin/cf-hub -- gen_context(system_u:object_r:cfengine_hub_exec_t,s0) +/var/cfengine/bin/pg.* -- gen_context(system_u:object_r:cfengine_postgres_exec_t,s0) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 8b6c89f293..8cdfc85980 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -313,6 +313,7 @@ allow cfengine_hub_t cfengine_execd_exec_t:file getattr; allow cfengine_hub_t cfengine_monitord_exec_t:file getattr; allow cfengine_hub_t cfengine_serverd_exec_t:file getattr; +allow cfengine_hub_t cfengine_postgres_t:unix_stream_socket connectto; allow cfengine_hub_t unreserved_port_t:tcp_socket name_connect; allow cfengine_hub_t cfengine_log_t:dir getattr; @@ -373,3 +374,64 @@ allow cfengine_hub_t ifconfig_exec_t:file { execute execute_no_trans open read g allow cfengine_hub_t shell_exec_t:file map; allow cfengine_hub_t shell_exec_t:file { execute execute_no_trans }; allow cfengine_hub_t proc_xen_t:dir search; + + +#============= cfengine_postgres_t ============== +type cfengine_postgres_t; +typeattribute cfengine_postgres_t domain; +role system_r types cfengine_postgres_t; + +# /var/cfengine/bin/cf-postgres has the 'cfengine_postgres_exec_t' context which is an +# entrypoint for the 'cfengine_postgres_t' domain +type cfengine_postgres_exec_t; +typeattribute cfengine_postgres_exec_t entry_type; +typeattribute cfengine_postgres_exec_t exec_type; +typeattribute cfengine_postgres_exec_t file_type, non_security_file_type, non_auth_file_type; +role object_r types cfengine_postgres_exec_t; + +type_transition init_t cfengine_postgres_exec_t:process cfengine_postgres_t; +allow init_t cfengine_postgres_t:process transition; +allow init_t cfengine_postgres_exec_t:file { execute open read }; + +allow cfengine_postgres_t cfengine_postgres_exec_t:file entrypoint; +allow cfengine_postgres_t cfengine_postgres_exec_t:file { ioctl read getattr lock map execute open }; + +# TODO: Why are 'map', 'execute' and 'execute_no_trans' needed for postgres? +allow cfengine_postgres_t cfengine_var_lib_t:file map; +allow cfengine_postgres_t cfengine_var_lib_t:file { create execute execute_no_trans getattr link open read rename unlink write }; + +allow cfengine_postgres_t cfengine_var_lib_t:dir { add_name getattr open read remove_name search write }; + +allow cfengine_postgres_t postgresql_port_t:tcp_socket name_bind; + +allow cfengine_postgres_t hugetlbfs_t:file map; +allow cfengine_postgres_t hugetlbfs_t:file { read write }; +allow cfengine_postgres_t init_t:unix_stream_socket { getattr ioctl }; +allow cfengine_postgres_t net_conf_t:file { getattr open read }; +allow cfengine_postgres_t node_t:tcp_socket node_bind; +allow cfengine_postgres_t node_t:udp_socket node_bind; +allow cfengine_postgres_t proc_t:file { getattr open read }; +allow cfengine_postgres_t self:netlink_route_socket { bind create getattr nlmsg_read }; +allow cfengine_postgres_t self:tcp_socket { bind create listen setopt }; +allow cfengine_postgres_t self:udp_socket { bind connect create getattr getopt }; +allow cfengine_postgres_t sssd_public_t:dir search; +allow cfengine_postgres_t sssd_public_t:file map; +allow cfengine_postgres_t sssd_public_t:file { getattr open read }; +allow cfengine_postgres_t sssd_var_lib_t:sock_file write; +allow cfengine_postgres_t sssd_var_lib_t:dir search; +allow cfengine_postgres_t sssd_t:unix_stream_socket connectto; +allow cfengine_postgres_t tmp_t:dir { add_name write remove_name }; +allow cfengine_postgres_t tmp_t:file { create write unlink }; +allow cfengine_postgres_t tmp_t:sock_file { create setattr unlink write }; +allow cfengine_postgres_t tmpfs_t:dir { add_name write remove_name }; +allow cfengine_postgres_t tmpfs_t:file { create open read write map unlink }; +allow cfengine_postgres_t tmpfs_t:filesystem getattr; +allow cfengine_postgres_t var_log_t:file { append open }; + +# Needed for systemd to be able to check PostgreSQL's PID file +allow init_t cfengine_var_lib_t:dir read; +allow init_t cfengine_var_lib_t:file { getattr open read }; + +# TODO: these should not be needed +allow cfengine_postgres_t shell_exec_t:file map; +allow cfengine_postgres_t shell_exec_t:file { execute execute_no_trans }; From 8a34e0b6d5395184b509d32d9b1216a16111096b Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Mon, 21 Sep 2020 15:10:54 +0200 Subject: [PATCH 149/333] Add SELinux rules for our web server files and processes Ticket: ENT-6103 Changelog: None (cherry picked from commit bd2c0c3bee79c68813f4a8f72e64e05cb3df4221) --- misc/selinux/cfengine-enterprise.fc | 2 + misc/selinux/cfengine-enterprise.te | 70 +++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+) diff --git a/misc/selinux/cfengine-enterprise.fc b/misc/selinux/cfengine-enterprise.fc index c0296eb638..15a7a73cfd 100644 --- a/misc/selinux/cfengine-enterprise.fc +++ b/misc/selinux/cfengine-enterprise.fc @@ -4,3 +4,5 @@ /var/cfengine/bin/cf-agent -- gen_context(system_u:object_r:cfengine_agent_exec_t,s0) /var/cfengine/bin/cf-hub -- gen_context(system_u:object_r:cfengine_hub_exec_t,s0) /var/cfengine/bin/pg.* -- gen_context(system_u:object_r:cfengine_postgres_exec_t,s0) +/var/cfengine/httpd/bin/.* -- gen_context(system_u:object_r:cfengine_httpd_exec_t,s0) +/var/cfengine/httpd/php/bin/.* -- gen_context(system_u:object_r:cfengine_httpd_exec_t,s0) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 8cdfc85980..c2e95a01e3 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -435,3 +435,73 @@ allow init_t cfengine_var_lib_t:file { getattr open read }; # TODO: these should not be needed allow cfengine_postgres_t shell_exec_t:file map; allow cfengine_postgres_t shell_exec_t:file { execute execute_no_trans }; + + +#============= cfengine_httpd_t ============== +type cfengine_httpd_t; +typeattribute cfengine_httpd_t domain; +role system_r types cfengine_httpd_t; + +# /var/cfengine/bin/cf-httpd has the 'cfengine_httpd_exec_t' context which is an +# entrypoint for the 'cfengine_httpd_t' domain +type cfengine_httpd_exec_t; +typeattribute cfengine_httpd_exec_t entry_type; +typeattribute cfengine_httpd_exec_t exec_type; +typeattribute cfengine_httpd_exec_t file_type, non_security_file_type, non_auth_file_type; +role object_r types cfengine_httpd_exec_t; + +type_transition init_t cfengine_httpd_exec_t:process cfengine_httpd_t; +allow init_t cfengine_httpd_t:process transition; +allow init_t cfengine_httpd_exec_t:file { execute getattr open read }; + +allow cfengine_httpd_t cfengine_httpd_exec_t:file entrypoint; +allow cfengine_httpd_t cfengine_httpd_exec_t:file { ioctl read getattr lock map execute open }; + +allow cfengine_httpd_t cert_t:dir search; +allow cfengine_httpd_t cert_t:file { getattr open read }; +allow cfengine_httpd_t cert_t:lnk_file read; +allow cfengine_httpd_t cfengine_httpd_exec_t:file execute_no_trans; +allow cfengine_httpd_t cfengine_postgres_t:unix_stream_socket connectto; + +# allow httpd to use our custom compiled module +allow cfengine_httpd_t cfengine_var_lib_t:file map; +allow cfengine_httpd_t cfengine_var_lib_t:file { append create execute getattr ioctl lock open read setattr unlink write }; + +allow cfengine_httpd_t cfengine_var_lib_t:dir { add_name getattr open read remove_name search write create }; +allow cfengine_httpd_t cfengine_var_lib_t:lnk_file read; + +allow cfengine_httpd_t devlog_t:lnk_file read; +allow cfengine_httpd_t devlog_t:sock_file write; +allow cfengine_httpd_t http_port_t:tcp_socket { name_bind name_connect }; +allow cfengine_httpd_t init_t:dbus send_msg; +allow cfengine_httpd_t init_t:unix_stream_socket { getattr ioctl }; +allow cfengine_httpd_t init_var_run_t:dir search; +allow cfengine_httpd_t kernel_t:unix_dgram_socket sendto; +allow cfengine_httpd_t net_conf_t:file { getattr open read }; +allow cfengine_httpd_t node_t:tcp_socket node_bind; +allow cfengine_httpd_t self:capability { dac_override dac_read_search kill net_bind_service setgid setuid }; +allow cfengine_httpd_t self:netlink_route_socket { bind create getattr nlmsg_read }; +allow cfengine_httpd_t self:process execmem; +allow cfengine_httpd_t self:tcp_socket { accept bind connect create getattr getopt listen setopt shutdown }; +allow cfengine_httpd_t self:udp_socket { connect create getattr }; +allow cfengine_httpd_t self:unix_dgram_socket { connect create }; +allow cfengine_httpd_t sssd_public_t:dir search; +allow cfengine_httpd_t sssd_public_t:file map; +allow cfengine_httpd_t sssd_public_t:file { getattr open read }; +allow cfengine_httpd_t sssd_t:unix_stream_socket connectto; +allow cfengine_httpd_t sssd_var_lib_t:dir search; +allow cfengine_httpd_t sssd_var_lib_t:sock_file write; +allow cfengine_httpd_t syslogd_var_run_t:dir search; +allow cfengine_httpd_t tmp_t:sock_file write; + +# Bidirectional DBus communication between httpd and systemd +allow cfengine_httpd_t system_dbusd_t:dbus send_msg; +allow cfengine_httpd_t system_dbusd_t:unix_stream_socket connectto; +allow cfengine_httpd_t system_dbusd_var_run_t:dir search; +allow cfengine_httpd_t system_dbusd_var_run_t:sock_file write; +allow init_t cfengine_httpd_t:dbus send_msg; + +# TODO: these should not be needed +allow cfengine_httpd_t passwd_file_t:file { getattr open read }; +allow cfengine_httpd_t shell_exec_t:file map; +allow cfengine_httpd_t shell_exec_t:file { execute execute_no_trans }; From e476f8701f129d82beb51150abeaeab04b9e110d Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Mon, 21 Sep 2020 17:13:39 +0200 Subject: [PATCH 150/333] Add extra SELinux policy rules for CFEngine hub Ticket: ENT-6103 Changelog: None (cherry picked from commit 47d9bc89a98f8ad9166caaab8fa73f0b674dd37c) --- misc/selinux/cfengine-enterprise.te | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index c2e95a01e3..dd45d3cb65 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -166,6 +166,7 @@ allow cfengine_execd_t cfengine_var_lib_t:file execute_no_trans; # TODO: this should not be needed allow cfengine_execd_t proc_xen_t:dir search; +allow cfengine_execd_t ssh_port_t:tcp_socket name_connect; allow cfengine_execd_t cfengine_log_t:file { read unlink write }; allow cfengine_execd_t cfengine_log_t:lnk_file { create getattr read unlink }; @@ -216,6 +217,8 @@ allow cfengine_monitord_t cfengine_serverd_exec_t:file getattr; allow cfengine_monitord_t cfengine_agent_exec_t:file getattr; allow cfengine_monitord_t cfengine_hub_exec_t:file getattr; +allow cfengine_monitord_t var_log_t:file { open read }; + allow cfengine_monitord_t self:capability { dac_override dac_read_search sys_ptrace }; allow cfengine_monitord_t crontab_exec_t:file getattr; @@ -255,6 +258,7 @@ allow cfengine_serverd_t unreserved_port_t:tcp_socket name_connect; # TODO: this should not be needed allow cfengine_serverd_t proc_xen_t:dir search; +allow cfengine_serverd_t ssh_port_t:tcp_socket name_connect; allow cfengine_serverd_t cfengine_execd_exec_t:file getattr; allow cfengine_serverd_t cfengine_monitord_exec_t:file getattr; From f52de0b189197a148d2561a216c69da3225d6916 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Mon, 21 Sep 2020 18:07:04 +0200 Subject: [PATCH 151/333] Add SELinux build artifacts to .gitignore (cherry picked from commit 559c4ffbdaafbf043c8e303396cb3d17d501388f) --- .gitignore | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.gitignore b/.gitignore index 5a0e26c6b1..612cc69f83 100644 --- a/.gitignore +++ b/.gitignore @@ -167,3 +167,8 @@ cscope.* # python __pycache__ + +# SELinux policy build artifacts +misc/selinux/cfengine-enterprise.pp +misc/selinux/cfengine-enterprise.if +misc/selinux/tmp \ No newline at end of file From ef3e75fca992b4218209ad0a2bc2942200853afe Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Mon, 21 Sep 2020 19:45:49 +0200 Subject: [PATCH 152/333] Add missing declarations for SELinux policy Ticket: ENT-6103 Changelog: None (cherry picked from commit 6f802334c5a108f8ef4eeaebeae0579b1c559aae) --- misc/selinux/cfengine-enterprise.te | 38 ++++++++++++++++++++++++----- 1 file changed, 32 insertions(+), 6 deletions(-) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index dd45d3cb65..5072ab1cfc 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -16,23 +16,42 @@ require { attribute non_security_file_type; attribute non_auth_file_type; type bin_t; + type cert_t; + type devlog_t; + type kernel_t; type var_t; + type var_log_t; type fs_t; type unconfined_t; type unreserved_port_t; type user_cron_spool_t; type cfengine_serverd_t; type cfengine_execd_exec_t; + type net_conf_t; + type node_t; + type passwd_file_t; type ping_exec_t; + type proc_t; type proc_net_t; type proc_xen_t; type cfengine_serverd_exec_t; + type http_port_t; + type postgresql_port_t; type smtp_port_t; + type ssh_port_t; type rpm_exec_t; type rpm_var_lib_t; + type sssd_t; + type sssd_public_t; + type sssd_var_lib_t; + type sysfs_t; + type sysctl_net_t; type system_cron_spool_t; type systemd_unit_file_t; + type hugetlbfs_t; type init_exec_t; + type init_var_run_t; + type ifconfig_exec_t; type journalctl_exec_t; type cfengine_execd_t; type cfengine_log_t; @@ -48,17 +67,24 @@ require { type crontab_exec_t; type hostname_exec_t; type groupadd_exec_t; + type shell_exec_t; type semanage_exec_t; + type syslogd_var_run_t; + type system_dbusd_t; + type system_dbusd_var_run_t; + type tmp_t; + type tmpfs_t; role system_r; - class tcp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown name_connect accept listen name_bind }; - class udp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; + class tcp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown name_connect accept listen name_bind node_bind }; + class udp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown node_bind }; + class sock_file { create write setattr unlink }; class rawip_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; class netlink_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; class packet_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; class unix_stream_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; - class unix_dgram_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; + class unix_dgram_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown sendto }; class appletalk_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; - class netlink_route_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; + class netlink_route_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown nlmsg_read getopt }; class netlink_firewall_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; class netlink_tcpdiag_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; class netlink_nflog_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; @@ -114,11 +140,11 @@ require { class process { setrlimit transition dyntransition execstack execheap execmem }; class file { execute execute_no_trans getattr ioctl map open read unlink write entrypoint lock link rename append setattr create relabelfrom relabelto }; class fifo_file { create open getattr setattr read write append rename link unlink ioctl lock relabelfrom relabelto }; - class dir { getattr read search open write add_name remove_name lock ioctl }; + class dir { getattr read search open write add_name remove_name lock ioctl create }; class filesystem getattr; class lnk_file { create getattr read unlink }; class unix_stream_socket connectto; - class capability { dac_read_search sys_module chown dac_read_search dac_override fowner fsetid sys_admin mknod net_raw net_admin sys_nice sys_rawio sys_resource setuid setgid sys_nice sys_ptrace}; + class capability { dac_read_search sys_module chown dac_read_search dac_override fowner fsetid sys_admin mknod net_raw net_admin sys_nice sys_rawio sys_resource setuid setgid sys_nice sys_ptrace kill net_bind_service }; class capability2 { mac_admin mac_override block_suspend syslog compromise_kernel wake_alarm }; class association { sendto recvfrom setcontext polmatch }; class security setsecparam; From 0aa296de828197197ead132a1563165c4d7a57b4 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Mon, 12 Oct 2020 08:52:18 -0500 Subject: [PATCH 153/333] Refactor motd example Ticket: CFE-3457 Changelog: None (cherry picked from commit bd9a1e63535a5445b768c87aa87532b0ebed321f) --- examples/mustache_template_motd.cf | 176 +++++++++++++------- examples/mustache_template_motd.cf.mustache | 16 -- 2 files changed, 114 insertions(+), 78 deletions(-) delete mode 100644 examples/mustache_template_motd.cf.mustache diff --git a/examples/mustache_template_motd.cf b/examples/mustache_template_motd.cf index ab75d80ba5..bd3816d4db 100644 --- a/examples/mustache_template_motd.cf +++ b/examples/mustache_template_motd.cf @@ -1,71 +1,123 @@ -body file control { - # To make this policy "standalone" we load parts of the standard library that - # we use - inputs => { "$(sys.libdir)/files.cf", "$(sys.libdir)/packages.cf" }; +bundle agent env_classification +# @brief Classify environment +{ + vars: + # We use presence of key files to know the hosts environment + "environment_semaphores" slist => { "/etc/prod", "/etc/staging" }; + "environment" + string => ifelse( filesexist( @(environment_semaphores) ), "incoherent", + fileexists("/etc/prod"), "production", + fileexists("/etc/staging"), "staging", + "unknown" ); + "env_issue_msg" + string => ifelse( strcmp( "incoherent", $(environment) ), + "WARNING: Environment incoherent (multiple environment semaphores)", + strcmp( "unknown", $(environment) ), + "WARNING: Environment unknown (missing environment semaphores)", + "Host environment classified as $(environment)"); + + # This is to extract the cfengine role ( hub or client ) + "cf_role" + string => ifelse( "policy_server", "Policy Server", "Policy Client"); + + classes: + + # We define a class for the selected environment. It's useful for making + # decisions in other policies + + "env_$(environment)" + expression => "any", + scope => "namespace"; + } +bundle agent env_info +# @brief Relevant environment information +{ + vars: + ## Based on the environment we define different contacts. + "admin_contact" + slist => { "admin@acme.com", "oncall@acme.com" }, + if => strcmp( $(env_classification.environment), "production" ); + + "admin_contact" + slist => { "dev@acme.com" }, + if => strcmp( $(env_classification.environment), "staging" ); + "admin_contact" + slist => { "root@localhost" }, + if => strcmp( $(env_classification.environment), "unknown" ); + + ## This is to extract the available package updates status + "updates_available" + data => packageupdatesmatching(".*", ".*", ".*", ".*"); + "count_updates" int => length("updates_available"); + + classes: + + # We define a class indicating there are updates available, it might be + # useful for various different policies. + + "have_updates_available" + expression => isgreaterthan( $(count_updates), 0), + scope => "namespace"; + +} bundle agent motd { vars: - "motd_path" string => "/etc/motd"; - "template_path" string => "$(this.promise_filename).mustache"; - - ## This is to extract the cf-engine role ( hub or client ) - "cf_role" - string => ifelse( "policy_server", "Policy Server", "Policy Client"); - - ## This is to extract the available package updates status - "updates_available" - data => packageupdatesmatching(".*", ".*", ".*", ".*"); - "amount_updates" int => length("updates_available"); - - ## This is to define a production / stage example role for the server - "environment_list" slist => { "/etc/prod", "/etc/staging" }; - "environment" - string => ifelse( filesexist( @(environment_list) ), "unknown", - fileexists("/etc/prod"), "production", - fileexists("/etc/staging"), "staging", - "unknown" ); - - ## Based on the environment we define different contatcs. - "admin_contact" - slist => { "admin@acme.com", "oncall@acme.com" }, - if => strcmp( $(environment), "production" ); - - "admin_contact" - slist => { "dev@acme.com" }, - if => strcmp( $(environment), "staging" ); - - "admin_contact" - slist => { "root@localhost" }, - if => strcmp( $(environment), "unknown" ); - - ## We define a class to conditionally render the role status in the final - ## motd. - classes: - "env_$(environment)" expression => "any"; + "motd_path" string => "/etc/motd"; + + # It's considered best practice to prepare a data container holding the + # information you need to render the template instead of relying on + # current datastate() - ## In order to get package inventory, we need to have a legacy package - ## promise. - packages: - "vim" -> { "https://dev.cfengine.com/issues/7864" } - package_policy => "add"; + # First we extract currently defined classes from datastate(), then we + # construct the template data. + + "_state" data => datastate(), + if => not( isvariable ( $(this.promiser) ) ); # Limit recursive growth + # and multiple calls to + # datastate() over + # multiple passes. + + "template_data" + data => mergedata('{ "fqhost": "$(sys.fqhost)", + "primary_ip": "$(sys.ipv4)", + "cf_version": "$(sys.cf_version)", + "issue_msg": "$(env_classification.env_issue_msg)", + "cf_role": "$(env_classification.cf_role)", + "count_updates": "$(env_info.count_updates)", + "contacts": env_info.admin_contact, + "classes": _state[classes] + }'); - ## Here is where we render the mustache template files: - "$(motd_path)" - create => "true", - perms => mog( 444, "root", "root"), - template_method => "mustache", - edit_template => "$(template_path)"; - - ## These are good to understand what is going on. - reports: - DEBUG|DEBUG_motd:: - "DEBUG $(this.bundle): $(sys.cf_version) is the detected version"; - "DEBUG $(this.bundle): $(sys.fqhost) is the detected hostname"; - "DEBUG $(this.bundle): $(sys.ipv4) is the ipv4 address for $(sys.fqhost)"; - "DEBUG $(this.bundle): $(motd.cf_role) is the detected role"; - "DEBUG $(this.bundle): $(amount_updates) packages can be updated"; - "DEBUG $(this.bundle): This host is managed by $(admin_contact)"; + "$(motd_path)" + create => "true", + template_method => "inline_mustache", + template_data => @(template_data), + edit_template_string => '# Managed by CFEngine +{{{issue_msg}}} + *** + *** Welcome to {{{fqhost}}} + *** + +* *** * CFEngine Role: {{{cf_role}}} +* *** * CFEngine Version:{{{cf_version}}} +* *** * +* * Host IP: {{{primary_ip}}} + *** {{#classes.have_updates_available}}{{{count_updates}}} package updates available.{{/classes.have_updates_available}}{{^classes.have_updates_available}}No package updates available or status unknown.{{/classes.have_updates_available}} + * * + * * + * * + For support contact:{{#contacts}} + - {{{.}}}{{/contacts}}$(const.n)'; + +} +bundle agent __main__ +{ + methods: + "env_classification"; + "env_info"; + "motd"; } diff --git a/examples/mustache_template_motd.cf.mustache b/examples/mustache_template_motd.cf.mustache deleted file mode 100644 index a9ac5d2a25..0000000000 --- a/examples/mustache_template_motd.cf.mustache +++ /dev/null @@ -1,16 +0,0 @@ -{{#classes.env_unknown}} WARNING Environment Unknown{{/classes.env_unknown}} - ¤¤¤ - ¤¤¤ - ¤¤¤ Welcome into {{{vars.sys.fqhost}}} - - ¤ ¤¤¤ ¤ This system is controlled by - ¤ ¤¤¤ ¤ CFEngine {{{vars.sys.cf_version}}} - ¤ ¤¤¤ ¤ And is a {{{vars.motd.cf_role}}} - ¤ ¤ - ¤¤¤ - ¤ ¤ - ¤ ¤ Host IP {{{vars.sys.ipv4}}} - ¤ ¤ {{{vars.motd.amount_updates}}} package updates available. - {{#vars.motd.admin_contact}}Support Contact: - - {{{.}}} -{{/vars.motd.admin_contact}} From 2b920aa38b7a9e4c60843a25801e6f85a1edeb0e Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Tue, 6 Oct 2020 12:08:43 +0200 Subject: [PATCH 154/333] Fixed memory leak in users promises Found by running valgrind agent run on a demo hub. Changelog: Title Ticket: None Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 571d28089833675065f5a9e2df61805bf7cb76ba) --- cf-agent/verify_users_pam.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/cf-agent/verify_users_pam.c b/cf-agent/verify_users_pam.c index e29d426289..aba95cbb9e 100644 --- a/cf-agent/verify_users_pam.c +++ b/cf-agent/verify_users_pam.c @@ -1434,23 +1434,22 @@ void VerifyOneUsersPromise (const char *puser, const User *u, PromiseResult *res EvalContext *ctx, const Attributes *a, const Promise *pp) { assert(u != NULL); - bool res; - struct passwd *passwd_info; - StringSet *groups_to_set = StringSetNew(); - StringSet *current_secondary_groups = StringSetNew(); - StringSet *groups_missing = StringSetNew(); - passwd_info = GetPwEntry(puser); + struct passwd *passwd_info = GetPwEntry(puser); if (!passwd_info && errno != 0) { Log(LOG_LEVEL_ERR, "Could not get information from user database."); return; } + bool res; if (u->policy == USER_STATE_PRESENT || u->policy == USER_STATE_LOCKED) { if (passwd_info) { + StringSet *groups_to_set = StringSetNew(); + StringSet *current_secondary_groups = StringSetNew(); + StringSet *groups_missing = StringSetNew(); res = GetGroupInfo(puser, u, &groups_to_set, &groups_missing, ¤t_secondary_groups); if (res) { @@ -1476,6 +1475,9 @@ void VerifyOneUsersPromise (const char *puser, const User *u, PromiseResult *res { *result = PROMISE_RESULT_FAIL; } + StringSetDestroy(groups_to_set); + StringSetDestroy(current_secondary_groups); + StringSetDestroy(groups_missing); } else { From 94fe640dfd4505db54b3c909a4a0fc6508e4be38 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Mon, 19 Oct 2020 17:10:34 +0200 Subject: [PATCH 155/333] libntech: Updated to upstream master Signed-off-by: Ole Herman Schumacher Elgesem --- libntech | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libntech b/libntech index ec0065fbac..bb7cdd200b 160000 --- a/libntech +++ b/libntech @@ -1 +1 @@ -Subproject commit ec0065fbacfc8391458fd73c9676ec137e2b6e7b +Subproject commit bb7cdd200b302442770d5a4ca88294552d1a4932 From 66931f03cc3713c0a062a7783dc81bc95c9550ec Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Thu, 15 Oct 2020 12:12:09 +0200 Subject: [PATCH 156/333] libntech: Changes needed for use in Mender Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit b05e31f5ea7ac6a6362d6d7b362e7cbe3d873a21) --- cf-execd/cf-execd-runner.c | 1 + cf-execd/exec-config.c | 1 + libenv/unix_iface.c | 1 + libpromises/generic_agent.c | 1 + 4 files changed, 4 insertions(+) diff --git a/cf-execd/cf-execd-runner.c b/cf-execd/cf-execd-runner.c index 9bfc33f4be..606c0d2ee0 100644 --- a/cf-execd/cf-execd-runner.c +++ b/cf-execd/cf-execd-runner.c @@ -43,6 +43,7 @@ #include #include #include +#include /* StringMatchFullWithPrecompiledRegex() */ #include /* LoadProcessTable()/SelectProcesses() */ #include diff --git a/cf-execd/exec-config.c b/cf-execd/exec-config.c index fcb06958d6..4dda2d6b41 100644 --- a/cf-execd/exec-config.c +++ b/cf-execd/exec-config.c @@ -32,6 +32,7 @@ #include #include #include // TODO: fix +#include // pcre_free() #include diff --git a/libenv/unix_iface.c b/libenv/unix_iface.c index c62e1cb730..d023a55995 100644 --- a/libenv/unix_iface.c +++ b/libenv/unix_iface.c @@ -32,6 +32,7 @@ #include #include /* SeqStringFromString */ #include /* StringMatchFull */ +#include /* StringCaptureData() */ #include #include #include diff --git a/libpromises/generic_agent.c b/libpromises/generic_agent.c index 5cdc34872c..1cab6af0ed 100644 --- a/libpromises/generic_agent.c +++ b/libpromises/generic_agent.c @@ -49,6 +49,7 @@ #include #include #include +#include // pcre #include #include #include From 032f819e61be8ce1124de5c0eecf8d5d7d7c887f Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Mon, 19 Oct 2020 16:16:23 +0200 Subject: [PATCH 157/333] Added missing include for regex.h Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 2dad4604ac9209877de83ed6ca0e077446609b80) --- cf-agent/verify_users_pam.c | 1 + 1 file changed, 1 insertion(+) diff --git a/cf-agent/verify_users_pam.c b/cf-agent/verify_users_pam.c index e29d426289..fd6957d301 100644 --- a/cf-agent/verify_users_pam.c +++ b/cf-agent/verify_users_pam.c @@ -34,6 +34,7 @@ #include #include #include +#include // CompileRegex() #include #include From e1445ee06731bb8416b329cfe12095b7964ffc40 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Mon, 19 Oct 2020 17:31:28 +0200 Subject: [PATCH 158/333] Optimistic: Try to statically check length of string literals Let's see if this works everywhere. I doubt it. Changelog: None Ticket: CFE-3454 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 13a3cd01138968e4b800f41a7465b5b8b4c83731) Signed-off-by: Ole Herman Schumacher Elgesem --- cf-serverd/server_common.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cf-serverd/server_common.c b/cf-serverd/server_common.c index 100e7512b0..524a1531a9 100644 --- a/cf-serverd/server_common.c +++ b/cf-serverd/server_common.c @@ -861,8 +861,8 @@ int StatFile(ServerConnectionState *conn, char *sendbuffer, char *ofilename) bool CompareLocalHash(const char *filename, const char digest[EVP_MAX_MD_SIZE + 1], char sendbuffer[CFD_FALSE_SIZE]) { - assert(CFD_FALSE_SIZE == (strlen(CFD_FALSE) + 1)); - assert(strlen(CFD_FALSE) >= strlen(CFD_TRUE)); + nt_static_assert(CFD_FALSE_SIZE == (strlen(CFD_FALSE) + 1)); + nt_static_assert(strlen(CFD_FALSE) >= strlen(CFD_TRUE)); char translated_filename[CF_BUFSIZE] = { 0 }; TranslatePath(translated_filename, filename); From 615f4d4e4fe54858ad458c965ae34a159231f25d Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Thu, 6 Aug 2020 10:51:53 -0500 Subject: [PATCH 159/333] Adjusted to more reliable list of servers (for now) (cherry picked from commit ee1d03fbd06e95062bfd295486bc923790603649) (cherry picked from commit 50c8ecd071a7ae5622308c8a76fd277c2e27c95c) --- tests/acceptance/01_vars/02_functions/network/selectservers.cf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/acceptance/01_vars/02_functions/network/selectservers.cf b/tests/acceptance/01_vars/02_functions/network/selectservers.cf index 5a0f2c54ea..236b38e0a6 100644 --- a/tests/acceptance/01_vars/02_functions/network/selectservers.cf +++ b/tests/acceptance/01_vars/02_functions/network/selectservers.cf @@ -24,7 +24,7 @@ bundle agent test # The first two hosts listen to port 80, the third does not listen # to port 80, the fourth does not even resolve - "hosts" slist => { "cfengine.com", "www.ntua.gr", "smtp.mail.yahoo.com", "inexistent-server" }; + "hosts" slist => { "cfengine.com", "www.duckduckgo.com", "smtp.mail.yahoo.com", "inexistent-server" }; "retval" int => selectservers("@(hosts)","80","","","100","alive_servers"); } From 53bd0444dccdf3e5a81001cdadd8b80d5761c422 Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Thu, 1 Oct 2020 10:10:39 +0200 Subject: [PATCH 160/333] ENT-6200 Disable flaky test on whole Solaris and older suse, also (cherry picked from commit 1a55637153bd2989775262f14e047b810c01af89) --- .../00_basics/ifelapsed_and_expireafter/timed/defaults.cf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/defaults.cf b/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/defaults.cf index fef37de68a..c03f72b106 100644 --- a/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/defaults.cf +++ b/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/defaults.cf @@ -30,9 +30,9 @@ bundle agent init meta: # Backgrounding doesn't work properly on Windows or with fakeroot. # In addition, RHEL4 seems to have some problem killing the test - # afterwards, and HP-UX and Solaris 9 and 10 are too slow for timing - # to be accurate. - "test_skip_needs_work" string => "windows|using_fakeroot|redhat_4|hpux|sunos_5_9|sunos_5_10"; + # afterwards, and HP-UX is too slow for timing to be accurate. Also, + # this test randomly fails on Solaris and SUSE 11 and 12. + "test_skip_needs_work" string => "windows|using_fakeroot|redhat_4|hpux|sunos|suse_11|suse_12"; methods: test_pass_1:: From 921d4a9c95166a0b7198eff70d068fd61f83b94a Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 21 Oct 2020 17:17:07 +0200 Subject: [PATCH 161/333] Added acceptance test for not() Backported test from master, had to use strcmp() to get correct return type, since 3.15.x doesn't have the changed return type for not(). Changelog: None Ticket: CFE-3470 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 7af98c413c0ca5102a0a678bcaa8c219236e88df) --- tests/acceptance/01_vars/02_functions/not.cf | 100 +++++++++++++++++++ 1 file changed, 100 insertions(+) create mode 100644 tests/acceptance/01_vars/02_functions/not.cf diff --git a/tests/acceptance/01_vars/02_functions/not.cf b/tests/acceptance/01_vars/02_functions/not.cf new file mode 100644 index 0000000000..d9c6f4f7ff --- /dev/null +++ b/tests/acceptance/01_vars/02_functions/not.cf @@ -0,0 +1,100 @@ +###################################################### +# +# Test that not() behaves as expected +# +##################################################### +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +####################################################### + +bundle agent test +{ + meta: + "description" -> { "CFE-3470" } + string => "Test that not() behaves as expected"; + + classes: + "ok" + scope => "namespace", + and => { + # Simple case: + "any", + strcmp("any", not("!any")), + "cfengine", + strcmp("any", not("!cfengine")), + + # Variable expansion: + "$(true_variable)", + strcmp("any", not("$(false_variable)")), + strcmp("any", not(not("$(true_variable)"))), + strcmp("any", not(not(not("$(false_variable)")))), + + # Double expand: + strcmp("any", not(not("$($(true_variable_name))"))), + strcmp("any", not(not(not("$($(false_variable_name))")))), + + # Triple expand: + strcmp("any", not(not("$($($(true_variable_name_name)))"))), + strcmp("any", not(not(not("$($($(false_variable_name_name)))")))), + + # not() should always return any or !any, + # this is important for backwards compatibility: + strcmp(not("any"), "!any"), + strcmp(not("!any"), "any"), + strcmp(not("!cfengine"), "any"), + strcmp(not("!(cfengine.!cfengine)"), "!any"), + strcmp(not("$(true_variable)"), "!any"), + strcmp(not("$(false_variable)"), "any"), + }; + + # In both of these cases we expect the function call (and promise) + # to be skipped because of the unresolved variable: + # This tests function/promise skipping because of unresolved variables, + # but also that there is no syntax / type error, when the unresolved + # function call stays in the and list forever (Never resolved). + "fail_one" + and => { + "any", + strcmp("!any", not("$(unresolved_var)")), + }; + "fail_two" + and => { + "any", + strcmp("!any", not(not("$(unresolved_var)"))), + }; + "fail" + scope => "namespace", + expression => "fail_one|fail_two"; + + vars: + "false_variable" + string => "cfengine.(!cfengine)"; + "true_variable" + string => "cfengine|(!cfengine)"; + "false_variable_name" + string => "false_variable"; + "true_variable_name" + string => "true_variable"; + "false_variable_name_name" + string => "false_variable_name"; + "true_variable_name_name" + string => "true_variable_name"; + +} + +####################################################### + +bundle agent check +{ + + reports: + ok.(!fail):: + "$(this.promise_filename) Pass"; + (!ok)|fail:: + "$(this.promise_filename) FAIL"; +} From 43a8b0ca1203d0aec6efa88398d72b2a3c1a9b86 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Sat, 24 Oct 2020 01:39:05 +0200 Subject: [PATCH 162/333] Extensive test of or() function Backported test from master, had to use strcmp() to get correct return type, since 3.15.x doesn't have the changed return type for or(). Changelog: None Ticket: CFE-3471 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 324840cc640bfec72d35308914f462a70be9790b) --- tests/acceptance/01_vars/02_functions/or.cf | 235 ++++++++++++++++++++ 1 file changed, 235 insertions(+) create mode 100644 tests/acceptance/01_vars/02_functions/or.cf diff --git a/tests/acceptance/01_vars/02_functions/or.cf b/tests/acceptance/01_vars/02_functions/or.cf new file mode 100644 index 0000000000..fee2d1c8d0 --- /dev/null +++ b/tests/acceptance/01_vars/02_functions/or.cf @@ -0,0 +1,235 @@ +###################################################### +# +# Test that or() behaves as expected +# +##################################################### +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +####################################################### + +bundle agent test +{ + meta: + "description" -> { "CFE-3470" } + string => "Test that or() behaves as expected"; + + vars: + "f" # false + string => "(cfengine.(!cfengine))"; + "T" # true, uppercase to be more different visually + string => "(cfengine|(!cfengine))"; + "f_name" + string => "f"; + "T_name" + string => "T"; + "f_name_name" + string => "f_name"; + "T_name_name" + string => "T_name"; + "many_true" + slist => { + "any", + "$(T)", + concat(not(or("$(f)"))), + "(any.cfengine)", + concat(not(or())), + concat(not(or(or()))), + }; + "many_false" + slist => { + "(!any)", + "$(f)", + concat(or(not("$(T)"))), + "(any.!cfengine)", + concat(not("any")), + concat(or()), + }; + "many_both" + slist => { @(many_true), @(many_false) }; + + classes: + + # All elements should be true, fail if one is false: + "ok" + scope => "namespace", + and => { + # Sanity check: + "any", + "cfengine", + + # or() with 0 arguments should default to false: + strcmp("!any", or()), + + # or() with 1 static string: + strcmp("any", or("any")), + strcmp("any", or("cfengine")), + strcmp("any", or("!(!cfengine)")), + + # or() with 1 string with variable expansion(s): + strcmp("any", or("$(T)")), + strcmp("any", or("!$(f)")), + strcmp("any", or("$(T).any")), + strcmp("any", or("$(T).!(!any)")), + strcmp("any", or("any.$(T)")), + strcmp("any", or("!(!any).$(T)")), + strcmp("any", or("any|$(f)")), + strcmp("any", or("!(!any)|$(f)")), + strcmp("any", or("$(T)|$(f)")), + strcmp("any", or("$(f)|$(T)")), + + # or() with slist: + # Careful, if there are expressions in list (using | operator) + # they should be parenthesized for this to work: + strcmp("any", or(join(".", many_true))), + strcmp("any", or(join("|", many_true))), + strcmp("any", or(join("|", many_both))), + strcmp("!any", or(join(".", many_false))), + strcmp("!any", or(join("|", many_false))), + strcmp("!any", or(join(".", many_both))), + + # or() with 1 function call as argument: + strcmp("any", or(or("any"))), + strcmp("any", or(or("cfengine"))), + strcmp("!any", or("!cfengine")), + strcmp("!any", or(not("cfengine"))), + strcmp("!any", or("$(f)")), + strcmp("!any", or(not("$(T)"))), + strcmp("any", or(strcmp("cfengine", "cfengine"))), + strcmp("any", or(strcmp("any", not("$(f)")))), + + # or() with 2 arguments: + strcmp("any", or("any", "cfengine")), + strcmp("any", or("any", "!any")), + strcmp("any", or("!any", "any")), + strcmp("any", or("!(!any)", "!(!cfengine)")), + strcmp("any", or("$(T)", "$(T)")), + strcmp("any", or("$(T)", "$(f)")), + strcmp("any", or("$(T)", "!$(f)")), + strcmp("any", or("$(T)", not("$(f)"))), + strcmp("any", or(not("$(f)"), not("$(f)"))), + + # or() with 3+ arguments (strings): + strcmp("any", or("any", "any", "any")), + strcmp("any", or("any", "any", "any", "any")), + strcmp("any", or("any", "any", "any", "any", "any")), + strcmp("any", or("any", "any", "any", "any", "any", "any")), + strcmp("any", or("any", "any", "any", "any", "any", "!any")), + strcmp("any", or("any", "any", "any", "any", "!any", "!any")), + strcmp("any", or("any", "any", "any", "!any", "!any", "!any")), + strcmp("any", or("any", "any", "!any", "!any", "!any", "!any")), + strcmp("any", or("any", "!any", "!any", "!any", "!any", "!any")), + strcmp("any", or("!any", "!any", "!any", "!any", "!any", "any")), + strcmp("any", or("!any", "!any", "!any", "!any", "any", "any")), + strcmp("any", or("!any", "!any", "!any", "any", "any", "any")), + strcmp("any", or("!any", "!any", "any", "any", "any", "any")), + strcmp("any", or("!any", "any", "any", "any", "any", "any")), + strcmp("any", or("$(T)", "$(T)", "$(T)")), + strcmp("any", or("$(T)", "$(T)", "$(T)", "$(T)")), + strcmp("any", or("$(T)", "$(T)", "$(T)", "$(T)", "$(T)")), + strcmp("any", or("$(T)", "$(T)", "$(T)", "$(T)", "$(T)", "$(T)")), + strcmp("any", or("$(T)", "$(T)", "$(T)", "$(T)", "$(T)", "$(f)")), + strcmp("any", or("$(T)", "$(T)", "$(T)", "$(T)", "$(f)", "$(f)")), + strcmp("any", or("$(T)", "$(T)", "$(T)", "$(f)", "$(f)", "$(f)")), + strcmp("any", or("$(T)", "$(T)", "$(f)", "$(f)", "$(f)", "$(f)")), + strcmp("any", or("$(T)", "$(f)", "$(f)", "$(f)", "$(f)", "$(f)")), + strcmp("any", or("$(f)", "$(f)", "$(f)", "$(f)", "$(f)", "$(T)")), + strcmp("any", or("$(f)", "$(f)", "$(f)", "$(f)", "$(T)", "$(T)")), + strcmp("any", or("$(f)", "$(f)", "$(f)", "$(T)", "$(T)", "$(T)")), + strcmp("any", or("$(f)", "$(f)", "$(T)", "$(T)", "$(T)", "$(T)")), + strcmp("any", or("$(f)", "$(T)", "$(T)", "$(T)", "$(T)", "$(T)")), + + # or() with 3+ function calls: + strcmp("any", or(not("any"), not("any"), not("!any"))), + strcmp("any", or(not("any"), not("any"), not("any"), not("!any"))), + strcmp("any", or(not("any"), not("any"), not("any"), not("any"), not("!any"))), + strcmp("any", or(not("any"), not("any"), not("any"), not("any"), not("any"), not("!any"))), + strcmp("any", or(not("$(T)"), not("$(T)"), not("$(f)"))), + strcmp("any", or(not("$(T)"), not("$(T)"), not("$(T)"), not("$(f)"))), + strcmp("any", or(not("$(T)"), not("$(T)"), not("$(T)"), not("$(T)"), not("$(f)"))), + strcmp("any", or(not("$(T)"), not("$(T)"), not("$(T)"), not("$(T)"), not("$(T)"), not("$(f)"))), + + # or() with deep nesting: + strcmp("!any", or(or())), + strcmp("!any", or(or(or()))), + strcmp("!any", or(or(or(or())))), + strcmp("!any", or(or(or(or(or()))))), + strcmp("!any", or(or(or(or(or(or())))))), + strcmp("any", or(or(or(or(or(or("any"))))))), + strcmp("any", or(or(or(or(or(or("any", "cfengine"))))))), + + # Double expansion: + strcmp("any", or("$($(T_name))")), + strcmp("any", or("$($(f_name))", "$($(T_name))")), + strcmp("any", or("$($(f_name))", "$($(f_name))", "$($(T_name))")), + strcmp("any", or("!$($(T_name))", "!$($(f_name))")), + strcmp("any", or("!$($(T_name))", "!$($(T_name))", "!$($(f_name))")), + strcmp("any", or(not("$($(T_name))"), not("$($(f_name))"))), + + # Triple expansion: + strcmp("any", or("$($($(T_name_name)))")), + strcmp("any", or("$($($(f_name_name)))", "$($($(T_name_name)))")), + strcmp("any", or("$($($(f_name_name)))", "$($($(f_name_name)))", "$($($(T_name_name)))")), + strcmp("any", or("!$($(T_name_name))", "!$($(f_name_name))")), + strcmp("any", or("!$($(T_name_name))", "!$($(T_name_name))", "!$($(f_name_name))")), + strcmp("any", or(not("$($(T_name_name))"), not("$($(f_name_name))"))), + + # or() should always return any or !any, + # this is important for backwards compatibility: + strcmp(or("any"), "any"), + strcmp(or("!any"), "!any"), + strcmp(or("!cfengine"), "!any"), + strcmp(or("!(cfengine|!cfengine)"), "!any"), + strcmp(or("$(T)"), "any"), + strcmp(or("$(f)"), "!any"), + strcmp(or("$(T)", "$(T)"), "any"), + strcmp(or("$(T)", "$(f)"), "any"), + strcmp(or("$(f)", "$(T)"), "any"), + strcmp(or("$(f)", "$(f)"), "!any"), + }; + + # Cases where or() should return false (fail if one is true): + "fail_false" + or => { + strcmp("any", or("$(f)")), + strcmp("any", or("$(f)", "$(f)")), + strcmp("any", or("$(f)", "$(f)", "$(f)")), + strcmp("any", or("$(f)", "$(f)", "$(f)", "$(f)")), + strcmp("any", or("$(f)", "$(f)", "$(f)", "$(f)", "$(f)")), + strcmp("any", or("$(f)", "$(f)", "$(f)", "$(f)", "$(f)", "$(f)")), + strcmp("any", or("$(f)", "$(f)", "$(f)", "$(f)", "$(f)", "$(f)", "$(f)")), + strcmp("any", or("$(f)", "$(f)", "$(f)", "$(f)", "$(f)", "$(f)", "$(f)", "$(f)")), + }; + # Should be skipped because of unresolved variable: + "fail_unresolved" + and => { + "any", + strcmp("any", or("$(unresolved_var)")), + }; + # Check that it's really skipped because of unresolved, + # and not that it accidentally becomes false: + "fail_not_of_unresolved" + and => { + "any", + strcmp("any", or(not("$(unresolved_var)"))), + }; + "fail" + scope => "namespace", + expression => "fail_false|fail_unresolved|fail_not_of_unresolved"; +} + +####################################################### + +bundle agent check +{ + + reports: + ok.(!fail):: + "$(this.promise_filename) Pass"; + (!ok)|fail:: + "$(this.promise_filename) FAIL"; +} From 1d4d8f6e3d1bb9186d1d356834210d9292e3b0b4 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Fri, 23 Oct 2020 19:00:05 +0200 Subject: [PATCH 163/333] Deleted old and() test Did this in separate commit to make review easier. Had a conflict because of ifvarclass / if. Changelog: None Ticket: CFE-3471 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 1903f4510f8dfb619db9b3943b6930b3794ffb29) --- tests/acceptance/01_vars/02_functions/and.cf | 49 -------------------- 1 file changed, 49 deletions(-) delete mode 100644 tests/acceptance/01_vars/02_functions/and.cf diff --git a/tests/acceptance/01_vars/02_functions/and.cf b/tests/acceptance/01_vars/02_functions/and.cf deleted file mode 100644 index b9685bdbb6..0000000000 --- a/tests/acceptance/01_vars/02_functions/and.cf +++ /dev/null @@ -1,49 +0,0 @@ -####################################################### -# -# Test and() with ifvarclass in vars -# -####################################################### - -body common control -{ - inputs => { "../../default.cf.sub" }; - bundlesequence => { default("$(this.promise_filename)") }; - version => "1.0"; -} - -####################################################### - -bundle agent init -{ -vars: - "dummy" string => "dummy"; -} - -####################################################### - -bundle agent test -{ -vars: - "dummy" string => "dummy"; -} - - -####################################################### - -bundle agent check -{ - classes: - "foo" expression => "any"; - "bar" expression => "any"; - "ok" expression => strcmp("$(var)", "whatever"); - - vars: - "var" string => "whatever", - ifvarclass => and("foo", "bar"); - - reports: - ok:: - "$(this.promise_filename) Pass"; -# !ok:: -# "$(this.promise_filename) FAIL"; -} From 64d2c16ef72f1110a7a3fa389be529ad38c20076 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Fri, 23 Oct 2020 18:21:16 +0200 Subject: [PATCH 164/333] Extensive test of and() function Backported test from master, had to use strcmp() to get correct return type, since 3.15.x doesn't have the changed return type for and(). Changelog: None Ticket: CFE-3471 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 1a5a400377f42b37a1f327ec8e61cb599ffed737) --- tests/acceptance/01_vars/02_functions/and.cf | 219 +++++++++++++++++++ 1 file changed, 219 insertions(+) create mode 100644 tests/acceptance/01_vars/02_functions/and.cf diff --git a/tests/acceptance/01_vars/02_functions/and.cf b/tests/acceptance/01_vars/02_functions/and.cf new file mode 100644 index 0000000000..8c514e4e34 --- /dev/null +++ b/tests/acceptance/01_vars/02_functions/and.cf @@ -0,0 +1,219 @@ +###################################################### +# +# Test that and() behaves as expected +# +##################################################### +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +####################################################### + +bundle agent test +{ + meta: + "description" -> { "CFE-3470" } + string => "Test that and() behaves as expected"; + + vars: + "f" # false + string => "(cfengine.(!cfengine))"; + "T" # true, uppercase to be more different visually + string => "(cfengine|(!cfengine))"; + "f_name" + string => "f"; + "T_name" + string => "T"; + "f_name_name" + string => "f_name"; + "T_name_name" + string => "T_name"; + "many_true" + slist => { + "any", + "$(T)", + concat(not(and("$(f)"))), + "(any.cfengine)", + concat(and()), + concat(and(and())), + }; + "many_false" + slist => { + "(!any)", + "$(f)", + concat(and(not("$(T)"))), + "(any.!cfengine)", + concat(not("any")), + concat(not(and())), + }; + "many_both" + slist => { @(many_true), @(many_false) }; + + classes: + + # All elements should be true, fail if one is false: + "ok" + scope => "namespace", + and => { + # Sanity check: + "any", + "cfengine", + + # and() with 0 arguments should default to true: + strcmp("any", and()), + + # and() with 1 static string: + strcmp("any", and("any")), + strcmp("any", and("cfengine")), + strcmp("any", and("!(!cfengine)")), + + # and() with 1 string with variable expansion(s): + strcmp("any", and("$(T)")), + strcmp("any", and("!$(f)")), + strcmp("any", and("$(T).any")), + strcmp("any", and("$(T).!(!any)")), + strcmp("any", and("any.$(T)")), + strcmp("any", and("!(!any).$(T)")), + strcmp("any", and("any|$(f)")), + strcmp("any", and("!(!any)|$(f)")), + strcmp("any", and("$(T)|$(f)")), + strcmp("any", and("$(f)|$(T)")), + + # and() with slist: + # Careful, if there are expressions in list (using | operator) + # they should be parenthesized for this to work: + strcmp("any", and(join(".", many_true))), + strcmp("any", and(join("|", many_true))), + strcmp("any", and(join("|", many_both))), + strcmp("any", and(not(join(".", many_false)))), + strcmp("any", and(not(join("|", many_false)))), + strcmp("any", and(not(join(".", many_both)))), + + # and() with 1 function call as argument: + strcmp("any", and(and("any"))), + strcmp("any", and(and("cfengine"))), + strcmp("any", and(not("!cfengine"))), + strcmp("any", and(not(not("cfengine")))), + strcmp("any", and(not("$(f)"))), + strcmp("any", and(not(not("$(T)")))), + strcmp("any", and(strcmp("cfengine", "cfengine"))), + strcmp("any", and(strcmp("any", not("$(f)")))), + + # and() with 2 arguments: + strcmp("any", and("any", "cfengine")), + strcmp("any", and("!(!any)", "!(!cfengine)")), + strcmp("any", and("$(T)", "$(T)")), + strcmp("any", and("$(T)", "!$(f)")), + strcmp("any", and("$(T)", not("$(f)"))), + strcmp("any", and(not("$(f)"), not("$(f)"))), + + # and() with 3+ arguments (strings): + strcmp("any", and("any", "any", "any")), + strcmp("any", and("any", "any", "any", "any")), + strcmp("any", and("any", "any", "any", "any", "any")), + strcmp("any", and("any", "any", "any", "any", "any", "any")), + strcmp("any", and("$(T)", "$(T)", "$(T)")), + strcmp("any", and("$(T)", "$(T)", "$(T)", "$(T)")), + strcmp("any", and("$(T)", "$(T)", "$(T)", "$(T)", "$(T)")), + strcmp("any", and("$(T)", "$(T)", "$(T)", "$(T)", "$(T)", "$(T)")), + + # and() with 3+ function calls: + strcmp("any", and(not("!any"), not("!any"), not("!any"))), + strcmp("any", and(not("!any"), not("!any"), not("!any"), not("!any"))), + strcmp("any", and(not("!any"), not("!any"), not("!any"), not("!any"), not("!any"))), + strcmp("any", and(not("!any"), not("!any"), not("!any"), not("!any"), not("!any"), not("!any"))), + strcmp("any", and(not("$(f)"), not("$(f)"), not("$(f)"))), + strcmp("any", and(not("$(f)"), not("$(f)"), not("$(f)"), not("$(f)"))), + strcmp("any", and(not("$(f)"), not("$(f)"), not("$(f)"), not("$(f)"), not("$(f)"))), + strcmp("any", and(not("$(f)"), not("$(f)"), not("$(f)"), not("$(f)"), not("$(f)"), not("$(f)"))), + + # and() with deep nesting: + strcmp("any", and(and())), + strcmp("any", and(and(and()))), + strcmp("any", and(and(and(and())))), + strcmp("any", and(and(and(and(and()))))), + strcmp("any", and(and(and(and(and(and())))))), + strcmp("any", and(and(and(and(and(and("any"))))))), + strcmp("any", and(and(and(and(and(and("any", "cfengine"))))))), + + # Double expansion: + strcmp("any", and("$($(T_name))")), + strcmp("any", and("$($(T_name))", "$($(T_name))")), + strcmp("any", and("$($(T_name))", "$($(T_name))", "$($(T_name))")), + strcmp("any", and("!$($(f_name))", "!$($(f_name))")), + strcmp("any", and("!$($(f_name))", "!$($(f_name))", "!$($(f_name))")), + strcmp("any", and(not("$($(f_name))"), not("$($(f_name))"))), + + # Triple expansion: + strcmp("any", and("$($($(T_name_name)))")), + strcmp("any", and("$($($(T_name_name)))", "$($($(T_name_name)))")), + strcmp("any", and("$($($(T_name_name)))", "$($($(T_name_name)))", "$($($(T_name_name)))")), + strcmp("any", and("!$($(f_name_name))", "!$($(f_name_name))")), + strcmp("any", and("!$($(f_name_name))", "!$($(f_name_name))", "!$($(f_name_name))")), + strcmp("any", and(not("$($(f_name_name))"), not("$($(f_name_name))"))), + + # and() should always return any or !any, + # this is important for backwards compatibility: + strcmp(and("any"), "any"), + strcmp(and("!any"), "!any"), + strcmp(and("!cfengine"), "!any"), + strcmp(and("!(cfengine.!cfengine)"), "any"), + strcmp(and("$(T)"), "any"), + strcmp(and("$(f)"), "!any"), + strcmp(and("$(T)", "$(T)"), "any"), + strcmp(and("$(T)", "$(f)"), "!any"), + strcmp(and("$(f)", "$(T)"), "!any"), + strcmp(and("$(f)", "$(f)"), "!any"), + }; + + # Cases where and() should return false (fail if one is true): + "fail_false" + or => { + strcmp("any", and("$(f)")), + strcmp("any", and("$(T)", "$(f)")), + strcmp("any", and("$(T)", "$(T)", "$(f)")), + strcmp("any", and("$(T)", "$(T)", "$(T)", "$(f)")), + strcmp("any", and("$(T)", "$(T)", "$(T)", "$(T)", "$(f)")), + strcmp("any", and("$(T)", "$(T)", "$(T)", "$(T)", "$(T)", "$(f)")), + strcmp("any", and("$(T)", "$(T)", "$(T)", "$(T)", "$(T)", "$(T)", "$(f)")), + strcmp("any", and("$(T)", "$(T)", "$(T)", "$(T)", "$(T)", "$(T)", "$(T)", "$(f)")), + strcmp("any", and("$(T)", "$(T)", "$(T)", "$(T)", "$(T)", "$(T)", "$(f)", "$(T)")), + strcmp("any", and("$(T)", "$(T)", "$(T)", "$(T)", "$(T)", "$(f)", "$(T)", "$(T)")), + strcmp("any", and("$(T)", "$(T)", "$(T)", "$(T)", "$(f)", "$(T)", "$(T)", "$(T)")), + strcmp("any", and("$(T)", "$(T)", "$(T)", "$(f)", "$(T)", "$(T)", "$(T)", "$(T)")), + strcmp("any", and("$(T)", "$(T)", "$(f)", "$(T)", "$(T)", "$(T)", "$(T)", "$(T)")), + strcmp("any", and("$(T)", "$(f)", "$(T)", "$(T)", "$(T)", "$(T)", "$(T)", "$(T)")), + strcmp("any", and("$(f)", "$(T)", "$(T)", "$(T)", "$(T)", "$(T)", "$(T)", "$(T)")), + }; + # Should be skipped because of unresolved variable: + "fail_unresolved" + and => { + "any", + strcmp("any", and("$(unresolved_var)")), + }; + # Check that it's really skipped because of unresolved, + # and not that it accidentally becomes false: + "fail_not_of_unresolved" + and => { + "any", + strcmp("any", and(not("$(unresolved_var)"))), + }; + "fail" + scope => "namespace", + expression => "fail_false|fail_unresolved|fail_not_of_unresolved"; +} + +####################################################### + +bundle agent check +{ + + reports: + ok.(!fail):: + "$(this.promise_filename) Pass"; + (!ok)|fail:: + "$(this.promise_filename) FAIL"; +} From c97107c5656b290aadefdea02ff8b78f6d7d9c1f Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 23 Oct 2020 14:33:52 +0200 Subject: [PATCH 165/333] Call cleanup functions in cf-check:main() Ticket: CFE-3456 Changelog: None (cherry picked from commit a38cc4fe3f4276a272f2428e95352aed7f7b42c2) --- cf-check/cf-check.c | 136 +++++++++++++++++++++++++------------------- 1 file changed, 79 insertions(+), 57 deletions(-) diff --git a/cf-check/cf-check.c b/cf-check/cf-check.c index 54c490c658..1f5fb281ae 100644 --- a/cf-check/cf-check.c +++ b/cf-check/cf-check.c @@ -6,6 +6,7 @@ #include #include #include +#include /* CallCleanupFunctions() */ static void print_version() { @@ -125,7 +126,9 @@ int main(int argc, const char *const *argv) if (StringEndsWith(argv[0], "lmdump")) { // Compatibility mode; act like lmdump if symlinked or renamed: - return lmdump_main(argc, argv); + int ret = lmdump_main(argc, argv); + CallCleanupFunctions(); + return ret; } // When run separately it makes sense for cf-check to have INFO messages @@ -137,6 +140,7 @@ int main(int argc, const char *const *argv) { print_help(); Log(LOG_LEVEL_ERR, "No command given"); + CallCleanupFunctions(); return EXIT_FAILURE; } @@ -144,60 +148,64 @@ int main(int argc, const char *const *argv) int start_index = 1; const char *optstr = "+hMg:dvI"; // + means stop for non opt arg. :) while ((c = getopt_long(argc, (char *const *) argv, optstr, OPTIONS, &start_index)) - != -1) + != -1) { switch (c) { - case 'd': - { - LogSetGlobalLevel(LOG_LEVEL_DEBUG); - break; - } - case 'v': - { - LogSetGlobalLevel(LOG_LEVEL_VERBOSE); - break; - } - case 'I': - { - LogSetGlobalLevel(LOG_LEVEL_INFO); - break; - } - case 'g': - { - LogSetGlobalLevelArgOrExit(optarg); - break; - } - case 'V': - { - print_version(); - return EXIT_SUCCESS; - break; - } - case 'h': - { - print_help(); - return EXIT_SUCCESS; - break; - } - case 'M': - { - Writer *out = FileWriter(stdout); - ManPageWrite(out, "cf-check", time(NULL), - CF_CHECK_SHORT_DESCRIPTION, - CF_CHECK_MANPAGE_LONG_DESCRIPTION, - OPTIONS, HINTS, - NULL, false, - true); - FileWriterDetach(out); - return EXIT_SUCCESS; - break; - } - default: - { - return EXIT_FAILURE; - break; - } + case 'd': + { + LogSetGlobalLevel(LOG_LEVEL_DEBUG); + break; + } + case 'v': + { + LogSetGlobalLevel(LOG_LEVEL_VERBOSE); + break; + } + case 'I': + { + LogSetGlobalLevel(LOG_LEVEL_INFO); + break; + } + case 'g': + { + LogSetGlobalLevelArgOrExit(optarg); + break; + } + case 'V': + { + print_version(); + CallCleanupFunctions(); + return EXIT_SUCCESS; + break; + } + case 'h': + { + print_help(); + CallCleanupFunctions(); + return EXIT_SUCCESS; + break; + } + case 'M': + { + Writer *out = FileWriter(stdout); + ManPageWrite(out, "cf-check", time(NULL), + CF_CHECK_SHORT_DESCRIPTION, + CF_CHECK_MANPAGE_LONG_DESCRIPTION, + OPTIONS, HINTS, + NULL, false, + true); + FileWriterDetach(out); + CallCleanupFunctions(); + return EXIT_SUCCESS; + break; + } + default: + { + CallCleanupFunctions(); + return EXIT_FAILURE; + break; + } } } const char *const *const cmd_argv = argv + optind; @@ -206,30 +214,41 @@ int main(int argc, const char *const *argv) if (StringEqual_IgnoreCase(command, "lmdump")) { - return lmdump_main(cmd_argc, cmd_argv); + int ret = lmdump_main(cmd_argc, cmd_argv); + CallCleanupFunctions(); + return ret; } if (StringEqual_IgnoreCase(command, "dump")) { - return dump_main(cmd_argc, cmd_argv); + int ret = dump_main(cmd_argc, cmd_argv); + CallCleanupFunctions(); + return ret; } if (StringEqual_IgnoreCase(command, "diagnose")) { - return diagnose_main(cmd_argc, cmd_argv); + int ret = diagnose_main(cmd_argc, cmd_argv); + CallCleanupFunctions(); + return ret; } if (StringEqual_IgnoreCase(command, "backup")) { - return backup_main(cmd_argc, cmd_argv); + int ret = backup_main(cmd_argc, cmd_argv); + CallCleanupFunctions(); + return ret; } if (StringEqual_IgnoreCase(command, "repair") || StringEqual_IgnoreCase(command, "remediate")) { - return repair_main(cmd_argc, cmd_argv); + int ret = repair_main(cmd_argc, cmd_argv); + CallCleanupFunctions(); + return ret; } if (StringEqual_IgnoreCase(command, "help")) { if (cmd_argc > 2) { Log(LOG_LEVEL_ERR, "help takes exactly 0 or 1 arguments"); + CallCleanupFunctions(); return EXIT_FAILURE; } else if (cmd_argc <= 1) @@ -241,15 +260,18 @@ int main(int argc, const char *const *argv) assert(cmd_argc == 2); CFCheckHelpTopic(cmd_argv[1]); } + CallCleanupFunctions(); return EXIT_SUCCESS; } if (StringEqual_IgnoreCase(command, "version")) { print_version(); + CallCleanupFunctions(); return EXIT_SUCCESS; } print_help(); Log(LOG_LEVEL_ERR, "Unrecognized command: '%s'", command); + CallCleanupFunctions(); return EXIT_FAILURE; } From 41286ce90f15c113f2f2b640e88e4d93a865705a Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 23 Oct 2020 14:35:19 +0200 Subject: [PATCH 166/333] Call cleanup functions in cf-execd:main() Ticket: CFE-3456 Changelog: None (cherry picked from commit 382e55450f1e5c47c1ce90220f62f4bb51cb477e) --- cf-execd/cf-execd.c | 1 + 1 file changed, 1 insertion(+) diff --git a/cf-execd/cf-execd.c b/cf-execd/cf-execd.c index 9b5253a319..8f524267fb 100644 --- a/cf-execd/cf-execd.c +++ b/cf-execd/cf-execd.c @@ -176,6 +176,7 @@ int main(int argc, char *argv[]) GenericAgentFinalize(ctx, config); ExecConfigDestroy(exec_config); ExecdConfigDestroy(execd_config); + CallCleanupFunctions(); return 0; } From 62ac699197483c10ce88c7cb77f598dc5515dc22 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 23 Oct 2020 14:37:27 +0200 Subject: [PATCH 167/333] Call cleanup functions in cf-key:main() Ticket: CFE-3456 Changelog: None (cherry picked from commit 531003ce5c1e5dd8ccccd0d953fa661b057617a4) --- cf-key/cf-key.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/cf-key/cf-key.c b/cf-key/cf-key.c index f81c241421..9c43226498 100644 --- a/cf-key/cf-key.c +++ b/cf-key/cf-key.c @@ -151,11 +151,13 @@ int main(int argc, char *argv[]) { SetupSignalsForCfKey(handleShowKeysSignal); ShowLastSeenHosts(!NO_TRUNCATE); + CallCleanupFunctions(); return 0; } if (print_digest_arg) { + CallCleanupFunctions(); return PrintDigest(print_digest_arg); } @@ -182,17 +184,20 @@ int main(int argc, char *argv[]) if (status == 0 || status == 1) { Log (LOG_LEVEL_VERBOSE, - "Forced removal of entry '%s' was successful", - remove_keys_host); + "Forced removal of entry '%s' was successful", + remove_keys_host); + CallCleanupFunctions(); return 0; } } + CallCleanupFunctions(); return status; } if(LICENSE_INSTALL) { bool success = LicenseInstall(LICENSE_SOURCE); + CallCleanupFunctions(); return success ? 0 : 1; } @@ -214,6 +219,7 @@ int main(int argc, char *argv[]) bool ret = TrustKey(filename, ipaddr, username); free(arg); + CallCleanupFunctions(); return ret ? EXIT_SUCCESS : EXIT_FAILURE; } @@ -237,6 +243,7 @@ int main(int argc, char *argv[]) GenericAgentFinalize(ctx, config); + CallCleanupFunctions(); return ret ? EXIT_SUCCESS : EXIT_FAILURE; } From 4e70402021f5407104141b42045aa8bef0c83d27 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 23 Oct 2020 14:40:49 +0200 Subject: [PATCH 168/333] Call cleanup functions in cf-monitord:main() Ticket: CFE-3456 Changelog: None (cherry picked from commit f8bee03ee48b5795788630ba59ad60f0bcfc77ac) --- cf-monitord/cf-monitord.c | 1 + 1 file changed, 1 insertion(+) diff --git a/cf-monitord/cf-monitord.c b/cf-monitord/cf-monitord.c index 160e86905e..b38e708e83 100644 --- a/cf-monitord/cf-monitord.c +++ b/cf-monitord/cf-monitord.c @@ -132,6 +132,7 @@ int main(int argc, char *argv[]) PolicyDestroy(policy); GenericAgentFinalize(ctx, config); + CallCleanupFunctions(); return 0; } From 26346f53ecc2cc4f5c01f4a2234b262376b7e466 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 23 Oct 2020 14:44:05 +0200 Subject: [PATCH 169/333] Call cleanup functions in cf-promises:main() Ticket: CFE-3456 Changelog: None (cherry picked from commit eb27cec8876837660b11d31c1dc1c71a08754b5a) --- cf-promises/cf-promises.c | 1 + 1 file changed, 1 insertion(+) diff --git a/cf-promises/cf-promises.c b/cf-promises/cf-promises.c index f234738e22..118d7005e7 100644 --- a/cf-promises/cf-promises.c +++ b/cf-promises/cf-promises.c @@ -497,5 +497,6 @@ GenericAgentConfig *CheckOpts(int argc, char **argv) DoCleanupAndExit(EXIT_FAILURE); } + CallCleanupFunctions(); return config; } From c09341d60e24464d7771dee05a9b2c90f2b441c8 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 23 Oct 2020 14:45:44 +0200 Subject: [PATCH 170/333] Call cleanup functions in cf-runagent:main() Ticket: CFE-3456 Changelog: None (cherry picked from commit f6983f15717cb1a1b341cf652009b8582a820bb4) --- cf-runagent/cf-runagent.c | 1 + 1 file changed, 1 insertion(+) diff --git a/cf-runagent/cf-runagent.c b/cf-runagent/cf-runagent.c index 6a16a77f5d..e29634def9 100644 --- a/cf-runagent/cf-runagent.c +++ b/cf-runagent/cf-runagent.c @@ -247,6 +247,7 @@ int main(int argc, char *argv[]) PolicyDestroy(policy); GenericAgentFinalize(ctx, config); + CallCleanupFunctions(); return 0; } From 123ff5b9c0300614d76af6e700c8aa53561fad3c Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 23 Oct 2020 14:47:54 +0200 Subject: [PATCH 171/333] Call cleanup functions in cf-secret:main() Ticket: CFE-3456 Changelog: None (cherry picked from commit ebba518aba008855d357b745fbf89cba94f9754e) --- cf-secret/cf-secret.c | 36 +++++++++++++++++++----------------- 1 file changed, 19 insertions(+), 17 deletions(-) diff --git a/cf-secret/cf-secret.c b/cf-secret/cf-secret.c index 72e159a5cd..2a63466606 100644 --- a/cf-secret/cf-secret.c +++ b/cf-secret/cf-secret.c @@ -59,6 +59,7 @@ #include #include #include +#include /* DoCleanupAndExit(), CallCleanupFunctions() */ #define BUFSIZE 1024 @@ -815,7 +816,7 @@ int main(int argc, char *argv[]) if (argc < 2) { CFKeyCryptHelp(); - exit(EXIT_FAILURE); + DoCleanupAndExit(EXIT_FAILURE); } opterr = 0; @@ -851,11 +852,11 @@ int main(int argc, char *argv[]) { case 'h': CFKeyCryptHelp(); - exit(EXIT_SUCCESS); + DoCleanupAndExit(EXIT_SUCCESS); break; case 'M': CFKeyCryptMan(); - exit(EXIT_SUCCESS); + DoCleanupAndExit(EXIT_SUCCESS); break; case 'd': LogSetGlobalLevel(LOG_LEVEL_DEBUG); @@ -884,7 +885,7 @@ int main(int argc, char *argv[]) default: Log(LOG_LEVEL_ERR, "Unknown option '-%c'", optopt); CFKeyCryptHelp(); - exit(EXIT_FAILURE); + DoCleanupAndExit(EXIT_FAILURE); } } @@ -892,7 +893,7 @@ int main(int argc, char *argv[]) { printf("Command required. Specify either 'encrypt', 'decrypt' or 'print-headers'\n"); CFKeyCryptHelp(); - exit(EXIT_FAILURE); + DoCleanupAndExit(EXIT_FAILURE); } /* Increment 'optind' because of command being argv[0]. */ @@ -903,7 +904,7 @@ int main(int argc, char *argv[]) if (argc > (optind + offset)) { Log(LOG_LEVEL_ERR, "Unexpected non-option argument: '%s'", argv[optind + 1]); - exit(EXIT_FAILURE); + DoCleanupAndExit(EXIT_FAILURE); } if (print_headers) @@ -919,7 +920,7 @@ int main(int argc, char *argv[]) if (!CheckHeader(key, value)) { fclose(input_file); - exit(EXIT_FAILURE); + DoCleanupAndExit(EXIT_FAILURE); } printf("%s: %s\n", key, value); @@ -935,7 +936,7 @@ int main(int argc, char *argv[]) } } fclose(input_file); - exit(EXIT_SUCCESS); + DoCleanupAndExit(EXIT_SUCCESS); } if (decrypt && (host_arg == NULL) && (key_path_arg == NULL)) @@ -951,18 +952,18 @@ int main(int argc, char *argv[]) { Log(LOG_LEVEL_ERR, "--host/-H is used to specify a public key and cannot be used with --key/-k"); - exit(EXIT_FAILURE); + DoCleanupAndExit(EXIT_FAILURE); } if (input_path == NULL) { Log(LOG_LEVEL_ERR, "No input file specified (Use -h for help)"); - exit(EXIT_FAILURE); + DoCleanupAndExit(EXIT_FAILURE); } if (output_path == NULL) { Log(LOG_LEVEL_ERR, "No output file specified (Use -h for help)"); - exit(EXIT_FAILURE); + DoCleanupAndExit(EXIT_FAILURE); } CryptoInitialize(); @@ -994,7 +995,7 @@ int main(int argc, char *argv[]) Log(LOG_LEVEL_ERR, "Unable to locate key for host '%s'", host); SeqDestroy(hosts); SeqDestroy(key_paths); - exit(EXIT_FAILURE); + DoCleanupAndExit(EXIT_FAILURE); } SeqAppend(key_paths, host_key_path); } @@ -1012,7 +1013,7 @@ int main(int argc, char *argv[]) if (pub_keys == NULL) { Log(LOG_LEVEL_ERR, "Failed to load public key(s)"); - exit(EXIT_FAILURE); + DoCleanupAndExit(EXIT_FAILURE); } success = RSAEncrypt(pub_keys, input_path, output_path); @@ -1020,7 +1021,7 @@ int main(int argc, char *argv[]) if (!success) { Log(LOG_LEVEL_ERR, "Encryption failed"); - exit(EXIT_FAILURE); + DoCleanupAndExit(EXIT_FAILURE); } } else if (decrypt) @@ -1031,7 +1032,7 @@ int main(int argc, char *argv[]) { Log(LOG_LEVEL_ERR, "--decrypt requires only one key/host to be specified"); SeqDestroy(key_paths); - exit(EXIT_FAILURE); + DoCleanupAndExit(EXIT_FAILURE); } RSA *private_key = ReadPrivateKey((char *) SeqAt(key_paths, 0)); SeqDestroy(key_paths); @@ -1040,14 +1041,15 @@ int main(int argc, char *argv[]) if (!success) { Log(LOG_LEVEL_ERR, "Decryption failed"); - exit(EXIT_FAILURE); + DoCleanupAndExit(EXIT_FAILURE); } } else { ProgrammingError("Unexpected error in cf-secret"); - exit(EXIT_FAILURE); + DoCleanupAndExit(EXIT_FAILURE); } + CallCleanupFunctions(); return 0; } From 3d524cd871ced672a3bcab5c2a069266dc71e253 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 23 Oct 2020 14:48:32 +0200 Subject: [PATCH 172/333] Call cleanup functions in cf-serverd:main() Ticket: CFE-3456 Changelog: None (cherry picked from commit b85afcdfd2d7329f2ba137a1cff698521ceb44a8) --- cf-serverd/cf-serverd.c | 1 + 1 file changed, 1 insertion(+) diff --git a/cf-serverd/cf-serverd.c b/cf-serverd/cf-serverd.c index 817fc4cbb8..603ee6713f 100644 --- a/cf-serverd/cf-serverd.c +++ b/cf-serverd/cf-serverd.c @@ -80,5 +80,6 @@ int main(int argc, char *argv[]) GenericAgentFinalize(ctx, config); CleanReportBookFilterSet(); } + CallCleanupFunctions(); return 0; } From 98306c49382c962624ed3b429b2e9a6f367f3127 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 23 Oct 2020 14:49:15 +0200 Subject: [PATCH 173/333] Call cleanup functions in cf-testd:main() Ticket: CFE-3456 Changelog: None (cherry picked from commit ea83b0dedae1e797e1229dcfd031e9b16d822fd5) --- cf-testd/cf-testd.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/cf-testd/cf-testd.c b/cf-testd/cf-testd.c index 94564aa259..57e3842c60 100644 --- a/cf-testd/cf-testd.c +++ b/cf-testd/cf-testd.c @@ -820,6 +820,7 @@ int main(int argc, char *argv[]) if (failure) { + CallCleanupFunctions(); return EXIT_FAILURE; } @@ -837,5 +838,6 @@ int main(int argc, char *argv[]) CFTestD_ConfigDestroy(thread_configs[i]); } + CallCleanupFunctions(); return failure ? EXIT_FAILURE : EXIT_SUCCESS; } From 96c4b2a8b08367db44e69ec6703da888a01843e8 Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Thu, 29 Oct 2020 13:33:46 +0100 Subject: [PATCH 174/333] Added changelog for 3.15.3 --- ChangeLog | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/ChangeLog b/ChangeLog index e4bdc7290e..1a72b8acf2 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,31 @@ +3.15.3: + - AVCs are no longer produced for CFEngine processes accessing /proc/net + (CFE-3240) + - Added sys.cf_secret component variable + - Backgrounded commands are now correctly executed in the child process + (CFE-3379) + - Changed bootstrap to loopback to a warning instead of exit (CFE-3304) + - Changed the syntax of cf-secret to match other components + - Directory listings in files changes monitoring are now only updated + when there is a change (CFE-3382) + - Expand variables in data/list references (CFE-3299) + - Fix how we check for `--cols` argument to `ps`. (ENT-6098) + - Fixed locking of promises using log_repaired / log_string with + timestamps (CFE-3376) + - Fixed memory leak in users promises + - Fixed race condition when multiple agents are acquiring critical + section locks simultaneously (CFE-3361) + - LMDB files are now created with correct permissions (ENT-5986) + - Promises with 'action => bg()' no longer break reporting data + (ENT-6042) + - Spaces inside square brackets (slist/data index) are now allowed in + class expressions (CFE-3320) + - Variables specifying data/list names in @() references are now expanded + (CFE-2434) + - cf-secret binary for managing secrets was added (CFE-2613) + - cf-monitord now uses /proc/net/* files to get network information if + possible (CFE-2945) + 3.15.2: - 'if' constraint now works in combination with class contexts (CFE-2615) From df1bdfa6916b18998a6fbaa61727d02257539bc1 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Fri, 30 Oct 2020 14:34:53 +0100 Subject: [PATCH 175/333] Update ChangeLog --- ChangeLog | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/ChangeLog b/ChangeLog index 1a72b8acf2..b624eb5c32 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,30 +1,29 @@ 3.15.3: - - AVCs are no longer produced for CFEngine processes accessing /proc/net - (CFE-3240) + - cf-secret binary for managing secrets was added (CFE-2613) - Added sys.cf_secret component variable - - Backgrounded commands are now correctly executed in the child process - (CFE-3379) + - cf-monitord now uses /proc/net/* files to get network information if possible + (CFE-2945) - Changed bootstrap to loopback to a warning instead of exit (CFE-3304) - - Changed the syntax of cf-secret to match other components - Directory listings in files changes monitoring are now only updated when there is a change (CFE-3382) - - Expand variables in data/list references (CFE-3299) - - Fix how we check for `--cols` argument to `ps`. (ENT-6098) - - Fixed locking of promises using log_repaired / log_string with - timestamps (CFE-3376) - - Fixed memory leak in users promises - - Fixed race condition when multiple agents are acquiring critical - section locks simultaneously (CFE-3361) + - Variables are now correctly expanded in data/list references (CFE-3299) - LMDB files are now created with correct permissions (ENT-5986) - Promises with 'action => bg()' no longer break reporting data (ENT-6042) - - Spaces inside square brackets (slist/data index) are now allowed in + - Backgrounded commands are now correctly executed in the child process + (CFE-3379) + - AVCs are no longer produced for CFEngine processes accessing /proc/net + (CFE-3240) + - Spaces inside square brackets (slist/data indices) are now allowed in class expressions (CFE-3320) - Variables specifying data/list names in @() references are now expanded (CFE-2434) - - cf-secret binary for managing secrets was added (CFE-2613) - - cf-monitord now uses /proc/net/* files to get network information if - possible (CFE-2945) + - Fixed how we check for `--cols` argument to `ps` (ENT-6098) + - Fixed locking of promises using log_repaired / log_string with + timestamps (CFE-3376) + - Fixed memory leak in users promises + - Fixed race condition when multiple agents are acquiring critical + section locks simultaneously (CFE-3361) 3.15.2: - 'if' constraint now works in combination with class contexts From 52cc6d8bac3b015449d2bdd76d4526ca7f950b20 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 11 Nov 2020 00:36:04 +0100 Subject: [PATCH 176/333] Bumped .CFVERSION number to 3.15.4 Signed-off-by: Ole Herman Schumacher Elgesem --- .CFVERSION | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.CFVERSION b/.CFVERSION index b7b6bc30c0..8f6e357b33 100644 --- a/.CFVERSION +++ b/.CFVERSION @@ -1 +1 @@ -3.15.3 +3.15.4 From 1c87d00322e97bc034bc77b9b83033bfba586c25 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Fri, 13 Nov 2020 14:09:39 +0100 Subject: [PATCH 177/333] Added warning when trying to use {{.}} to expand containers in mustache templates Changelog: Title Ticket: CFE-3457 CFE-3489 Signed-off-by: Ole Herman Schumacher Elgesem --- libntech | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libntech b/libntech index bb7cdd200b..4e9efcb841 160000 --- a/libntech +++ b/libntech @@ -1 +1 @@ -Subproject commit bb7cdd200b302442770d5a4ca88294552d1a4932 +Subproject commit 4e9efcb84172110fa92742836b8d34688983c2e7 From a5c80cf60782ff3e21225cba63139c1672f3b366 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Wed, 18 Nov 2020 14:11:05 -0600 Subject: [PATCH 178/333] Switched SETUID warning to notice Ticket: ENT-6519 Changelog: New observations of root owned SETUID programs moved from WARN to NOTICE (cherry picked from commit 88ac1a343d0f3980053ee04bb5d389c1dbe45152) --- cf-agent/verify_files_utils.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/cf-agent/verify_files_utils.c b/cf-agent/verify_files_utils.c index 4ed5284e4e..b544a6a7cd 100644 --- a/cf-agent/verify_files_utils.c +++ b/cf-agent/verify_files_utils.c @@ -3317,8 +3317,7 @@ static PromiseResult VerifySetUidGid(EvalContext *ctx, const char *file, const s { if (amroot) { - cfPS(ctx, LOG_LEVEL_WARNING, PROMISE_RESULT_WARN, pp, attr, "NEW SETUID root PROGRAM '%s'", file); - result = PromiseResultUpdate(result, PROMISE_RESULT_WARN); + Log(LOG_LEVEL_NOTICE, "NEW SETUID root PROGRAM '%s' ", file); } PrependItem(&VSETXIDLIST, file, NULL); From ea0fefc6b968a0153e143b13e98aaf8f6453ac0f Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Mon, 7 Dec 2020 20:50:32 +0100 Subject: [PATCH 179/333] Policy function storejson() no longer truncates strings lager than 4096 bytes Changelog: Title Ticket: CFE-2507 Signed-off-by: Lars Erik Wik (cherry picked from commit f2531480258ae62acc0cb753cd334710e50be53f) --- libpromises/evalfunction.c | 14 +----------- .../02_functions/storejson_edge_case.cf | 22 +++++++++++++++++++ .../storejson_edge_case.cf.expected.json | 3 +++ 3 files changed, 26 insertions(+), 13 deletions(-) create mode 100644 tests/acceptance/01_vars/02_functions/storejson_edge_case.cf create mode 100644 tests/acceptance/01_vars/02_functions/storejson_edge_case.cf.expected.json diff --git a/libpromises/evalfunction.c b/libpromises/evalfunction.c index b05d34253c..c37f0bb960 100644 --- a/libpromises/evalfunction.c +++ b/libpromises/evalfunction.c @@ -7034,24 +7034,12 @@ static FnCallResult FnCallStoreJson(EvalContext *ctx, ARG_UNUSED const Policy *p } Writer *w = StringWriter(); - int length; JsonWrite(w, json, 0); JsonDestroyMaybe(json, allocated); Log(LOG_LEVEL_DEBUG, "%s: from data container %s, got JSON data '%s'", fp->name, name_str, StringWriterData(w)); - length = strlen(StringWriterData(w)); - if (length >= CF_BUFSIZE) - { - Log(LOG_LEVEL_INFO, "%s: truncating data container %s JSON data from %d bytes to %d", - fp->name, name_str, length, CF_BUFSIZE); - } - - char buf[CF_BUFSIZE]; - snprintf(buf, CF_BUFSIZE, "%s", StringWriterData(w)); - WriterClose(w); - - return FnReturn(buf); + return FnReturnNoCopy(StringWriterClose(w)); } diff --git a/tests/acceptance/01_vars/02_functions/storejson_edge_case.cf b/tests/acceptance/01_vars/02_functions/storejson_edge_case.cf new file mode 100644 index 0000000000..ef12c462b0 --- /dev/null +++ b/tests/acceptance/01_vars/02_functions/storejson_edge_case.cf @@ -0,0 +1,22 @@ +# Test fix for previous bug that truncates JSON data greater than 4096 bytes + +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +########################################################## + +bundle agent test { + vars: + "data" string => storejson('{ "A very long text": "Hello, everyone! This is the LONGEST TEXT EVER! I was inspired by the various other longest texts ever on the internet, and I wanted to make my own. So here it is! This is going to be a WORLD RECORD! This is actually my third attempt at doing this. The first time, I didnt save it. The second time, the Neocities editor crashed. Now Im writing this in Notepad, then copying it into the Neocities editor instead of typing it directly in the Neocities editor to avoid crashing. It sucks that my past two attempts are gone now. Those actually got pretty long. Not the longest, but still pretty long. I hope this one wont get lost somehow. Anyways, lets talk about WAFFLES! I like waffles. Waffles are cool. Waffles is a funny word. Theres a Teen Titans Go episode called Waffles where the word Waffles is said a hundred-something times. Its pretty annoying. Theres also a Teen Titans Go episode about Pig Latin. Dont know what Pig Latin is? Its a language where you take all the consonants before the first vowel, move them to the end, and add -ay to the end. If the word begins with a vowel, you just add -way to the end. For example, Waffles becomes Afflesway. Ive been speaking Pig Latin fluently since the fourth grade, so it surprised me when I saw the episode for the first time. I speak Pig Latin with my sister sometimes. Its pretty fun. I like speaking it in public so that everyone around us gets confused. Thats never actually happened before, but if it ever does, twill be pretty funny. By the way, twill is a word I invented recently, and its a contraction of it will. I really hope it gains popularity in the near future, because twill is WAY more fun than saying itll. Itll is too boring. Nobody likes boring. This is nowhere near being the longest text ever, but eventually it will be! I might still be writing this a decade later, who knows? But right now, its not very long. But Ill just keep writing until it is the longest! Have you ever heard the song Dau Dau by Awesome Scampis? Its an amazing song. Look it up on YouTube! I play that song all the time around my sister! It drives her crazy, and I love it. Another way I like driving my sister crazy is by speaking my own made up language to her. She hates the languages I make! The only language that we both speak besides English is Pig Latin. I think you already knew that. Whatever. I think Im gonna go for now. Bye! Hi, Im back now. Im gonna contribute more to this soon-to-be giant wall of text. I just realised I have a giant stuffed frog on my bed. I forgot his name. Im pretty sure it was something stupid though. I think it was FROG in Morse Code or something. Morse Code is cool. I know a bit of it, but Im not very good at it. Im also not very good at French. I barely know anything in French, and my pronunciation probably sucks. But Im learning it, at least. Im also learning Esperanto. Its this language that was made up by some guy a long time ago to be the universal language. A lot of people speak it. I am such a language nerd. Half of this text is probably gonna be about languages. But hey, as long as its long! Ha, get it? As LONG as its LONG? Im so funny, right? No, Im not. I should probably get some sleep. Goodnight! Hello, Im back again. I basically have only two interests nowadays: languages and furries. What? Oh, sorry, I thought you knew I was a furry. Haha, oops. Anyway, yeah, Im a furry, but since Im a young furry, I cant really do as much as I would like to do in the fandom. When Im older, I would like to have a fursuit, go to furry conventions, all that stuff. But for now I can only dream of that. Sorry you had to deal with me talking about furries, but Im honestly very desperate for this to be the longest text ever. Last night I was watching nothing but fursuit unboxings. I think I need help. This one time, me and my mom were going to go to a furry Christmas party, but we didnt end up going because of the fact that there was alcohol on the premises, and that she didnt wanna have to be a mom dragging her son through a crowd of furries. Both of those reasons were understandable. Okay, hopefully I wont have to talk about furries anymore. I dont care if youre a furry reading this right now, I just dont wanna have to torture everyone else."}'); +} + +bundle agent check { + methods: + "check" usebundle => dcs_check_state(test, + "$(this.promise_filename).expected.json", + $(this.promise_filename)); +} diff --git a/tests/acceptance/01_vars/02_functions/storejson_edge_case.cf.expected.json b/tests/acceptance/01_vars/02_functions/storejson_edge_case.cf.expected.json new file mode 100644 index 0000000000..2a82a0fcaa --- /dev/null +++ b/tests/acceptance/01_vars/02_functions/storejson_edge_case.cf.expected.json @@ -0,0 +1,3 @@ +{ + "data": "{\n \"A very long text\": \"Hello, everyone! This is the LONGEST TEXT EVER! I was inspired by the various other longest texts ever on the internet, and I wanted to make my own. So here it is! This is going to be a WORLD RECORD! This is actually my third attempt at doing this. The first time, I didnt save it. The second time, the Neocities editor crashed. Now Im writing this in Notepad, then copying it into the Neocities editor instead of typing it directly in the Neocities editor to avoid crashing. It sucks that my past two attempts are gone now. Those actually got pretty long. Not the longest, but still pretty long. I hope this one wont get lost somehow. Anyways, lets talk about WAFFLES! I like waffles. Waffles are cool. Waffles is a funny word. Theres a Teen Titans Go episode called Waffles where the word Waffles is said a hundred-something times. Its pretty annoying. Theres also a Teen Titans Go episode about Pig Latin. Dont know what Pig Latin is? Its a language where you take all the consonants before the first vowel, move them to the end, and add -ay to the end. If the word begins with a vowel, you just add -way to the end. For example, Waffles becomes Afflesway. Ive been speaking Pig Latin fluently since the fourth grade, so it surprised me when I saw the episode for the first time. I speak Pig Latin with my sister sometimes. Its pretty fun. I like speaking it in public so that everyone around us gets confused. Thats never actually happened before, but if it ever does, twill be pretty funny. By the way, twill is a word I invented recently, and its a contraction of it will. I really hope it gains popularity in the near future, because twill is WAY more fun than saying itll. Itll is too boring. Nobody likes boring. This is nowhere near being the longest text ever, but eventually it will be! I might still be writing this a decade later, who knows? But right now, its not very long. But Ill just keep writing until it is the longest! Have you ever heard the song Dau Dau by Awesome Scampis? Its an amazing song. Look it up on YouTube! I play that song all the time around my sister! It drives her crazy, and I love it. Another way I like driving my sister crazy is by speaking my own made up language to her. She hates the languages I make! The only language that we both speak besides English is Pig Latin. I think you already knew that. Whatever. I think Im gonna go for now. Bye! Hi, Im back now. Im gonna contribute more to this soon-to-be giant wall of text. I just realised I have a giant stuffed frog on my bed. I forgot his name. Im pretty sure it was something stupid though. I think it was FROG in Morse Code or something. Morse Code is cool. I know a bit of it, but Im not very good at it. Im also not very good at French. I barely know anything in French, and my pronunciation probably sucks. But Im learning it, at least. Im also learning Esperanto. Its this language that was made up by some guy a long time ago to be the universal language. A lot of people speak it. I am such a language nerd. Half of this text is probably gonna be about languages. But hey, as long as its long! Ha, get it? As LONG as its LONG? Im so funny, right? No, Im not. I should probably get some sleep. Goodnight! Hello, Im back again. I basically have only two interests nowadays: languages and furries. What? Oh, sorry, I thought you knew I was a furry. Haha, oops. Anyway, yeah, Im a furry, but since Im a young furry, I cant really do as much as I would like to do in the fandom. When Im older, I would like to have a fursuit, go to furry conventions, all that stuff. But for now I can only dream of that. Sorry you had to deal with me talking about furries, but Im honestly very desperate for this to be the longest text ever. Last night I was watching nothing but fursuit unboxings. I think I need help. This one time, me and my mom were going to go to a furry Christmas party, but we didnt end up going because of the fact that there was alcohol on the premises, and that she didnt wanna have to be a mom dragging her son through a crowd of furries. Both of those reasons were understandable. Okay, hopefully I wont have to talk about furries anymore. I dont care if youre a furry reading this right now, I just dont wanna have to torture everyone else.\"\n}" +} From 5c94856aa64d9be659f86e5c3360e1ce51b4094b Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Fri, 11 Dec 2020 13:54:43 +0100 Subject: [PATCH 180/333] Added edge case test files for acceptance test Ticket: CFE-2686 Signed-off-by: Lars Erik Wik (cherry picked from commit 8b1fa7f7512656528f9f4067f726a1832d228881) --- .../01_vars/02_functions/format_edge_case.cf | 28 +++++++++++++++++++ .../format_edge_case.cf.expected.json | 11 ++++++++ 2 files changed, 39 insertions(+) create mode 100644 tests/acceptance/01_vars/02_functions/format_edge_case.cf create mode 100644 tests/acceptance/01_vars/02_functions/format_edge_case.cf.expected.json diff --git a/tests/acceptance/01_vars/02_functions/format_edge_case.cf b/tests/acceptance/01_vars/02_functions/format_edge_case.cf new file mode 100644 index 0000000000..818ae3168f --- /dev/null +++ b/tests/acceptance/01_vars/02_functions/format_edge_case.cf @@ -0,0 +1,28 @@ +# Test bug fix +# Former bug truncates strings greater than 4096 bytes +# There is no reason for format() to truncate strings + +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +########################################################## + +bundle agent test { + vars: + "str" string => format('%s', 'Hello, everyone! This is the LONGEST TEXT EVER! I was inspired by the various other longest texts ever on the internet, and I wanted to make my own. So here it is! This is going to be a WORLD RECORD! This is actually my third attempt at doing this. The first time, I didnt save it. The second time, the Neocities editor crashed. Now Im writing this in Notepad, then copying it into the Neocities editor instead of typing it directly in the Neocities editor to avoid crashing. It sucks that my past two attempts are gone now. Those actually got pretty long. Not the longest, but still pretty long. I hope this one wont get lost somehow. Anyways, lets talk about WAFFLES! I like waffles. Waffles are cool. Waffles is a funny word. Theres a Teen Titans Go episode called Waffles where the word Waffles is said a hundred-something times. Its pretty annoying. Theres also a Teen Titans Go episode about Pig Latin. Dont know what Pig Latin is? Its a language where you take all the consonants before the first vowel, move them to the end, and add -ay to the end. If the word begins with a vowel, you just add -way to the end. For example, Waffles becomes Afflesway. Ive been speaking Pig Latin fluently since the fourth grade, so it surprised me when I saw the episode for the first time. I speak Pig Latin with my sister sometimes. Its pretty fun. I like speaking it in public so that everyone around us gets confused. Thats never actually happened before, but if it ever does, twill be pretty funny. By the way, twill is a word I invented recently, and its a contraction of it will. I really hope it gains popularity in the near future, because twill is WAY more fun than saying itll. Itll is too boring. Nobody likes boring. This is nowhere near being the longest text ever, but eventually it will be! I might still be writing this a decade later, who knows? But right now, its not very long. But Ill just keep writing until it is the longest! Have you ever heard the song Dau Dau by Awesome Scampis? Its an amazing song. Look it up on YouTube! I play that song all the time around my sister! It drives her crazy, and I love it. Another way I like driving my sister crazy is by speaking my own made up language to her. She hates the languages I make! The only language that we both speak besides English is Pig Latin. I think you already knew that. Whatever. I think Im gonna go for now. Bye! Hi, Im back now. Im gonna contribute more to this soon-to-be giant wall of text. I just realised I have a giant stuffed frog on my bed. I forgot his name. Im pretty sure it was something stupid though. I think it was FROG in Morse Code or something. Morse Code is cool. I know a bit of it, but Im not very good at it. Im also not very good at French. I barely know anything in French, and my pronunciation probably sucks. But Im learning it, at least. Im also learning Esperanto. Its this language that was made up by some guy a long time ago to be the universal language. A lot of people speak it. I am such a language nerd. Half of this text is probably gonna be about languages. But hey, as long as its long! Ha, get it? As LONG as its LONG? Im so funny, right? No, Im not. I should probably get some sleep. Goodnight! Hello, Im back again. I basically have only two interests nowadays: languages and furries. What? Oh, sorry, I thought you knew I was a furry. Haha, oops. Anyway, yeah, Im a furry, but since Im a young furry, I cant really do as much as I would like to do in the fandom. When Im older, I would like to have a fursuit, go to furry conventions, all that stuff. But for now I can only dream of that. Sorry you had to deal with me talking about furries, but Im honestly very desperate for this to be the longest text ever. Last night I was watching nothing but fursuit unboxings. I think I need help. This one time, me and my mom were going to go to a furry Christmas party, but we didnt end up going because of the fact that there was alcohol on the premises, and that she didnt wanna have to be a mom dragging her son through a crowd of furries. Both of those reasons were understandable. Okay, hopefully I wont have to talk about furries anymore. I dont care if youre a furry reading this right now, I just dont wanna have to torture everyone else.'); + "list" slist => { $(str) }; + "list_str" string => format('%S', "list"); + "container" data => parsejson(' ["$(str)"] '); + "container_str" string => format('%S', "container"); +} + +bundle agent check { + methods: + "check" usebundle => dcs_check_state(test, + "$(this.promise_filename).expected.json", + $(this.promise_filename)); +} diff --git a/tests/acceptance/01_vars/02_functions/format_edge_case.cf.expected.json b/tests/acceptance/01_vars/02_functions/format_edge_case.cf.expected.json new file mode 100644 index 0000000000..85f6b5b648 --- /dev/null +++ b/tests/acceptance/01_vars/02_functions/format_edge_case.cf.expected.json @@ -0,0 +1,11 @@ +{ + "container": [ + "Hello, everyone! This is the LONGEST TEXT EVER! I was inspired by the various other longest texts ever on the internet, and I wanted to make my own. So here it is! This is going to be a WORLD RECORD! This is actually my third attempt at doing this. The first time, I didnt save it. The second time, the Neocities editor crashed. Now Im writing this in Notepad, then copying it into the Neocities editor instead of typing it directly in the Neocities editor to avoid crashing. It sucks that my past two attempts are gone now. Those actually got pretty long. Not the longest, but still pretty long. I hope this one wont get lost somehow. Anyways, lets talk about WAFFLES! I like waffles. Waffles are cool. Waffles is a funny word. Theres a Teen Titans Go episode called Waffles where the word Waffles is said a hundred-something times. Its pretty annoying. Theres also a Teen Titans Go episode about Pig Latin. Dont know what Pig Latin is? Its a language where you take all the consonants before the first vowel, move them to the end, and add -ay to the end. If the word begins with a vowel, you just add -way to the end. For example, Waffles becomes Afflesway. Ive been speaking Pig Latin fluently since the fourth grade, so it surprised me when I saw the episode for the first time. I speak Pig Latin with my sister sometimes. Its pretty fun. I like speaking it in public so that everyone around us gets confused. Thats never actually happened before, but if it ever does, twill be pretty funny. By the way, twill is a word I invented recently, and its a contraction of it will. I really hope it gains popularity in the near future, because twill is WAY more fun than saying itll. Itll is too boring. Nobody likes boring. This is nowhere near being the longest text ever, but eventually it will be! I might still be writing this a decade later, who knows? But right now, its not very long. But Ill just keep writing until it is the longest! Have you ever heard the song Dau Dau by Awesome Scampis? Its an amazing song. Look it up on YouTube! I play that song all the time around my sister! It drives her crazy, and I love it. Another way I like driving my sister crazy is by speaking my own made up language to her. She hates the languages I make! The only language that we both speak besides English is Pig Latin. I think you already knew that. Whatever. I think Im gonna go for now. Bye! Hi, Im back now. Im gonna contribute more to this soon-to-be giant wall of text. I just realised I have a giant stuffed frog on my bed. I forgot his name. Im pretty sure it was something stupid though. I think it was FROG in Morse Code or something. Morse Code is cool. I know a bit of it, but Im not very good at it. Im also not very good at French. I barely know anything in French, and my pronunciation probably sucks. But Im learning it, at least. Im also learning Esperanto. Its this language that was made up by some guy a long time ago to be the universal language. A lot of people speak it. I am such a language nerd. Half of this text is probably gonna be about languages. But hey, as long as its long! Ha, get it? As LONG as its LONG? Im so funny, right? No, Im not. I should probably get some sleep. Goodnight! Hello, Im back again. I basically have only two interests nowadays: languages and furries. What? Oh, sorry, I thought you knew I was a furry. Haha, oops. Anyway, yeah, Im a furry, but since Im a young furry, I cant really do as much as I would like to do in the fandom. When Im older, I would like to have a fursuit, go to furry conventions, all that stuff. But for now I can only dream of that. Sorry you had to deal with me talking about furries, but Im honestly very desperate for this to be the longest text ever. Last night I was watching nothing but fursuit unboxings. I think I need help. This one time, me and my mom were going to go to a furry Christmas party, but we didnt end up going because of the fact that there was alcohol on the premises, and that she didnt wanna have to be a mom dragging her son through a crowd of furries. Both of those reasons were understandable. Okay, hopefully I wont have to talk about furries anymore. I dont care if youre a furry reading this right now, I just dont wanna have to torture everyone else." + ], + "container_str": "[\"Hello, everyone! This is the LONGEST TEXT EVER! I was inspired by the various other longest texts ever on the internet, and I wanted to make my own. So here it is! This is going to be a WORLD RECORD! This is actually my third attempt at doing this. The first time, I didnt save it. The second time, the Neocities editor crashed. Now Im writing this in Notepad, then copying it into the Neocities editor instead of typing it directly in the Neocities editor to avoid crashing. It sucks that my past two attempts are gone now. Those actually got pretty long. Not the longest, but still pretty long. I hope this one wont get lost somehow. Anyways, lets talk about WAFFLES! I like waffles. Waffles are cool. Waffles is a funny word. Theres a Teen Titans Go episode called Waffles where the word Waffles is said a hundred-something times. Its pretty annoying. Theres also a Teen Titans Go episode about Pig Latin. Dont know what Pig Latin is? Its a language where you take all the consonants before the first vowel, move them to the end, and add -ay to the end. If the word begins with a vowel, you just add -way to the end. For example, Waffles becomes Afflesway. Ive been speaking Pig Latin fluently since the fourth grade, so it surprised me when I saw the episode for the first time. I speak Pig Latin with my sister sometimes. Its pretty fun. I like speaking it in public so that everyone around us gets confused. Thats never actually happened before, but if it ever does, twill be pretty funny. By the way, twill is a word I invented recently, and its a contraction of it will. I really hope it gains popularity in the near future, because twill is WAY more fun than saying itll. Itll is too boring. Nobody likes boring. This is nowhere near being the longest text ever, but eventually it will be! I might still be writing this a decade later, who knows? But right now, its not very long. But Ill just keep writing until it is the longest! Have you ever heard the song Dau Dau by Awesome Scampis? Its an amazing song. Look it up on YouTube! I play that song all the time around my sister! It drives her crazy, and I love it. Another way I like driving my sister crazy is by speaking my own made up language to her. She hates the languages I make! The only language that we both speak besides English is Pig Latin. I think you already knew that. Whatever. I think Im gonna go for now. Bye! Hi, Im back now. Im gonna contribute more to this soon-to-be giant wall of text. I just realised I have a giant stuffed frog on my bed. I forgot his name. Im pretty sure it was something stupid though. I think it was FROG in Morse Code or something. Morse Code is cool. I know a bit of it, but Im not very good at it. Im also not very good at French. I barely know anything in French, and my pronunciation probably sucks. But Im learning it, at least. Im also learning Esperanto. Its this language that was made up by some guy a long time ago to be the universal language. A lot of people speak it. I am such a language nerd. Half of this text is probably gonna be about languages. But hey, as long as its long! Ha, get it? As LONG as its LONG? Im so funny, right? No, Im not. I should probably get some sleep. Goodnight! Hello, Im back again. I basically have only two interests nowadays: languages and furries. What? Oh, sorry, I thought you knew I was a furry. Haha, oops. Anyway, yeah, Im a furry, but since Im a young furry, I cant really do as much as I would like to do in the fandom. When Im older, I would like to have a fursuit, go to furry conventions, all that stuff. But for now I can only dream of that. Sorry you had to deal with me talking about furries, but Im honestly very desperate for this to be the longest text ever. Last night I was watching nothing but fursuit unboxings. I think I need help. This one time, me and my mom were going to go to a furry Christmas party, but we didnt end up going because of the fact that there was alcohol on the premises, and that she didnt wanna have to be a mom dragging her son through a crowd of furries. Both of those reasons were understandable. Okay, hopefully I wont have to talk about furries anymore. I dont care if youre a furry reading this right now, I just dont wanna have to torture everyone else.\"]", + "list": [ + "Hello, everyone! This is the LONGEST TEXT EVER! I was inspired by the various other longest texts ever on the internet, and I wanted to make my own. So here it is! This is going to be a WORLD RECORD! This is actually my third attempt at doing this. The first time, I didnt save it. The second time, the Neocities editor crashed. Now Im writing this in Notepad, then copying it into the Neocities editor instead of typing it directly in the Neocities editor to avoid crashing. It sucks that my past two attempts are gone now. Those actually got pretty long. Not the longest, but still pretty long. I hope this one wont get lost somehow. Anyways, lets talk about WAFFLES! I like waffles. Waffles are cool. Waffles is a funny word. Theres a Teen Titans Go episode called Waffles where the word Waffles is said a hundred-something times. Its pretty annoying. Theres also a Teen Titans Go episode about Pig Latin. Dont know what Pig Latin is? Its a language where you take all the consonants before the first vowel, move them to the end, and add -ay to the end. If the word begins with a vowel, you just add -way to the end. For example, Waffles becomes Afflesway. Ive been speaking Pig Latin fluently since the fourth grade, so it surprised me when I saw the episode for the first time. I speak Pig Latin with my sister sometimes. Its pretty fun. I like speaking it in public so that everyone around us gets confused. Thats never actually happened before, but if it ever does, twill be pretty funny. By the way, twill is a word I invented recently, and its a contraction of it will. I really hope it gains popularity in the near future, because twill is WAY more fun than saying itll. Itll is too boring. Nobody likes boring. This is nowhere near being the longest text ever, but eventually it will be! I might still be writing this a decade later, who knows? But right now, its not very long. But Ill just keep writing until it is the longest! Have you ever heard the song Dau Dau by Awesome Scampis? Its an amazing song. Look it up on YouTube! I play that song all the time around my sister! It drives her crazy, and I love it. Another way I like driving my sister crazy is by speaking my own made up language to her. She hates the languages I make! The only language that we both speak besides English is Pig Latin. I think you already knew that. Whatever. I think Im gonna go for now. Bye! Hi, Im back now. Im gonna contribute more to this soon-to-be giant wall of text. I just realised I have a giant stuffed frog on my bed. I forgot his name. Im pretty sure it was something stupid though. I think it was FROG in Morse Code or something. Morse Code is cool. I know a bit of it, but Im not very good at it. Im also not very good at French. I barely know anything in French, and my pronunciation probably sucks. But Im learning it, at least. Im also learning Esperanto. Its this language that was made up by some guy a long time ago to be the universal language. A lot of people speak it. I am such a language nerd. Half of this text is probably gonna be about languages. But hey, as long as its long! Ha, get it? As LONG as its LONG? Im so funny, right? No, Im not. I should probably get some sleep. Goodnight! Hello, Im back again. I basically have only two interests nowadays: languages and furries. What? Oh, sorry, I thought you knew I was a furry. Haha, oops. Anyway, yeah, Im a furry, but since Im a young furry, I cant really do as much as I would like to do in the fandom. When Im older, I would like to have a fursuit, go to furry conventions, all that stuff. But for now I can only dream of that. Sorry you had to deal with me talking about furries, but Im honestly very desperate for this to be the longest text ever. Last night I was watching nothing but fursuit unboxings. I think I need help. This one time, me and my mom were going to go to a furry Christmas party, but we didnt end up going because of the fact that there was alcohol on the premises, and that she didnt wanna have to be a mom dragging her son through a crowd of furries. Both of those reasons were understandable. Okay, hopefully I wont have to talk about furries anymore. I dont care if youre a furry reading this right now, I just dont wanna have to torture everyone else." + ], + "list_str": "{ \"Hello, everyone! This is the LONGEST TEXT EVER! I was inspired by the various other longest texts ever on the internet, and I wanted to make my own. So here it is! This is going to be a WORLD RECORD! This is actually my third attempt at doing this. The first time, I didnt save it. The second time, the Neocities editor crashed. Now Im writing this in Notepad, then copying it into the Neocities editor instead of typing it directly in the Neocities editor to avoid crashing. It sucks that my past two attempts are gone now. Those actually got pretty long. Not the longest, but still pretty long. I hope this one wont get lost somehow. Anyways, lets talk about WAFFLES! I like waffles. Waffles are cool. Waffles is a funny word. Theres a Teen Titans Go episode called Waffles where the word Waffles is said a hundred-something times. Its pretty annoying. Theres also a Teen Titans Go episode about Pig Latin. Dont know what Pig Latin is? Its a language where you take all the consonants before the first vowel, move them to the end, and add -ay to the end. If the word begins with a vowel, you just add -way to the end. For example, Waffles becomes Afflesway. Ive been speaking Pig Latin fluently since the fourth grade, so it surprised me when I saw the episode for the first time. I speak Pig Latin with my sister sometimes. Its pretty fun. I like speaking it in public so that everyone around us gets confused. Thats never actually happened before, but if it ever does, twill be pretty funny. By the way, twill is a word I invented recently, and its a contraction of it will. I really hope it gains popularity in the near future, because twill is WAY more fun than saying itll. Itll is too boring. Nobody likes boring. This is nowhere near being the longest text ever, but eventually it will be! I might still be writing this a decade later, who knows? But right now, its not very long. But Ill just keep writing until it is the longest! Have you ever heard the song Dau Dau by Awesome Scampis? Its an amazing song. Look it up on YouTube! I play that song all the time around my sister! It drives her crazy, and I love it. Another way I like driving my sister crazy is by speaking my own made up language to her. She hates the languages I make! The only language that we both speak besides English is Pig Latin. I think you already knew that. Whatever. I think Im gonna go for now. Bye! Hi, Im back now. Im gonna contribute more to this soon-to-be giant wall of text. I just realised I have a giant stuffed frog on my bed. I forgot his name. Im pretty sure it was something stupid though. I think it was FROG in Morse Code or something. Morse Code is cool. I know a bit of it, but Im not very good at it. Im also not very good at French. I barely know anything in French, and my pronunciation probably sucks. But Im learning it, at least. Im also learning Esperanto. Its this language that was made up by some guy a long time ago to be the universal language. A lot of people speak it. I am such a language nerd. Half of this text is probably gonna be about languages. But hey, as long as its long! Ha, get it? As LONG as its LONG? Im so funny, right? No, Im not. I should probably get some sleep. Goodnight! Hello, Im back again. I basically have only two interests nowadays: languages and furries. What? Oh, sorry, I thought you knew I was a furry. Haha, oops. Anyway, yeah, Im a furry, but since Im a young furry, I cant really do as much as I would like to do in the fandom. When Im older, I would like to have a fursuit, go to furry conventions, all that stuff. But for now I can only dream of that. Sorry you had to deal with me talking about furries, but Im honestly very desperate for this to be the longest text ever. Last night I was watching nothing but fursuit unboxings. I think I need help. This one time, me and my mom were going to go to a furry Christmas party, but we didnt end up going because of the fact that there was alcohol on the premises, and that she didnt wanna have to be a mom dragging her son through a crowd of furries. Both of those reasons were understandable. Okay, hopefully I wont have to talk about furries anymore. I dont care if youre a furry reading this right now, I just dont wanna have to torture everyone else.\" }", + "str": "Hello, everyone! This is the LONGEST TEXT EVER! I was inspired by the various other longest texts ever on the internet, and I wanted to make my own. So here it is! This is going to be a WORLD RECORD! This is actually my third attempt at doing this. The first time, I didnt save it. The second time, the Neocities editor crashed. Now Im writing this in Notepad, then copying it into the Neocities editor instead of typing it directly in the Neocities editor to avoid crashing. It sucks that my past two attempts are gone now. Those actually got pretty long. Not the longest, but still pretty long. I hope this one wont get lost somehow. Anyways, lets talk about WAFFLES! I like waffles. Waffles are cool. Waffles is a funny word. Theres a Teen Titans Go episode called Waffles where the word Waffles is said a hundred-something times. Its pretty annoying. Theres also a Teen Titans Go episode about Pig Latin. Dont know what Pig Latin is? Its a language where you take all the consonants before the first vowel, move them to the end, and add -ay to the end. If the word begins with a vowel, you just add -way to the end. For example, Waffles becomes Afflesway. Ive been speaking Pig Latin fluently since the fourth grade, so it surprised me when I saw the episode for the first time. I speak Pig Latin with my sister sometimes. Its pretty fun. I like speaking it in public so that everyone around us gets confused. Thats never actually happened before, but if it ever does, twill be pretty funny. By the way, twill is a word I invented recently, and its a contraction of it will. I really hope it gains popularity in the near future, because twill is WAY more fun than saying itll. Itll is too boring. Nobody likes boring. This is nowhere near being the longest text ever, but eventually it will be! I might still be writing this a decade later, who knows? But right now, its not very long. But Ill just keep writing until it is the longest! Have you ever heard the song Dau Dau by Awesome Scampis? Its an amazing song. Look it up on YouTube! I play that song all the time around my sister! It drives her crazy, and I love it. Another way I like driving my sister crazy is by speaking my own made up language to her. She hates the languages I make! The only language that we both speak besides English is Pig Latin. I think you already knew that. Whatever. I think Im gonna go for now. Bye! Hi, Im back now. Im gonna contribute more to this soon-to-be giant wall of text. I just realised I have a giant stuffed frog on my bed. I forgot his name. Im pretty sure it was something stupid though. I think it was FROG in Morse Code or something. Morse Code is cool. I know a bit of it, but Im not very good at it. Im also not very good at French. I barely know anything in French, and my pronunciation probably sucks. But Im learning it, at least. Im also learning Esperanto. Its this language that was made up by some guy a long time ago to be the universal language. A lot of people speak it. I am such a language nerd. Half of this text is probably gonna be about languages. But hey, as long as its long! Ha, get it? As LONG as its LONG? Im so funny, right? No, Im not. I should probably get some sleep. Goodnight! Hello, Im back again. I basically have only two interests nowadays: languages and furries. What? Oh, sorry, I thought you knew I was a furry. Haha, oops. Anyway, yeah, Im a furry, but since Im a young furry, I cant really do as much as I would like to do in the fandom. When Im older, I would like to have a fursuit, go to furry conventions, all that stuff. But for now I can only dream of that. Sorry you had to deal with me talking about furries, but Im honestly very desperate for this to be the longest text ever. Last night I was watching nothing but fursuit unboxings. I think I need help. This one time, me and my mom were going to go to a furry Christmas party, but we didnt end up going because of the fact that there was alcohol on the premises, and that she didnt wanna have to be a mom dragging her son through a crowd of furries. Both of those reasons were understandable. Okay, hopefully I wont have to talk about furries anymore. I dont care if youre a furry reading this right now, I just dont wanna have to torture everyone else." +} From fe885725043d261d4d8144d818374cd8e84f8baa Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Mon, 14 Dec 2020 16:17:11 +0100 Subject: [PATCH 181/333] Policy function format() no longer truncates strings lager than 4KiB Changelog: Title Ticket: CFE-2686 Signed-off-by: Lars Erik Wik (cherry picked from commit 11ec249cf963e95bdb28f439020f899695e333d6) --- libpromises/evalfunction.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/libpromises/evalfunction.c b/libpromises/evalfunction.c index c37f0bb960..462ff39d0b 100644 --- a/libpromises/evalfunction.c +++ b/libpromises/evalfunction.c @@ -5212,9 +5212,7 @@ static FnCallResult FnCallFormat(EvalContext *ctx, ARG_UNUSED const Policy *poli } else if (strrchr(format_piece, 's')) { - snprintf(piece, CF_BUFSIZE, format_piece, data); - BufferAppend(buf, piece, strlen(piece)); - // CfOut(OUTPUT_LEVEL_INFORM, "", "format: appending string format piece = '%s' with data '%s'", format_piece, data); + BufferAppendF(buf, format_piece, data); } else if (strrchr(format_piece, 'S')) { @@ -5243,9 +5241,8 @@ static FnCallResult FnCallFormat(EvalContext *ctx, ARG_UNUSED const Policy *poli { Writer *w = StringWriter(); JsonWriteCompact(w, value); - snprintf(piece, CF_BUFSIZE, format_rewrite, StringWriterData(w)); + BufferAppendF(buf, format_rewrite, StringWriterData(w)); WriterClose(w); - BufferAppend(buf, piece, strlen(piece)); } else // it might be a list reference { @@ -5268,9 +5265,8 @@ static FnCallResult FnCallFormat(EvalContext *ctx, ARG_UNUSED const Policy *poli } WriterWrite(w, " }"); - snprintf(piece, CF_BUFSIZE, format_rewrite, StringWriterData(w)); + BufferAppendF(buf, format_rewrite, StringWriterData(w)); WriterClose(w); - BufferAppend(buf, piece, strlen(piece)); } else // whatever this is, it's not a list reference or a data container { From 3de285ba9c67e2b7058b32f7eadf7f78b88f8738 Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Mon, 14 Dec 2020 18:45:57 +0100 Subject: [PATCH 182/333] Removed commented out code Ticket: CFE-2686 Signed-off-by: Lars Erik Wik (cherry picked from commit bf079328cfec658899d58a657609125039cef261) --- libpromises/evalfunction.c | 6 ------ 1 file changed, 6 deletions(-) diff --git a/libpromises/evalfunction.c b/libpromises/evalfunction.c index 462ff39d0b..aa2f59af12 100644 --- a/libpromises/evalfunction.c +++ b/libpromises/evalfunction.c @@ -5171,8 +5171,6 @@ static FnCallResult FnCallFormat(EvalContext *ctx, ARG_UNUSED const Policy *poli char piece[CF_BUFSIZE]; memset(piece, 0, CF_BUFSIZE); - // CfOut(OUTPUT_LEVEL_INFORM, "", "format: processing format piece = '%s' with data '%s'", format_piece, percent ? "%" : data); - const char bad_modifiers[] = "hLqjzt"; const size_t length = strlen(bad_modifiers); for (int b = 0; b < length; b++) @@ -5194,13 +5192,11 @@ static FnCallResult FnCallFormat(EvalContext *ctx, ARG_UNUSED const Policy *poli sscanf(data, "%ld%s", &x, piece); // we don't care about the remainder and will overwrite it snprintf(piece, CF_BUFSIZE, format_piece, x); BufferAppend(buf, piece, strlen(piece)); - // CfOut(OUTPUT_LEVEL_INFORM, "", "format: appending int format piece = '%s' with data '%s'", format_piece, data); } else if (percent) { // "%%" -> "%" BufferAppend(buf, "%", 1); - // CfOut(OUTPUT_LEVEL_INFORM, "", "format: appending int format piece = '%s' with data '%s'", format_piece, data); } else if (strrchr(format_piece, 'f')) { @@ -5208,7 +5204,6 @@ static FnCallResult FnCallFormat(EvalContext *ctx, ARG_UNUSED const Policy *poli sscanf(data, "%lf%s", &x, piece); // we don't care about the remainder and will overwrite it snprintf(piece, CF_BUFSIZE, format_piece, x); BufferAppend(buf, piece, strlen(piece)); - // CfOut(OUTPUT_LEVEL_INFORM, "", "format: appending float format piece = '%s' with data '%s'", format_piece, data); } else if (strrchr(format_piece, 's')) { @@ -5282,7 +5277,6 @@ static FnCallResult FnCallFormat(EvalContext *ctx, ARG_UNUSED const Policy *poli { char error[] = "(unhandled format)"; BufferAppend(buf, error, strlen(error)); - // CfOut(OUTPUT_LEVEL_INFORM, "", "format: error appending unhandled format piece = '%s' with data '%s'", format_piece, data); } } else From e5786021748c8802729eddfc0700fe9c53da321d Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Tue, 15 Dec 2020 09:22:57 +0100 Subject: [PATCH 183/333] Added explicit comparisons Ticket: CFE-2686 Signed-off-by: Lars Erik Wik (cherry picked from commit fe22f4fa564366e0400c8105c795423637902363) --- libpromises/evalfunction.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/libpromises/evalfunction.c b/libpromises/evalfunction.c index aa2f59af12..f13f52d339 100644 --- a/libpromises/evalfunction.c +++ b/libpromises/evalfunction.c @@ -5118,14 +5118,14 @@ static FnCallResult FnCallFormat(EvalContext *ctx, ARG_UNUSED const Policy *poli } } - if (!finalargs) + if (finalargs == NULL) { return FnFailure(); } char *format = RlistScalarValue(finalargs); - if (!format) + if (format == NULL) { return FnFailure(); } @@ -5136,12 +5136,12 @@ static FnCallResult FnCallFormat(EvalContext *ctx, ARG_UNUSED const Policy *poli char check_buffer[CF_BUFSIZE]; Buffer *buf = BufferNew(); - if (check) + if (check != NULL) { BufferAppend(buf, format, check - format); Seq *s; - while (check && + while (check != NULL && (s = StringMatchCaptures("^(%%|%[^diouxXeEfFgGaAcsCSpnm%]*?[diouxXeEfFgGaAcsCSpnm])([^%]*)(.*)$", check, false))) { { @@ -5155,7 +5155,7 @@ static FnCallResult FnCallFormat(EvalContext *ctx, ARG_UNUSED const Policy *poli { // "%%" in format string } - else if (rp) + else if (rp != NULL) { data = RlistScalarValue(rp); rp = rp->next; @@ -5186,7 +5186,7 @@ static FnCallResult FnCallFormat(EvalContext *ctx, ARG_UNUSED const Policy *poli } } - if (strrchr(format_piece, 'd') || strrchr(format_piece, 'o') || strrchr(format_piece, 'x')) + if (strrchr(format_piece, 'd') != NULL || strrchr(format_piece, 'o') != NULL || strrchr(format_piece, 'x') != NULL) { long x = 0; sscanf(data, "%ld%s", &x, piece); // we don't care about the remainder and will overwrite it @@ -5198,18 +5198,18 @@ static FnCallResult FnCallFormat(EvalContext *ctx, ARG_UNUSED const Policy *poli // "%%" -> "%" BufferAppend(buf, "%", 1); } - else if (strrchr(format_piece, 'f')) + else if (strrchr(format_piece, 'f') != NULL) { double x = 0; sscanf(data, "%lf%s", &x, piece); // we don't care about the remainder and will overwrite it snprintf(piece, CF_BUFSIZE, format_piece, x); BufferAppend(buf, piece, strlen(piece)); } - else if (strrchr(format_piece, 's')) + else if (strrchr(format_piece, 's') != NULL) { BufferAppendF(buf, format_piece, data); } - else if (strrchr(format_piece, 'S')) + else if (strrchr(format_piece, 'S') != NULL) { char *found_format_spec = NULL; char format_rewrite[CF_BUFSIZE]; @@ -5217,7 +5217,7 @@ static FnCallResult FnCallFormat(EvalContext *ctx, ARG_UNUSED const Policy *poli strlcpy(format_rewrite, format_piece, CF_BUFSIZE); found_format_spec = strrchr(format_rewrite, 'S'); - if (found_format_spec) + if (found_format_spec != NULL) { *found_format_spec = 's'; } From ac5cbf68852627e9bba7fe26e7872d6f2bbe1bff Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Wed, 16 Dec 2020 16:05:19 +0100 Subject: [PATCH 184/333] Removed snprintf and assigned a string literal instead Ticket: CFE-2686 Signed-off-by: Lars Erik Wik (cherry picked from commit 0043008cae7a3078b14da4aa243ffe5b127f4e60) --- libpromises/evalfunction.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/libpromises/evalfunction.c b/libpromises/evalfunction.c index f13f52d339..55e1daaaae 100644 --- a/libpromises/evalfunction.c +++ b/libpromises/evalfunction.c @@ -5104,9 +5104,7 @@ static FnCallResult FnCallSort(EvalContext *ctx, ARG_UNUSED const Policy *policy static FnCallResult FnCallFormat(EvalContext *ctx, ARG_UNUSED const Policy *policy, const FnCall *fp, const Rlist *finalargs) { - char id[CF_BUFSIZE]; - - snprintf(id, CF_BUFSIZE, "built-in FnCall %s-arg", fp->name); + const char *const id = "built-in FnCall format-arg"; /* We need to check all the arguments, ArgTemplate does not check varadic functions */ for (const Rlist *arg = finalargs; arg; arg = arg->next) From 627da840f94c70ba8c97356df6e0d52f2f8ac45b Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Thu, 17 Dec 2020 13:49:41 +0100 Subject: [PATCH 185/333] Fixed buffer overflow vulnerabillity in policy function format() Fixed vulnerabillity where malformed input could trigger buffer overflow in policy function format. Changelog: Body Ticket: CFE-3525 Signed-off-by: Lars Erik Wik (cherry picked from commit 3fb57366017715f0b91031909350fed9b07d3224) --- libpromises/evalfunction.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libpromises/evalfunction.c b/libpromises/evalfunction.c index 55e1daaaae..57546eb014 100644 --- a/libpromises/evalfunction.c +++ b/libpromises/evalfunction.c @@ -5187,7 +5187,7 @@ static FnCallResult FnCallFormat(EvalContext *ctx, ARG_UNUSED const Policy *poli if (strrchr(format_piece, 'd') != NULL || strrchr(format_piece, 'o') != NULL || strrchr(format_piece, 'x') != NULL) { long x = 0; - sscanf(data, "%ld%s", &x, piece); // we don't care about the remainder and will overwrite it + sscanf(data, "%ld", &x); snprintf(piece, CF_BUFSIZE, format_piece, x); BufferAppend(buf, piece, strlen(piece)); } @@ -5199,7 +5199,7 @@ static FnCallResult FnCallFormat(EvalContext *ctx, ARG_UNUSED const Policy *poli else if (strrchr(format_piece, 'f') != NULL) { double x = 0; - sscanf(data, "%lf%s", &x, piece); // we don't care about the remainder and will overwrite it + sscanf(data, "%lf", &x); snprintf(piece, CF_BUFSIZE, format_piece, x); BufferAppend(buf, piece, strlen(piece)); } From bb975baa4064932681f37c5168bc1cf7ee675e69 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Thu, 14 Jan 2021 12:10:40 -0600 Subject: [PATCH 186/333] Clarified description of depth_search xdev attribute behavior Ticket: CFE-3541 Changelog: None (cherry picked from commit 7be9b3bd82ac9ebc06b375017d12bc7bebda25f5) --- libpromises/mod_files.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libpromises/mod_files.c b/libpromises/mod_files.c index 9df6584a07..2c4ff30a6a 100644 --- a/libpromises/mod_files.c +++ b/libpromises/mod_files.c @@ -197,7 +197,7 @@ static const ConstraintSyntax depth_search_constraints[] = ConstraintSyntaxNewStringList("include_dirs", ".*", "List of regexes of directory names to include in depth search", SYNTAX_STATUS_NORMAL), ConstraintSyntaxNewBool("rmdeadlinks", "true/false remove links that point to nowhere. Default value: false", SYNTAX_STATUS_NORMAL), ConstraintSyntaxNewBool("traverse_links", "true/false traverse symbolic links to directories. Default value: false", SYNTAX_STATUS_NORMAL), - ConstraintSyntaxNewBool("xdev", "true/false exclude directories that are on different devices. Default value: false", SYNTAX_STATUS_NORMAL), + ConstraintSyntaxNewBool("xdev", "When true files and directories on different devices from the promiser will be excluded from depth_search results. Default value: false", SYNTAX_STATUS_NORMAL), ConstraintSyntaxNewNull() }; From 2f1a55677926da0aa30fc7a73bdfef0034864a7f Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Thu, 21 Jan 2021 14:59:19 -0600 Subject: [PATCH 187/333] Added example illustrating inline yaml Ticket: CFE-3551 Changelog: None (cherry picked from commit 06da60ff2edde37b219b3394913dbaf61d06e8cc) --- examples/inline-yaml.cf | 58 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 examples/inline-yaml.cf diff --git a/examples/inline-yaml.cf b/examples/inline-yaml.cf new file mode 100644 index 0000000000..231b3fe890 --- /dev/null +++ b/examples/inline-yaml.cf @@ -0,0 +1,58 @@ +#+begin_src cfengine3 +bundle agent example_inline_yaml +# @brief Example illustrating inline yaml +{ + vars: + # YAML requires "---" header (followed by newline) + # NOTE \n is not interpreted as a newline, instead use $(const.n) + + "yaml_multi_line" data => '--- +- "CFEngine Champions": + - Name: "Aleksey Tsalolikhin" + Year: 2011 + - Name: "Ted Zlatanov" + Year : 2013'; + + + "yaml_single_line" data => '---$(const.n)- key1: value1$(const.n)- key2: value2'; + + reports: + "Data container defined from yaml_multi_line: $(with)" + with => storejson( @(yaml_multi_line) ); + + "Data container defined from yaml_single_line: $(with)" + with => storejson( @(yaml_single_line) ); +} +bundle agent __main__ +{ + methods: + "example_inline_yaml"; +} +#+end_src +############################################################################### +#+begin_src example_output +#@ ``` +#@ R: Data container defined from yaml_multi_line: [ +#@ { +#@ "CFEngine Champions": [ +#@ { +#@ "Name": "Aleksey Tsalolikhin", +#@ "Year": 2011 +#@ }, +#@ { +#@ "Name": "Ted Zlatanov", +#@ "Year": 2013 +#@ } +#@ ] +#@ } +#@ ] +#@ R: Data container defined from yaml_single_line: [ +#@ { +#@ "key1": "value1" +#@ }, +#@ { +#@ "key2": "value2" +#@ } +#@ ] +#@ ``` +#+end_src From cb94ce2d7d0133915ddeeab891611d7246d82442 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Mon, 1 Feb 2021 15:18:23 -0600 Subject: [PATCH 188/333] Stopped emitting warning and recording result when observing new SETGID files Ticket: ENT-6750 Changelog: Title (cherry picked from commit d34fd24dde3d83864bd5938813a4eb2b7beb0631) --- cf-agent/verify_files_utils.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/cf-agent/verify_files_utils.c b/cf-agent/verify_files_utils.c index b544a6a7cd..cc90db0c94 100644 --- a/cf-agent/verify_files_utils.c +++ b/cf-agent/verify_files_utils.c @@ -3358,8 +3358,7 @@ static PromiseResult VerifySetUidGid(EvalContext *ctx, const char *file, const s { if (amroot) { - cfPS(ctx, LOG_LEVEL_WARNING, PROMISE_RESULT_WARN, pp, attr, "NEW SETGID root PROGRAM '%s'", file); - result = PromiseResultUpdate(result, PROMISE_RESULT_WARN); + Log(LOG_LEVEL_NOTICE, "NEW SETGID root PROGRAM '%s' ", file); } PrependItem(&VSETXIDLIST, file, NULL); From 459ae81943b9ac22a96fb77d30302b291cd957aa Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Fri, 29 Jan 2021 13:56:59 -0600 Subject: [PATCH 189/333] Added inline json example Ticket: CFE-3551 Changelog: None (cherry picked from commit 1b58451af4f3fc01eb99dda7777204b7150495db) --- examples/inline-json.cf | 59 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 examples/inline-json.cf diff --git a/examples/inline-json.cf b/examples/inline-json.cf new file mode 100644 index 0000000000..0d633592be --- /dev/null +++ b/examples/inline-json.cf @@ -0,0 +1,59 @@ +#+begin_src cfengine3 +bundle agent example_inline_json +# @brief Example illustrating inline json +{ + vars: + "json_multi_line" data => '{ + "CFEngine Champions": [ + { + "Name": "Aleksey Tsalolikhin", + "Year": "2011" + }, + { + "Name": "Ted Zlatanov", + "Year": "2013" + } + ] +}'; + + + "json_single_line" data => '[{"key1":"value1"},{"key2":"value2"}]'; + + reports: + "Data container defined from json_multi_line: $(with)" + with => storejson( @(json_multi_line) ); + + "Data container defined from json_single_line: $(with)" + with => storejson( @(json_single_line) ); +} +bundle agent __main__ +{ + methods: + "example_inline_json"; +} +#+end_src +############################################################################### +#+begin_src example_output +#@ ``` +#@ R: Data container defined from json_multi_line: { +#@ "CFEngine Champions": [ +#@ { +#@ "Name": "Aleksey Tsalolikhin", +#@ "Year": "2011" +#@ }, +#@ { +#@ "Name": "Ted Zlatanov", +#@ "Year": "2013" +#@ } +#@ ] +#@ } +#@ R: Data container defined from json_single_line: [ +#@ { +#@ "key1": "value1" +#@ }, +#@ { +#@ "key2": "value2" +#@ } +#@ ] +#@ ``` +#+end_src From 585b819383d2603141c9ba5a03e7249f9c5739b8 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Mon, 8 Feb 2021 13:58:31 +0100 Subject: [PATCH 190/333] Travis: Disabled MacOS Failing Mac builds have caused master to be red for some days. This seems to be because Travis changed the installed packages on their default image. Since we are already testing MacOS on GitHub Actions, and they are much faster and more reliable, I don't see a point in fixing the travis build, so I disabled it. Had to resolve conflicts since this was already an allowed failure on 3.15.x (but not on master). Ticket: None Changelog: None Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 1539824f12107a18652c4ce8a995f5fb269b6c7f) --- .travis.yml | 7 ------- 1 file changed, 7 deletions(-) diff --git a/.travis.yml b/.travis.yml index 1d21b70cb4..ada06b4a24 100644 --- a/.travis.yml +++ b/.travis.yml @@ -49,13 +49,6 @@ matrix: - os: linux sudo: required env: JOB_TYPE=serverd_multi_versions COVERAGE=no - - os: osx - env: JOB_TYPE=compile_and_unit_test COVERAGE=no - sudo: false - allow_failures: - - os: osx - env: JOB_TYPE=compile_and_unit_test COVERAGE=no - sudo: false before_install: - chmod ug+x ./travis-scripts/* From 9a4a6bc873d5694fc06e94100138b7ce80737317 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 19 Feb 2021 10:44:16 +0100 Subject: [PATCH 191/333] SELinux: Allow cf-serverd getattr on cfengine_log_t:lnk_file Discovered while working on ENT-6735. (cherry picked from commit 01b3bf0cf62ba7c1857ee84af63dd4391328f366) --- misc/selinux/cfengine-enterprise.te | 1 + 1 file changed, 1 insertion(+) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 5072ab1cfc..f10cb7b842 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -289,6 +289,7 @@ allow cfengine_serverd_t ssh_port_t:tcp_socket name_connect; allow cfengine_serverd_t cfengine_execd_exec_t:file getattr; allow cfengine_serverd_t cfengine_monitord_exec_t:file getattr; allow cfengine_serverd_t cfengine_hub_exec_t:file getattr; +allow cfengine_serverd_t cfengine_log_t:lnk_file getattr; allow cfengine_serverd_t crontab_exec_t:file getattr; allow cfengine_serverd_t dmidecode_exec_t:file getattr; From 686f34abde1622c526d1ef9506a9be553ce2e407 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Tue, 2 Mar 2021 11:01:30 -0600 Subject: [PATCH 192/333] Clarified error log message about untrusted state directory not being private Ticket: CFE-3599 Changelog: Title (cherry picked from commit 219640670c9e7798b3693c7c0329d58607534f2b) --- libpromises/generic_agent.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libpromises/generic_agent.c b/libpromises/generic_agent.c index 1cab6af0ed..7e6862ccbf 100644 --- a/libpromises/generic_agent.c +++ b/libpromises/generic_agent.c @@ -1667,7 +1667,7 @@ static void CheckWorkingDirectories(EvalContext *ctx) #ifndef __MINGW32__ if (statbuf.st_mode & 022) { - Log(LOG_LEVEL_ERR, "UNTRUSTED: State directory %s (mode %jo) was not private!", workdir, + Log(LOG_LEVEL_ERR, "UNTRUSTED: State directory %s (mode %jo) was not private, world and/or group writeable!", statedir, (uintmax_t)(statbuf.st_mode & 0777)); } #endif /* !__MINGW32__ */ From 9efb0fc30f9450365eacfb5f19937e91f88e82cc Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 4 Mar 2021 10:49:08 +0100 Subject: [PATCH 193/333] Fix iteration over the 'extra_hashes' map in PolicyMerge 'extra_hashes' is a StringMap which means the actual hash map is in its 'impl' field. (cherry picked from commit ee881fce018476f91914ca0d643fb00399f585d7) --- libpromises/policy.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libpromises/policy.c b/libpromises/policy.c index 885e8c89b3..3b79183a04 100644 --- a/libpromises/policy.c +++ b/libpromises/policy.c @@ -428,7 +428,7 @@ Policy *PolicyMerge(Policy *a, Policy *b) if (extra_hashes != NULL) { - MapIterator it = MapIteratorInit((Map*) extra_hashes); + MapIterator it = MapIteratorInit(extra_hashes->impl); MapKeyValue *item; while ((item = MapIteratorNext(&it)) != NULL) { From 3104e1063abb440506e57eb82f1ff1e0e1897c88 Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Sat, 27 Mar 2021 23:57:02 +0100 Subject: [PATCH 194/333] ENT-6939 disable defaults.cf on all platforms (cherry picked from commit ad6c9b169545e67ccfb3638abc52d67242e49556) --- .../00_basics/ifelapsed_and_expireafter/timed/defaults.cf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/defaults.cf b/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/defaults.cf index c03f72b106..ae7c717b89 100644 --- a/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/defaults.cf +++ b/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/defaults.cf @@ -31,8 +31,8 @@ bundle agent init # Backgrounding doesn't work properly on Windows or with fakeroot. # In addition, RHEL4 seems to have some problem killing the test # afterwards, and HP-UX is too slow for timing to be accurate. Also, - # this test randomly fails on Solaris and SUSE 11 and 12. - "test_skip_needs_work" string => "windows|using_fakeroot|redhat_4|hpux|sunos|suse_11|suse_12"; + # this test was disabled on all platforms in ENT-6939 + "test_skip_needs_work" string => "any"; methods: test_pass_1:: From 5a67674601f2b253deb6556d6fd5115c23ed1074 Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Sat, 27 Mar 2021 23:58:21 +0100 Subject: [PATCH 195/333] ENT-6939 disable package_lock.cf on all platforms (cherry picked from commit 735602404322b8fd943b11b7d3bf2b2b726d773b) --- .../00_basics/ifelapsed_and_expireafter/timed/package_lock.cf | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/package_lock.cf b/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/package_lock.cf index 334b1de0ac..120a8fca0f 100644 --- a/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/package_lock.cf +++ b/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/package_lock.cf @@ -13,7 +13,8 @@ bundle agent init # "no_fds" won't work correctly on Windows. # Also Solaris 9 and 10 don't seem to handle the backgrounding used in # this test. But it seems unrelated to the actual fix. - "test_skip_needs_work" string => "windows|sunos_5_9|sunos_5_10"; + # Moreover, this test was disabled on all platforms in ENT-6939 + "test_skip_needs_work" string => "any"; # The backgrounding doesn't work well under fakeroot. "test_skip_unsupported" string => "using_fakeroot"; From eca388ce2911ec692cff2be3e6518327702a641d Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Wed, 24 Mar 2021 12:01:49 +0100 Subject: [PATCH 196/333] Make sure hard class suse is defined on SUSE Ticket: CFE-3569 Changelog: None Signed-off-by: Lars Erik Wik (cherry picked from commit 2d0c678df5b9b64e792ad6b96ff14e9efcb45fac) --- libenv/sysinfo.c | 5 +++ .../01_basic/expected_os_classes.cf | 42 +++++++++++++++++++ 2 files changed, 47 insertions(+) create mode 100644 tests/acceptance/02_classes/01_basic/expected_os_classes.cf diff --git a/libenv/sysinfo.c b/libenv/sysinfo.c index cae60e41c7..d234f7047b 100644 --- a/libenv/sysinfo.c +++ b/libenv/sysinfo.c @@ -1143,6 +1143,11 @@ static void OSReleaseParse(EvalContext *ctx, const char *file_path) { alias = "redhat"; } + else if (StringEqual(os_release_id, "opensuse") || + StringEqual(os_release_id, "sles")) + { + alias = "suse"; + } if (os_release_version_id == NULL) { diff --git a/tests/acceptance/02_classes/01_basic/expected_os_classes.cf b/tests/acceptance/02_classes/01_basic/expected_os_classes.cf new file mode 100644 index 0000000000..36c65f35f3 --- /dev/null +++ b/tests/acceptance/02_classes/01_basic/expected_os_classes.cf @@ -0,0 +1,42 @@ +body common control +{ + bundlesequence => { "test", "check" }; +} + +bundle agent test +{ + meta: + "description" -> { "CFE-3608" } + string => "Make sure at least one of the expected os hard classes are defined"; + + vars: + DEBUG:: + "defined_classes" string => join("$(const.n)", classesmatching(".*")); +} + +bundle agent check +{ + classes: + "passed" or => { "debian", + "ubuntu", + "redhat", + "centos", + "fedora", + "aix", + "hpux", + "suse", + "windows", + "freebsd", + "macos", + "solaris" }; + + reports: + DEBUG&!passed:: + "None of the expected classes were defined."; + "Here is a list of defined classes:"; + "$(test.defined_classes)"; + passed:: + "$(this.promise_filename) Pass"; + !passed:: + "$(this.promise_filename) FAIL"; +} From 0cb7edd9ef93077d74389c7d75b3c0bf2e298f4f Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Tue, 30 Mar 2021 22:07:28 +0200 Subject: [PATCH 197/333] ENT-6975 update 'bin' gid for suse 15 in tests. Issue was that since recently, it changed from 1 (like on RHEL) to 2 (like on other Linuxes). Changelog: none (cherry picked from commit e54da26fb0dbfb46816779ac7cc6acfab36106ec) --- tests/acceptance/17_users/unsafe/user_queries.cf.sub | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/acceptance/17_users/unsafe/user_queries.cf.sub b/tests/acceptance/17_users/unsafe/user_queries.cf.sub index 7f3824d29c..25a57d635b 100644 --- a/tests/acceptance/17_users/unsafe/user_queries.cf.sub +++ b/tests/acceptance/17_users/unsafe/user_queries.cf.sub @@ -5,9 +5,9 @@ bundle common user_tests "group1" string => "bin"; "group2" string => "sys"; "gid2" string => "3"; - redhat|suse:: + redhat|suse.!suse_15:: "gid1" string => "1"; - !redhat.!suse.!windows:: + !redhat.(!suse|suse_15).!windows:: "gid1" string => "2"; windows:: From 1d09f0da4f7c1180fde1d1568e4c8be9a73729b1 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Thu, 1 Apr 2021 14:17:19 -0500 Subject: [PATCH 198/333] Fixed log level for failure to stat remote files without missing_ok In versions before 3.12.0 it was not possible to suppress errors generated from promises to copy remote files that did not exist. Because of this limitation, this specific log message was moved from ERROR to INFO as it could generate many logs that were impossible to avoid, whereas INFO log messages are not always emitted, especially by regularly scheduled runs. Version 3.12.0 introduced the missing_ok attribute which allows for clients to avoid considering a missing remote file as an error. This change restores the log level for the message to ERROR. Users should instrument their policy with missing_ok if they are promising copies of files that may not exist. Ticket: ENT-5879 Changelog: Restored log level for failure to state remote files without missing_ok to ERROR --- cf-agent/verify_files_utils.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cf-agent/verify_files_utils.c b/cf-agent/verify_files_utils.c index cc90db0c94..cf5a6828df 100644 --- a/cf-agent/verify_files_utils.c +++ b/cf-agent/verify_files_utils.c @@ -2685,7 +2685,7 @@ static PromiseResult CopyFileSources(EvalContext *ctx, char *destination, const } else { - cfPS(ctx, LOG_LEVEL_INFO, PROMISE_RESULT_FAIL, pp, attr, + cfPS(ctx, LOG_LEVEL_ERR, PROMISE_RESULT_FAIL, pp, attr, "Can't stat file '%s' on '%s' in files.copy_from promise", BufferData(source), conn ? conn->remoteip : "localhost"); BufferDestroy(source); From f60895119c4cddfc21edd7a6e0f338c84b4fd1b3 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 31 Mar 2021 10:35:50 +0200 Subject: [PATCH 199/333] Send exit code of the agent from cf-serverd in EXEC response When cf-runagent hails a server, it should be able to tell if the remote agent run succeeded or failed. Ticket: CFE-3594 Changelog: Exit code from remote agent run is now sent to cf-runagent (cherry picked from commit 2bbd7092e701de029c710a389e83fcf3675839d7) --- cf-serverd/server_common.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/cf-serverd/server_common.c b/cf-serverd/server_common.c index 524a1531a9..3b939478b5 100644 --- a/cf-serverd/server_common.c +++ b/cf-serverd/server_common.c @@ -1575,6 +1575,8 @@ bool DoExec2(const EvalContext *ctx, char *exec_args, char *sendbuf, size_t sendbuf_size) { + assert(conn != NULL); + /* STEP 0: Verify cfruncommand was successfully configured. */ if (NULL_OR_EMPTY(CFRUNCOMMAND)) { @@ -1789,7 +1791,15 @@ bool DoExec2(const EvalContext *ctx, } } free(line); - cf_pclose(pp); + int exit_code = cf_pclose(pp); + if (exit_code >= 0) + { + xsnprintf(sendbuf, sendbuf_size, "(exit code: %d)\n", exit_code); + if (SendTransaction(conn->conn_info, sendbuf, 0, CF_DONE) == -1) + { + Log(LOG_LEVEL_INFO, "Failed to send exit code from EXEC agent run"); + } + } return true; } From da8cf8c86f9b15d02e3cda9f7153a156d4e389ce Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 31 Mar 2021 13:59:23 +0200 Subject: [PATCH 200/333] Make cf-runagent exit with a code reflecting remote agent run status(es) To make it possible to tell if all agent runs succeeded or if there were some failures or errors. The exit codes are as follows: - exit code from the remote agent run if only 1 host was specified - number of failed remote agent runs up to 100 otherwise - >100 in case of other errors Ticket: CFE-3594 Changelog: cf-runagent now exits with a code reflecting remote agent run status(es) (cherry picked from commit ebeb84c0be31a2f4a3db5586387aa167b8dcdefa) --- cf-runagent/cf-runagent.c | 94 +++++++++++++++++++++++++++++++-------- 1 file changed, 76 insertions(+), 18 deletions(-) diff --git a/cf-runagent/cf-runagent.c b/cf-runagent/cf-runagent.c index e29634def9..d4fbcf2466 100644 --- a/cf-runagent/cf-runagent.c +++ b/cf-runagent/cf-runagent.c @@ -50,6 +50,8 @@ #include #include +#define CF_RA_EXIT_CODE_OTHER_ERR 101 + typedef enum { RUNAGENT_CONTROL_HOSTS, @@ -69,9 +71,9 @@ static void ThisAgentInit(void); static GenericAgentConfig *CheckOpts(int argc, char **argv); static void KeepControlPromises(EvalContext *ctx, const Policy *policy); -static bool HailServer(const EvalContext *ctx, const GenericAgentConfig *config, char *host); +static int HailServer(const EvalContext *ctx, const GenericAgentConfig *config, char *host); static void SendClassData(AgentConnection *conn); -static void HailExec(AgentConnection *conn, char *peer); +static int HailExec(AgentConnection *conn, char *peer); static FILE *NewStream(char *name); /*******************************************************************/ @@ -158,6 +160,41 @@ char REMOTEBUNDLES[CF_MAXVARSIZE] = ""; /*****************************************************************************/ +/** + * @param is_exit_code whether #remote_exit_status is a exit code directly + * (#true) or a status from wait() (#false) + */ +static inline void UpdateExitCode(int *exit_code, int remote_exit_status, bool one_host, bool is_exit_code) +{ + assert(exit_code != NULL); + + if (one_host) + { + if (is_exit_code) + { + *exit_code = remote_exit_status; + return; + } + + if (WIFEXITED(remote_exit_status)) + { + *exit_code = WEXITSTATUS(remote_exit_status); + return; + } + + *exit_code = CF_RA_EXIT_CODE_OTHER_ERR; + return; + } + + /* Other error should always take priority, otherwise, count failed remote + * agent runs. */ + if ((*exit_code < CF_RA_EXIT_CODE_OTHER_ERR) && + (!WIFEXITED(remote_exit_status) || (WEXITSTATUS(remote_exit_status) != EXIT_SUCCESS))) + { + *exit_code = MIN(*exit_code + 1, 100); + } +} + int main(int argc, char *argv[]) { #if !defined(__MINGW32__) @@ -178,17 +215,23 @@ int main(int argc, char *argv[]) KeepControlPromises(ctx, policy); // Set RUNATTR using copy + /* Exit codes: + * - exit code from the remote agent run if only 1 host specified + * - number of failed remote agent runs up to 100 otherwise + * - >100 in case of other errors */ + int exit_code = 0; + if (BACKGROUND && INTERACTIVE) { Log(LOG_LEVEL_ERR, "You cannot specify background mode and interactive mode together"); - DoCleanupAndExit(EXIT_FAILURE); + DoCleanupAndExit(CF_RA_EXIT_CODE_OTHER_ERR); } /* HvB */ + const bool one_host = (HOSTLIST != NULL) && (HOSTLIST->next == NULL); if (HOSTLIST) { const Rlist *rp = HOSTLIST; - while (rp != NULL) { @@ -206,8 +249,8 @@ int main(int argc, char *argv[]) { if (fork() == 0) /* child process */ { - HailServer(ctx, config, RlistScalarValue(rp)); - DoCleanupAndExit(EXIT_SUCCESS); + int remote_exit_code = HailServer(ctx, config, RlistScalarValue(rp)); + DoCleanupAndExit(remote_exit_code > 0 ? remote_exit_code : CF_RA_EXIT_CODE_OTHER_ERR); } else /* parent process */ { @@ -220,12 +263,14 @@ int main(int argc, char *argv[]) pid = wait(&status); Log(LOG_LEVEL_DEBUG, "child = %d, child number = %d", pid, count); count--; + UpdateExitCode(&exit_code, status, one_host, false); } } else /* serial */ #endif /* __MINGW32__ */ { - HailServer(ctx, config, RlistScalarValue(rp)); + int remote_exit_code = HailServer(ctx, config, RlistScalarValue(rp)); + UpdateExitCode(&exit_code, remote_exit_code, one_host, true); rp = rp->next; } } /* end while */ @@ -240,6 +285,7 @@ int main(int argc, char *argv[]) pid = wait(&status); Log(LOG_LEVEL_VERBOSE, "Child %d ended, number %d", pid, count); count--; + UpdateExitCode(&exit_code, status, one_host, false); } } #endif @@ -248,7 +294,7 @@ int main(int argc, char *argv[]) GenericAgentFinalize(ctx, config); CallCleanupFunctions(); - return 0; + return exit_code; } /*******************************************************************/ @@ -447,7 +493,7 @@ static void ThisAgentInit(void) /********************************************************************/ -static bool HailServer(const EvalContext *ctx, const GenericAgentConfig *config, char *host) +static int HailServer(const EvalContext *ctx, const GenericAgentConfig *config, char *host) { assert(host != NULL); @@ -463,7 +509,7 @@ static bool HailServer(const EvalContext *ctx, const GenericAgentConfig *config, if (hostname == NULL) { Log(LOG_LEVEL_INFO, "No remote hosts were specified to connect to"); - return false; + return -1; } if (port == NULL) { @@ -475,7 +521,7 @@ static bool HailServer(const EvalContext *ctx, const GenericAgentConfig *config, { Log(LOG_LEVEL_ERR, "HailServer: ERROR, could not resolve '%s'", hostname); - return false; + return -1; } Address2Hostkey(hostkey, sizeof(hostkey), ipaddr); @@ -556,13 +602,11 @@ static bool HailServer(const EvalContext *ctx, const GenericAgentConfig *config, if (conn == NULL) { Log(LOG_LEVEL_ERR, "Failed to connect to host: %s", hostname); - return false; + return -1; } /* Send EXEC command. */ - HailExec(conn, hostname); - - return true; + return HailExec(conn, hostname); } /********************************************************************/ @@ -704,7 +748,7 @@ static void SendClassData(AgentConnection *conn) /********************************************************************/ -static void HailExec(AgentConnection *conn, char *peer) +static int HailExec(AgentConnection *conn, char *peer) { char sendbuf[CF_BUFSIZE - CF_INBAND_OFFSET] = "EXEC"; size_t sendbuf_len = strlen(sendbuf); @@ -724,14 +768,14 @@ static void HailExec(AgentConnection *conn, char *peer) { Log(LOG_LEVEL_ERR, "Command longer than maximum transaction packet"); DisconnectServer(conn); - return; + return -1; } if (SendTransaction(conn->conn_info, sendbuf, 0, CF_DONE) == -1) { Log(LOG_LEVEL_ERR, "Transmission rejected. (send: %s)", GetErrorStr()); DisconnectServer(conn); - return; + return -1; } /* TODO we are sending class data right after EXEC, when the server might @@ -742,6 +786,7 @@ static void HailExec(AgentConnection *conn, char *peer) char recvbuffer[CF_BUFSIZE]; FILE *fp = NewStream(peer); + int exit_code = -1; while (true) { memset(recvbuffer, 0, sizeof(recvbuffer)); @@ -769,6 +814,18 @@ static void HailExec(AgentConnection *conn, char *peer) } else { + /* '(exit code: N)' is a special line, not prefixed with '>' (so not + * part of output) and sent last by new cf-serverd (3.18.0+) */ + if (StringStartsWith(recvbuffer, "(exit code:")) + { + /* Should never show up twice. */ + assert(exit_code == -1); + int scanned = sscanf(recvbuffer, "(exit code: %d)", &exit_code); + if (scanned != 1) + { + Log(LOG_LEVEL_ERR, "Failed to parse exit code from '%s'", recvbuffer); + } + } fprintf(fp, "%s> %s", ipaddr, recvbuffer); } @@ -787,6 +844,7 @@ static void HailExec(AgentConnection *conn, char *peer) fclose(fp); } DisconnectServer(conn); + return exit_code; } /********************************************************************/ From 833c6662162ed1789086ecc9a62cd31c329e40bd Mon Sep 17 00:00:00 2001 From: Georgios Kotzampopoulos Date: Sun, 28 Mar 2021 21:28:09 +0200 Subject: [PATCH 201/333] Changed log message about whitespace in class expressions to be error Output string was changed from: `"class names [...]"` to: `"class expressions [...]"` Ticket: CFE-3560 Changelog: Title (cherry picked from commit eb9dd6c5d315d5a7daebc50610529f306ed56a04) --- libpromises/eval_context.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libpromises/eval_context.c b/libpromises/eval_context.c index 261da8ffee..36b3208bef 100644 --- a/libpromises/eval_context.c +++ b/libpromises/eval_context.c @@ -548,7 +548,7 @@ ExpressionValue CheckClassExpression(const EvalContext *ctx, const char *context if (StringMatchFullWithPrecompiledRegex(context_expression_whitespace_rx, context)) { - Log(LOG_LEVEL_INFO, "class names can't be separated by whitespace without an intervening operator in expression '%s'", context); + Log(LOG_LEVEL_ERR, "class expressions can't be separated by whitespace without an intervening operator in expression '%s'", context); return EXPRESSION_VALUE_ERROR; } From 1d4c08d76b481cc12c6f354e90180061fbff9066 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Tue, 30 Mar 2021 10:46:49 -0500 Subject: [PATCH 202/333] Added flakey test result handling to testall If a tests/acceptance test outputs FLAKEY, testall will consider it a flakey failure and not fail the test run. If FLAKEY_IS_FAIL env var is present then flakey failures will count as real failures and fail the test run. A selftest.sh was added to test the testall script itself. If it fails it will simply error out and not provide details via JUnit xml output. Ticket: ENT-6935 Changelog: title (cherry picked from commit 34fe59f18f8bbaedb9b96042cbaf2d422ebf6483) --- tests/acceptance/Makefile.am | 4 + tests/acceptance/README | 11 ++- tests/acceptance/selftest.sh | 100 ++++++++++++++++++++++++ tests/acceptance/selftest/fail.cf | 5 ++ tests/acceptance/selftest/flaky_fail.cf | 5 ++ tests/acceptance/selftest/flaky_pass.cf | 5 ++ tests/acceptance/selftest/pass.cf | 5 ++ tests/acceptance/selftest/skipped.cf | 18 +++++ tests/acceptance/selftest/soft.cf | 5 ++ tests/acceptance/testall | 26 ++++-- 10 files changed, 176 insertions(+), 8 deletions(-) create mode 100755 tests/acceptance/selftest.sh create mode 100644 tests/acceptance/selftest/fail.cf create mode 100644 tests/acceptance/selftest/flaky_fail.cf create mode 100644 tests/acceptance/selftest/flaky_pass.cf create mode 100644 tests/acceptance/selftest/pass.cf create mode 100644 tests/acceptance/selftest/skipped.cf create mode 100644 tests/acceptance/selftest/soft.cf diff --git a/tests/acceptance/Makefile.am b/tests/acceptance/Makefile.am index c6a3b70ed5..2c1d6ceff0 100644 --- a/tests/acceptance/Makefile.am +++ b/tests/acceptance/Makefile.am @@ -87,6 +87,7 @@ endif # to run in parallel; TODO fix "make -n" not working: check-local: + $(TESTS_ENVIRONMENT) MAKE=$(MAKE) ./testall + + $(srcdir)/selftest.sh EXTRA_DIST = default.cf.sub dcs.cf.sub plucked.cf.sub run_with_server.cf.sub \ @@ -97,6 +98,9 @@ EXTRA_DIST = default.cf.sub dcs.cf.sub plucked.cf.sub run_with_server.cf.sub \ # Recursively include all tests in the dist tarball and set proper permissions EXTRA_DIST += $(wildcard [0-9]*) +# Include selftest.sh and selftest directory contents in dist tarball +EXTRA_DIST += selftest.sh selftest + dist-hook: chmod -R go-w $(distdir) diff --git a/tests/acceptance/README b/tests/acceptance/README index 4b0ada6a49..4ee8678114 100644 --- a/tests/acceptance/README +++ b/tests/acceptance/README @@ -15,6 +15,11 @@ In case you find a bug you are encouraged to create tests in format of testsuite which demonstrate bug found, so the test could be added to this testsuite and checked for in the future. +Note that the testall script generates JUnit style XML output for parsing by CI systems. + +https://llg.cubic.org/docs/junit/ + + ------------------------------------------------------------------------------ Preparing for running tests ------------------------------------------------------------------------------ @@ -204,16 +209,20 @@ possible variable names: - test_soft_fail - test_suppress_fail + - test_flakey_fail Runs the test, but will accept failure. Use this when there is a real failure, but it is acceptable for the time being. This variable requires a meta tag on the variable set to "redmine", where is a Redmine issue number. - There is a subtle difference between the two. Soft failures will + There is a subtle difference between the three. Soft failures will not be reported as a failure in the XML output, and is appropriate for test cases that document incoming bug reports. Suppressed failures will count as a failure, but it won't block the build, and is appropriate for regressions or bad test failures. + Flakey failures count in a category by themselves and won't block + the build. If any are present then a different exit code will be + produced from the test run so that CI runners can react accordingly. Additionally, a *description* meta variable can be added to the test to describe its function. diff --git a/tests/acceptance/selftest.sh b/tests/acceptance/selftest.sh new file mode 100755 index 0000000000..d714aedec9 --- /dev/null +++ b/tests/acceptance/selftest.sh @@ -0,0 +1,100 @@ +#!/bin/sh + + +# because results in test.xml should not be parsed by CI or otherwise +# failures there may be expected and will be checked by grep below +clean () +{ + rm test.xml +} +check () +{ + expect=$1 + log=$2 + if ! grep "$expect" "$log" >/dev/null + then + echo "error: failed to match regex \"$expect\" in log file $log" + return 1 + fi +} + +mkdir -p workdir + +./testall selftest > workdir/selftest_normal.log +ret=$? +if [ "$ret" -ne "0" ] +then + echo "error: exit code for selftest should be 0 but was $ret" + clean + exit 1 +fi + +log="workdir/selftest_normal.log" + +errors=0 +_pwd=$(pwd) + +for regex in \ +"./selftest/fail.cf FAIL (Suppressed, R: $_pwd/./selftest/fail.cf XFAIL)" \ +"./selftest/flaky_fail.cf Flakey fail (R: $_pwd/./selftest/flaky_fail.cf FLAKEY)" \ +"./selftest/flaky_pass.cf Pass" \ +"./selftest/pass.cf Pass" \ +"./selftest/skipped.cf Skipped (Test needs work)" \ +"./selftest/soft.cf Soft fail (R: $_pwd/./selftest/soft.cf SFAIL)" \ +"Passed tests: 2" \ +"Failed tests: 1 (1 are known and suppressed)" \ +"Skipped tests: 1" \ +"Soft failures: 1" \ +"Flakey failures: 1" \ +"Total tests: 6" +do + check "$regex" "$log" || errors=$((errors + 1)) +done +if [ "$errors" -ne "0" ] +then + echo "error: $errors error(s) occurred for selftest (flakey is not fail)" + echo "=== BEGIN $log ===" + cat $log + echo "=== END $log ===" + clean + exit 1 +fi + + +FLAKEY_IS_FAIL=1 ./testall selftest > workdir/selftest_flakey_is_fail.log +ret=$? +if [ "$ret" -ne "4" ] +then + echo "error: exit code for testall with FLAKEY_IS_FAIL=1 should be 4 but was $ret" + clean + exit 1 +fi + +errors=0 +log="workdir/selftest_flakey_is_fail.log" + +for regex in \ +"./selftest/fail.cf FAIL (Suppressed, R: $_pwd/./selftest/fail.cf XFAIL)" \ +"./selftest/flaky_fail.cf Flakey fail (R: $_pwd/./selftest/flaky_fail.cf FLAKEY)" \ +"./selftest/flaky_pass.cf Pass" \ +"./selftest/pass.cf Pass" \ +"./selftest/skipped.cf Skipped (Test needs work)" \ +"./selftest/soft.cf Soft fail (R: $_pwd/./selftest/soft.cf SFAIL)" \ +"Passed tests: 2" \ +"Failed tests: 1 (1 are known and suppressed)" \ +"Skipped tests: 1" \ +"Soft failures: 1" \ +"Flakey failures: 1" \ +"Total tests: 6" +do + check "$regex" "$log" || errors=$((errors + 1)) +done +if [ "$errors" -ne "0" ] +then + echo "error: $errors error(s) occurred for selftest (flakey is fail)" + echo "=== BEGIN $log ===" + cat $log + echo "=== END $log ===" + clean + exit 1 +fi diff --git a/tests/acceptance/selftest/fail.cf b/tests/acceptance/selftest/fail.cf new file mode 100644 index 0000000000..86ff9e9e84 --- /dev/null +++ b/tests/acceptance/selftest/fail.cf @@ -0,0 +1,5 @@ +bundle agent main +{ + reports: + "$(this.promise_filename) XFAIL"; +} diff --git a/tests/acceptance/selftest/flaky_fail.cf b/tests/acceptance/selftest/flaky_fail.cf new file mode 100644 index 0000000000..ab5a3f874f --- /dev/null +++ b/tests/acceptance/selftest/flaky_fail.cf @@ -0,0 +1,5 @@ +bundle agent main +{ + reports: + "$(this.promise_filename) FLAKEY"; +} diff --git a/tests/acceptance/selftest/flaky_pass.cf b/tests/acceptance/selftest/flaky_pass.cf new file mode 100644 index 0000000000..e5cca28800 --- /dev/null +++ b/tests/acceptance/selftest/flaky_pass.cf @@ -0,0 +1,5 @@ +bundle agent main +{ + reports: + "$(this.promise_filename) Pass"; +} diff --git a/tests/acceptance/selftest/pass.cf b/tests/acceptance/selftest/pass.cf new file mode 100644 index 0000000000..e5cca28800 --- /dev/null +++ b/tests/acceptance/selftest/pass.cf @@ -0,0 +1,5 @@ +bundle agent main +{ + reports: + "$(this.promise_filename) Pass"; +} diff --git a/tests/acceptance/selftest/skipped.cf b/tests/acceptance/selftest/skipped.cf new file mode 100644 index 0000000000..3f33de1a24 --- /dev/null +++ b/tests/acceptance/selftest/skipped.cf @@ -0,0 +1,18 @@ +body common control +{ + inputs => { "../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +bundle agent test +{ + meta: + "test_skip_needs_work" string => "any"; +} + +bundle agent check +{ + reports: + "$(this.promise_filename) FAIL"; +} diff --git a/tests/acceptance/selftest/soft.cf b/tests/acceptance/selftest/soft.cf new file mode 100644 index 0000000000..f9379abde0 --- /dev/null +++ b/tests/acceptance/selftest/soft.cf @@ -0,0 +1,5 @@ +bundle agent main +{ + reports: + "$(this.promise_filename) SFAIL"; +} diff --git a/tests/acceptance/testall b/tests/acceptance/testall index 907d3324ac..b43b9bd5fe 100755 --- a/tests/acceptance/testall +++ b/tests/acceptance/testall @@ -154,6 +154,7 @@ FAILED_TESTS=0 SUPPRESSED_FAILURES=0 SOFT_FAILURES=0 SKIPPED_TESTS=0 +FLAKEY_FAILURES=0 # # Many older platforms don't support date +%s, so check for compatibility @@ -615,6 +616,12 @@ export CFENGINE_TEST_OVERRIDE_WORKDIR TEMP CFENGINE_TEST_OVERRIDE_EXTENSION_LIBR then RESULT=Skip RESULT_MSG="${COLOR_WARNING}Skipped (Test needs work)${COLOR_NORMAL}" + elif egrep "R: .*$ESCAPED_TEST FLAKEY" $OUTFILE > /dev/null + then + RESULT=FLAKEY + TICKET="$(egrep "R: .*$ESCAPED_TEST FLAKEY" $OUTFILE | sed -e "s,.*FLAKEY/\(.*\),\1,")" + RESULT_MSG="${COLOR_WARNING}Flakey fail ($TICKET)${COLOR_NORMAL}" + else RESULT=FAIL RESULT_MSG="${COLOR_FAILURE}FAIL${COLOR_NORMAL}" @@ -1028,7 +1035,7 @@ then fi done else - ALL_TESTS="$ALL_TESTS${ALL_TESTS:+ }$(find . -name workdir -prune -o -name '*.cf' -print | sort)" + ALL_TESTS="$ALL_TESTS${ALL_TESTS:+ }$(find . \( -name selftest -o -name workdir \) -prune -o -name '*.cf' -print | sort)" fi for addtest in $ALL_TESTS @@ -1095,6 +1102,7 @@ print_footer() { FAILED_TESTS=`egrep '^(FAIL|XFAIL) ' $SUMMARY | wc -l | tr -d ' '` SUPPRESSED_FAILURES=`egrep '^XFAIL ' $SUMMARY | wc -l | tr -d ' '` SOFT_FAILURES=`egrep '^SFAIL ' $SUMMARY | wc -l | tr -d ' '` + FLAKEY_FAILURES=`egrep '^FLAKEY ' $SUMMARY | wc -l | tr -d ' '` SKIPPED_TESTS=`egrep '^Skip ' $SUMMARY | wc -l | tr -d ' '` ( echo @@ -1103,17 +1111,18 @@ print_footer() { ) | tee -a "$LOG" | tee -a "$SUMMARY" >&7 ( echo - echo "Passed tests: $PASSED_TESTS" - printf "Failed tests: $FAILED_TESTS" + echo "Passed tests: $PASSED_TESTS" + printf "Failed tests: $FAILED_TESTS" if [ $SUPPRESSED_FAILURES -gt 0 ] then echo " ($SUPPRESSED_FAILURES are known and suppressed)" else echo fi - echo "Skipped tests: $SKIPPED_TESTS" - echo "Soft failures: $SOFT_FAILURES" - echo "Total tests: $TESTS_COUNT" + echo "Skipped tests: $SKIPPED_TESTS" + echo "Soft failures: $SOFT_FAILURES" + echo "Flakey failures: $FLAKEY_FAILURES" + echo "Total tests: $TESTS_COUNT" ) | tee -a "$LOG" | tee -a "$SUMMARY" if [ -n "$PRINTLOG" ] @@ -1358,7 +1367,7 @@ then finish_xml fi -if [ $(($PASSED_TESTS+$FAILED_TESTS+$SOFT_FAILURES+$SKIPPED_TESTS)) -ne $TESTS_COUNT ] +if [ $(($PASSED_TESTS+$FAILED_TESTS+$SOFT_FAILURES+$SKIPPED_TESTS+$FLAKEY_FAILURES)) -ne $TESTS_COUNT ] then echo "WARNING: Number of test results does not match number of tests!" echo "Did something go wrong during execution?" @@ -1367,6 +1376,9 @@ fi if [ "$FAILED_TESTS" -gt "$SUPPRESSED_FAILURES" ] then exit 1 +elif [ "$FLAKEY_FAILURES" -gt "0" ] && [ -n "$FLAKEY_IS_FAIL" ] +then + exit 4 else exit 0 fi From ad75058a4000f922801f79eb8b41a56ef588282c Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Tue, 13 Apr 2021 03:52:15 -0500 Subject: [PATCH 203/333] Disable testall selftest Getting this selftest working in CI is proving difficult. Disable for now in preference for green builds. Ticket: ENT-6935 Changelog: title (cherry picked from commit e4fffb9223f455a2d995034f3403e182034a9e70) --- tests/acceptance/Makefile.am | 4 ---- 1 file changed, 4 deletions(-) diff --git a/tests/acceptance/Makefile.am b/tests/acceptance/Makefile.am index 2c1d6ceff0..c6a3b70ed5 100644 --- a/tests/acceptance/Makefile.am +++ b/tests/acceptance/Makefile.am @@ -87,7 +87,6 @@ endif # to run in parallel; TODO fix "make -n" not working: check-local: + $(TESTS_ENVIRONMENT) MAKE=$(MAKE) ./testall - + $(srcdir)/selftest.sh EXTRA_DIST = default.cf.sub dcs.cf.sub plucked.cf.sub run_with_server.cf.sub \ @@ -98,9 +97,6 @@ EXTRA_DIST = default.cf.sub dcs.cf.sub plucked.cf.sub run_with_server.cf.sub \ # Recursively include all tests in the dist tarball and set proper permissions EXTRA_DIST += $(wildcard [0-9]*) -# Include selftest.sh and selftest directory contents in dist tarball -EXTRA_DIST += selftest.sh selftest - dist-hook: chmod -R go-w $(distdir) From 2a817e2348d70c5b6c203e1a72792cd7ef2476ca Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Fri, 26 Mar 2021 12:02:43 +0100 Subject: [PATCH 204/333] Hard class with OS- name & version major on HP-UX Make sure hard class with OS- name & version major is defined on HP-UX. Ticket: CFE-3609 Changelog: None Signed-off-by: Lars Erik Wik (cherry picked from commit 701cbaf2a1e353a4f1a959dc03780052863e4524) --- libenv/sysinfo.c | 30 +++++++++++++++ .../02_classes/01_basic/hp_ux_major.cf | 38 +++++++++++++++++++ 2 files changed, 68 insertions(+) create mode 100644 tests/acceptance/02_classes/01_basic/hp_ux_major.cf diff --git a/libenv/sysinfo.c b/libenv/sysinfo.c index d234f7047b..a3b251e6b1 100644 --- a/libenv/sysinfo.c +++ b/libenv/sysinfo.c @@ -1396,6 +1396,36 @@ static void OSClasses(EvalContext *ctx) snprintf(context, CF_BUFSIZE, "%s_%s", VSYSNAME.sysname, vbuff); SetFlavor(ctx, context); + +#ifdef __hpux + /* + * Define a hard class with just the version major number of HP-UX + * + * For example, when being run on HP-UX B.11.23 the following class will + * be defined: hpux_11 + */ + + // Extract major version number + char *major = NULL; + for (char *sp = vbuff; *sp != '\0'; sp++) + { + if (major == NULL && isdigit(*sp)) + { + major = sp; + } + else if (!isdigit(*sp)) + { + *sp = '\0'; + } + } + + if (major != NULL) + { + snprintf(context, CF_BUFSIZE, "hpux_%s", major); + EvalContextClassPutHard(ctx, context, "source=agent,derived-from=sys.flavor"); + } +#endif + #ifdef __FreeBSD__ /* * Define a hard class with just the version major number on FreeBSD diff --git a/tests/acceptance/02_classes/01_basic/hp_ux_major.cf b/tests/acceptance/02_classes/01_basic/hp_ux_major.cf new file mode 100644 index 0000000000..5c96775a23 --- /dev/null +++ b/tests/acceptance/02_classes/01_basic/hp_ux_major.cf @@ -0,0 +1,38 @@ +body common control +{ + bundlesequence => { "test", "check" }; +} + +bundle agent test +{ + meta: + "description" -> { "CFE-3609" } + string => "Make sure the class 'hpux_' is defined on HP-UX"; + + vars: + DEBUG:: + "defined_classes" string => join("$(const.n)", classesmatching(".*")); +} + +bundle agent check +{ + classes: + # if 'hpux': make sure 'hpux_10' or 'hpux_11' is defined + hpux:: + "passed" expression => "hpux_10|hpux_11"; + # if not 'hpux': make sure neither 'hpux_10' nor 'hpux_11' is defined + !hpux:: + "passed" expression => "!(hpux_10|hpux_11)"; + + reports: + DEBUG&hpux&!passed:: + "No class containing OS- name & version major is defined on HP-UX."; + "Here is a list of defined classes:"; + "$(test.defined_classes)"; + DEBUG&!hpux&!passed:: + "A class containing 'hpux_' was defined on a non HP-UX system"; + passed:: + "$(this.promise_filename) Pass"; + !passed:: + "$(this.promise_filename) FAIL"; +} From 2020b5b22bebc82df1d26242f29e503487e026ce Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Tue, 30 Mar 2021 10:46:49 -0500 Subject: [PATCH 205/333] Added flakey test result handling to testall If a tests/acceptance test outputs FLAKEY, testall will consider it a flakey failure and not fail the test run. Ticket: ENT-6935 Changelog: title (cherry picked from commit 65db29137ece9d8cb183bbcd3cef6e8a99790f76) --- tests/acceptance/dcs.cf.sub | 6 +++- tests/acceptance/selftest.sh | 34 +++++++++---------- tests/acceptance/selftest/flakey_meta_fail.cf | 19 +++++++++++ .../selftest/flakey_meta_no_ticket.cf | 18 ++++++++++ tests/acceptance/selftest/soft_no_ticket.cf | 18 ++++++++++ 5 files changed, 77 insertions(+), 18 deletions(-) create mode 100644 tests/acceptance/selftest/flakey_meta_fail.cf create mode 100644 tests/acceptance/selftest/flakey_meta_no_ticket.cf create mode 100644 tests/acceptance/selftest/soft_no_ticket.cf diff --git a/tests/acceptance/dcs.cf.sub b/tests/acceptance/dcs.cf.sub index d2163c60c0..38843faf66 100644 --- a/tests/acceptance/dcs.cf.sub +++ b/tests/acceptance/dcs.cf.sub @@ -269,7 +269,7 @@ bundle agent collect_stories_metadata bundle agent test_precheck { vars: - "fail_types" slist => { "suppress", "soft" }; + "fail_types" slist => { "suppress", "soft", "flakey" }; "test_skip_unsupported" slist => variablesmatching(".*_meta\.test_skip_unsupported"); "test_skip_needs_work" slist => variablesmatching(".*_meta\.test_skip_needs_work"); @@ -304,6 +304,10 @@ bundle agent test_precheck "$(this.promise_filename) SFAIL/$(test_soft_fail_ticket)"; test_soft_fail_match.!test_soft_fail_ticket_set:: "$(this.promise_filename) FAIL/no_ticket_number"; + test_flakey_fail_match.test_flakey_fail_ticket_set:: + "$(this.promise_filename) FLAKEY/$(test_flakey_fail_ticket)"; + test_flakey_fail_match.!test_flakey_fail_ticket_set:: + "$(this.promise_filename) FAIL/no_ticket_number"; } ####################################################### diff --git a/tests/acceptance/selftest.sh b/tests/acceptance/selftest.sh index d714aedec9..eb250a3f2e 100755 --- a/tests/acceptance/selftest.sh +++ b/tests/acceptance/selftest.sh @@ -5,7 +5,7 @@ # failures there may be expected and will be checked by grep below clean () { - rm test.xml + rm -f test.xml } check () { @@ -22,9 +22,9 @@ mkdir -p workdir ./testall selftest > workdir/selftest_normal.log ret=$? -if [ "$ret" -ne "0" ] +if [ "$ret" -ne "1" ] then - echo "error: exit code for selftest should be 0 but was $ret" + echo "error: exit code for selftest should be 1 but was $ret" clean exit 1 fi @@ -36,17 +36,20 @@ _pwd=$(pwd) for regex in \ "./selftest/fail.cf FAIL (Suppressed, R: $_pwd/./selftest/fail.cf XFAIL)" \ +"./selftest/flakey_meta_fail.cf Flakey fail (ENT-6947)" \ +"./selftest/flakey_meta_no_ticket.cf FAIL (Tried to suppress failure, but no issue number is provided) (UNEXPECTED FAILURE)" \ "./selftest/flaky_fail.cf Flakey fail (R: $_pwd/./selftest/flaky_fail.cf FLAKEY)" \ "./selftest/flaky_pass.cf Pass" \ "./selftest/pass.cf Pass" \ "./selftest/skipped.cf Skipped (Test needs work)" \ "./selftest/soft.cf Soft fail (R: $_pwd/./selftest/soft.cf SFAIL)" \ +"./selftest/soft_no_ticket.cf FAIL (Tried to suppress failure, but no issue number is provided) (UNEXPECTED FAILURE)" \ "Passed tests: 2" \ -"Failed tests: 1 (1 are known and suppressed)" \ +"Failed tests: 3 (1 are known and suppressed)" \ "Skipped tests: 1" \ "Soft failures: 1" \ -"Flakey failures: 1" \ -"Total tests: 6" +"Flakey failures: 2" \ +"Total tests: 9" do check "$regex" "$log" || errors=$((errors + 1)) done @@ -61,7 +64,7 @@ then fi -FLAKEY_IS_FAIL=1 ./testall selftest > workdir/selftest_flakey_is_fail.log +FLAKEY_IS_FAIL=1 ./testall selftest/flakey_meta_fail.cf selftest/flaky_fail.cf selftest/flaky_pass.cf > workdir/selftest_flakey_is_fail.log ret=$? if [ "$ret" -ne "4" ] then @@ -74,18 +77,15 @@ errors=0 log="workdir/selftest_flakey_is_fail.log" for regex in \ -"./selftest/fail.cf FAIL (Suppressed, R: $_pwd/./selftest/fail.cf XFAIL)" \ +"./selftest/flakey_meta_fail.cf Flakey fail (ENT-6947)" \ "./selftest/flaky_fail.cf Flakey fail (R: $_pwd/./selftest/flaky_fail.cf FLAKEY)" \ "./selftest/flaky_pass.cf Pass" \ -"./selftest/pass.cf Pass" \ -"./selftest/skipped.cf Skipped (Test needs work)" \ -"./selftest/soft.cf Soft fail (R: $_pwd/./selftest/soft.cf SFAIL)" \ -"Passed tests: 2" \ -"Failed tests: 1 (1 are known and suppressed)" \ -"Skipped tests: 1" \ -"Soft failures: 1" \ -"Flakey failures: 1" \ -"Total tests: 6" +"Passed tests: 1" \ +"Failed tests: 0" \ +"Skipped tests: 0" \ +"Soft failures: 0" \ +"Flakey failures: 2" \ +"Total tests: 3" do check "$regex" "$log" || errors=$((errors + 1)) done diff --git a/tests/acceptance/selftest/flakey_meta_fail.cf b/tests/acceptance/selftest/flakey_meta_fail.cf new file mode 100644 index 0000000000..b9c813387d --- /dev/null +++ b/tests/acceptance/selftest/flakey_meta_fail.cf @@ -0,0 +1,19 @@ +body common control +{ + inputs => { "../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +bundle agent test +{ + meta: + "test_flakey_fail" string => "any", + meta => { "ENT-6947" }; +} + +bundle agent check +{ + reports: + "$(this.promise_filename) FAIL"; +} diff --git a/tests/acceptance/selftest/flakey_meta_no_ticket.cf b/tests/acceptance/selftest/flakey_meta_no_ticket.cf new file mode 100644 index 0000000000..20d6598e3e --- /dev/null +++ b/tests/acceptance/selftest/flakey_meta_no_ticket.cf @@ -0,0 +1,18 @@ +body common control +{ + inputs => { "../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +bundle agent test +{ + meta: + "test_flakey_fail" string => "any"; +} + +bundle agent check +{ + reports: + "$(this.promise_filename) FAIL"; +} diff --git a/tests/acceptance/selftest/soft_no_ticket.cf b/tests/acceptance/selftest/soft_no_ticket.cf new file mode 100644 index 0000000000..3cad6df23f --- /dev/null +++ b/tests/acceptance/selftest/soft_no_ticket.cf @@ -0,0 +1,18 @@ +body common control +{ + inputs => { "../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +bundle agent test +{ + meta: + "test_soft_fail" string => "any"; +} + +bundle agent check +{ + reports: + "$(this.promise_filename) FAIL"; +} From 1c3c43de80da35ab17f8c5dfa42b157450a48000 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Tue, 13 Apr 2021 16:26:19 +0200 Subject: [PATCH 206/333] Change SINGLE_COPY_CACHE to be a StringSet The SINGLE_COPY_CACHE is a helper structure to remember files that were copied by the agent because they matched a regular expression from the SINGLE_COPY_LIST ('files_single_copy' agent control attribute). It should not be treated as a list of regular expressions, it should be a list of simple strings (file paths). And because ordering doesn't matter, it can be a set which is optimized for fast presence checking. Also make sure the structure is not leaked by the agent run. Ticket: CFE-3621 Changelog: files_single_copy no longer treats paths of copied files as regular expressions (cherry picked from commit f25aba497a489c86df45b570775896c7006246cb) --- cf-agent/cf-agent.c | 2 ++ cf-agent/verify_files_utils.c | 10 +++++----- cf-agent/verify_files_utils.h | 1 + 3 files changed, 8 insertions(+), 5 deletions(-) diff --git a/cf-agent/cf-agent.c b/cf-agent/cf-agent.c index 6f7d8490bc..61eb205362 100644 --- a/cf-agent/cf-agent.c +++ b/cf-agent/cf-agent.c @@ -331,6 +331,7 @@ int main(int argc, char *argv[]) Nova_NoteAgentExecutionPerformance(config->input_file, start); GenericAgentFinalize(ctx, config); + StringSetDestroy(SINGLE_COPY_CACHE); #ifdef HAVE_LIBXML2 xmlCleanupParser(); @@ -1012,6 +1013,7 @@ static void KeepControlPromises(EvalContext *ctx, const Policy *policy, GenericA if (strcmp(cp->lval, CFA_CONTROLBODY[AGENT_CONTROL_FSINGLECOPY].lval) == 0) { SINGLE_COPY_LIST = value; + SINGLE_COPY_CACHE = StringSetNew(); Log(LOG_LEVEL_VERBOSE, "Setting file single copy list"); continue; } diff --git a/cf-agent/verify_files_utils.c b/cf-agent/verify_files_utils.c index cf5a6828df..aeec296e34 100644 --- a/cf-agent/verify_files_utils.c +++ b/cf-agent/verify_files_utils.c @@ -75,7 +75,7 @@ static const Rlist *AUTO_DEFINE_LIST = NULL; /* GLOBAL_P */ static Item *VSETXIDLIST = NULL; const Rlist *SINGLE_COPY_LIST = NULL; /* GLOBAL_P */ -static Rlist *SINGLE_COPY_CACHE = NULL; /* GLOBAL_X */ +StringSet *SINGLE_COPY_CACHE = NULL; /* GLOBAL_X */ static bool TransformFile(EvalContext *ctx, char *file, const Attributes *attr, const Promise *pp, PromiseResult *result); static PromiseResult VerifyName(EvalContext *ctx, char *path, const struct stat *sb, const Attributes *attr, const Promise *pp); @@ -250,7 +250,7 @@ static PromiseResult CfCopyFile(EvalContext *ctx, char *sourcefile, return PROMISE_RESULT_NOOP; } - if (RlistIsInListOfRegex(SINGLE_COPY_CACHE, destfile)) + if ((SINGLE_COPY_CACHE != NULL) && StringSetContains(SINGLE_COPY_CACHE, destfile)) { Log(LOG_LEVEL_INFO, "Skipping single-copied file '%s'", destfile); return PROMISE_RESULT_NOOP; @@ -407,7 +407,7 @@ static PromiseResult CfCopyFile(EvalContext *ctx, char *sourcefile, if (SINGLE_COPY_LIST) { - RlistPrependScalarIdemp(&SINGLE_COPY_CACHE, destfile); + StringSetAdd(SINGLE_COPY_CACHE, xstrdup(destfile)); } if (MatchRlistItem(ctx, AUTO_DEFINE_LIST, destfile)) @@ -575,7 +575,7 @@ static PromiseResult CfCopyFile(EvalContext *ctx, char *sourcefile, if (RlistIsInListOfRegex(SINGLE_COPY_LIST, destfile)) { - RlistPrependScalarIdemp(&SINGLE_COPY_CACHE, destfile); + StringSetAdd(SINGLE_COPY_CACHE, xstrdup(destfile)); } } else @@ -607,7 +607,7 @@ static PromiseResult CfCopyFile(EvalContext *ctx, char *sourcefile, if (RlistIsInListOfRegex(SINGLE_COPY_LIST, destfile)) { - RlistPrependScalarIdemp(&SINGLE_COPY_CACHE, destfile); + StringSetAdd(SINGLE_COPY_CACHE, xstrdup(destfile)); } cfPS(ctx, LOG_LEVEL_VERBOSE, PROMISE_RESULT_NOOP, pp, &attr, diff --git a/cf-agent/verify_files_utils.h b/cf-agent/verify_files_utils.h index 7a8bc7072d..e3e1a3ce7e 100644 --- a/cf-agent/verify_files_utils.h +++ b/cf-agent/verify_files_utils.h @@ -30,6 +30,7 @@ #include extern const Rlist *SINGLE_COPY_LIST; +extern StringSet *SINGLE_COPY_CACHE; void SetFileAutoDefineList(const Rlist *auto_define_list); From 62bfdd6caf0c88a5b15936c6fde96f1d81198d8d Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 14 Apr 2021 12:21:22 +0200 Subject: [PATCH 207/333] Allow 'files_single_copy' body agent control attribute to be an empty list It is a list of regular expressions identifying file that should only be copied once. So an empty list is a perfectly valid value. Also inform that an empty list is not a valid value for other attributes instead of saying that the specified attribute is invalid. Ticket: CFE-3622 Changelog: files_single_copy body agent control attribute can now be an empty list (cherry picked from commit 12af8af1dd6066aee2e0d443e1f584ee6d764332) --- cf-agent/cf-agent.c | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/cf-agent/cf-agent.c b/cf-agent/cf-agent.c index 61eb205362..fd9cb92cb4 100644 --- a/cf-agent/cf-agent.c +++ b/cf-agent/cf-agent.c @@ -867,13 +867,31 @@ static void KeepControlPromises(EvalContext *ctx, const Policy *policy, GenericA const void *value = EvalContextVariableGet(ctx, ref, &value_type); VarRefDestroy(ref); - /* If var not found, or if it's an empty list. */ - if (value_type == CF_DATA_TYPE_NONE || value == NULL) + /* If var not found */ + if (value_type == CF_DATA_TYPE_NONE) { Log(LOG_LEVEL_ERR, "Unknown lval '%s' in agent control body", cp->lval); continue; } + /* 'files_single_copy => { }' is a perfectly valid case. */ + if (strcmp(cp->lval, CFA_CONTROLBODY[AGENT_CONTROL_FSINGLECOPY].lval) == 0) + { + SINGLE_COPY_LIST = value; + SINGLE_COPY_CACHE = StringSetNew(); + Log(LOG_LEVEL_VERBOSE, "Setting file single copy list"); + continue; + } + + /* Empty list is not supported for the other constraints/attributes. */ + if (value == NULL) + { + Log(LOG_LEVEL_ERR, + "Empty list is not a valid value for '%s' attribute in agent control body", + cp->lval); + continue; + } + if (strcmp(cp->lval, CFA_CONTROLBODY[AGENT_CONTROL_MAXCONNECTIONS].lval) == 0) { CFA_MAXTHREADS = (int) IntFromString(value); @@ -1010,14 +1028,6 @@ static void KeepControlPromises(EvalContext *ctx, const Policy *policy, GenericA continue; } - if (strcmp(cp->lval, CFA_CONTROLBODY[AGENT_CONTROL_FSINGLECOPY].lval) == 0) - { - SINGLE_COPY_LIST = value; - SINGLE_COPY_CACHE = StringSetNew(); - Log(LOG_LEVEL_VERBOSE, "Setting file single copy list"); - continue; - } - if (strcmp(cp->lval, CFA_CONTROLBODY[AGENT_CONTROL_FAUTODEFINE].lval) == 0) { SetFileAutoDefineList(value); From bf2a48dcb4c625ccabbb099f6b76ff3cb8d8d094 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 14 Apr 2021 15:35:52 +0200 Subject: [PATCH 208/333] Log the slist from the files_single_copy body control agent attribute So that users can see the value that is actually being used. Also use StringEqual() instead of strcmp() and add an assert that the value is actually and slist (enforced by the parser, but better check twice in debug builds). Ticket: CFE-3622 Changelog: Value of the 'files_single_copy' body control attribute is now logged in verbose logging mode (cherry picked from commit 47a3ac9c6006bbe4508bb9950aea940ae9e68c98) --- cf-agent/cf-agent.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/cf-agent/cf-agent.c b/cf-agent/cf-agent.c index fd9cb92cb4..ff999a3621 100644 --- a/cf-agent/cf-agent.c +++ b/cf-agent/cf-agent.c @@ -875,11 +875,17 @@ static void KeepControlPromises(EvalContext *ctx, const Policy *policy, GenericA } /* 'files_single_copy => { }' is a perfectly valid case. */ - if (strcmp(cp->lval, CFA_CONTROLBODY[AGENT_CONTROL_FSINGLECOPY].lval) == 0) + if (StringEqual(cp->lval, CFA_CONTROLBODY[AGENT_CONTROL_FSINGLECOPY].lval)) { + assert(value_type == CF_DATA_TYPE_STRING_LIST); SINGLE_COPY_LIST = value; SINGLE_COPY_CACHE = StringSetNew(); - Log(LOG_LEVEL_VERBOSE, "Setting file single copy list"); + if (WouldLog(LOG_LEVEL_VERBOSE)) + { + char *rlist_str = RlistToString(SINGLE_COPY_LIST); + Log(LOG_LEVEL_VERBOSE, "Setting file single copy list to: %s", rlist_str); + free(rlist_str); + } continue; } From b784c2fee3ceb206966e151cd46618e6b10286ef Mon Sep 17 00:00:00 2001 From: flynn1973 Date: Tue, 27 Apr 2021 12:19:57 +0200 Subject: [PATCH 209/333] CFE-3625: Set Filedescriptor Limit to a more practial Size Ticket: CFE-3625 Changelog: Title (cherry picked from commit 5cbd7f58a7daf118728501cdfa15459b7cf89145) --- libpromises/pipes_unix.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libpromises/pipes_unix.c b/libpromises/pipes_unix.c index 153a85c0e8..f19ac82509 100644 --- a/libpromises/pipes_unix.c +++ b/libpromises/pipes_unix.c @@ -39,7 +39,7 @@ static bool CfSetuid(uid_t uid, gid_t gid); static int cf_pwait(pid_t pid); static pid_t *CHILDREN = NULL; /* GLOBAL_X */ -static int MAX_FD = 128; /* GLOBAL_X */ /* Max number of simultaneous pipes */ +static int MAX_FD = 2048; /* GLOBAL_X */ /* Max number of simultaneous pipes */ static void ChildrenFDInit() From 808c1aa1e32909a9d1cd0d1f8f935109fc9b85fb Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Mon, 26 Apr 2021 10:55:41 -0500 Subject: [PATCH 210/333] Added examples illustrating inherit_from Ticket: CFE-3636 Changelog: None (cherry picked from commit 4528cbc7b5e1df1d0e15545a605bdc1e05805e1b) --- examples/inherit_from.cf | 55 ++++++++++++++++++++++++++++++++ examples/inherit_from_classes.cf | 52 ++++++++++++++++++++++++++++++ 2 files changed, 107 insertions(+) create mode 100644 examples/inherit_from.cf create mode 100644 examples/inherit_from_classes.cf diff --git a/examples/inherit_from.cf b/examples/inherit_from.cf new file mode 100644 index 0000000000..2882b7189a --- /dev/null +++ b/examples/inherit_from.cf @@ -0,0 +1,55 @@ +# This example illustrates the use of inherit_from to inherit body attribute +# values from another body. +############################################################################### +#+begin_src cfengine3 +bundle agent __main__ +{ + files: + "$(this.promise_filename).txt" + content => "Hello World$(const.n)2$(const.n)3$(const.n)4$(const.n)half-way +6$(const.n)7$(const.n)8$(const.n)9$(const.n)Byeeeeeee", + create => "true"; + + reports: + "The first 3 lines of this file are:" + printfile => head_n( "$(this.promise_filename).txt", "3" ); + + "The whole file contains:" + printfile => cat( "$(this.promise_filename).txt" ); +} + +body printfile head_n(file, lines) +{ + # Sets file_to_print the same as cat + inherit_from => cat( $(file) ); + + # Overrides number_of_lines from cat + number_of_lines => "$(lines)"; +} +body printfile cat(file) +{ + file_to_print => "$(file)"; + number_of_lines => "inf"; +} +############################################################################### +#+begin_src example_output +#@ ``` +#@ R: The first 3 lines of this file are: +#@ R: Hello World +#@ R: 2 +#@ R: 3 +#@ R: The whole file contains: +#@ R: Hello World +#@ R: 2 +#@ R: 3 +#@ R: 4 +#@ R: half-way +#@ R: 6 +#@ R: 7 +#@ R: 8 +#@ R: 9 +#@ R: Byeeeeeee +#@ ``` +#+end_src + + diff --git a/examples/inherit_from_classes.cf b/examples/inherit_from_classes.cf new file mode 100644 index 0000000000..5e1debb83c --- /dev/null +++ b/examples/inherit_from_classes.cf @@ -0,0 +1,52 @@ +# This example illustrates the use of inherit_from to inherit body attribute +# values from another body. +############################################################################### +#+begin_src cfengine3 +bundle agent __main__ +{ + commands: + "/bin/true" + handle => "some meaningful string", + classes => my_set_some_extra_fancy_classes( "$(this.promiser)", + "$(this.handle)", + "some_class_to_handle_dependencies" ); + + "/bin/false" + handle => "some meaningless string", + classes => my_set_some_extra_fancy_classes( "$(this.promiser)", + "$(this.handle)", + "some_class_to_handle_dependencies" ); + + reports: + "Defined classes:$(const.n)$(with)" + with => join( "$(const.n)", sort( classesmatching( "some_.*"), "lex" )); +} + +body classes my_set_some_extra_fancy_classes(x, y, z) +# @brief In addition to the classes set by `set_some_fancy_classes` define `z` when the promise is repaired +{ + inherit_from => set_some_fancy_classes( $(x), $(y) ); + promise_repaired => { "some_fancy_class_${x}_${y}_repaired", $(z) }; +} + +body classes set_some_fancy_classes(x, y) +# @brief Define a class prefixed with `some_fancy_class_` followed by expansion +# of `x`_`y` and suffixed with the promise outcome for each promise outcome. +{ + promise_kept => { "some_fancy_class_${x}_${y}_kept" }; + promise_repaired => { "some_fancy_class_${x}_${y}_repaired" }; + repair_failed => { "some_fancy_class_${x}_${y}_notkept" }; + repair_denied => { "some_fancy_class_${x}_${y}_notkept" }; + repair_timeout => { "some_fancy_class_${x}_${y}_notkept" }; +} +#+end_src +############################################################################### +#+begin_src example_output +#@ ``` +#@ error: Finished command related to promiser '/bin/false' -- an error occurred, returned 1 +#@ R: Defined classes: +#@ some_class_to_handle_dependencies +#@ some_fancy_class__bin_false_some_meaningless_string_notkept +#@ some_fancy_class__bin_true_some_meaningful_string_repaired +#@ ``` +#+end_src From 7b5241df7df118e4376a650c30b212c7e01891b4 Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Tue, 27 Apr 2021 14:20:45 +0200 Subject: [PATCH 211/333] ENT-6941 Increased range of allowed values of number of processes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Issue was that even with current ±10% allowed spread of number of processes enumerated by CFEngine vs `ps` this test was failing sometimes, with errors like this: > 27 is not within the range 21-26 Changelog: None (cherry picked from commit 2c90edf9678d825ea23a37353aa904797cbb71b1) --- tests/unit/mon_processes_test.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/unit/mon_processes_test.c b/tests/unit/mon_processes_test.c index 2e72cda72a..50b59fc620 100644 --- a/tests/unit/mon_processes_test.c +++ b/tests/unit/mon_processes_test.c @@ -125,8 +125,8 @@ void test_processes_monitor(void) " the two numbers should be *about* the same since the 'ps'" " commands run very close to each other"); - int upper = (int) ((double) usr*1.10); - int lower = (int) ((double) usr*0.90); + int upper = (int) ((double) usr*1.20); + int lower = (int) ((double) usr*0.80); assert_in_range((long long) cf_this[ob_users], lower, upper); } From 08af4e8a7a0266b6b94321b5941c8c879a8c2f95 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Tue, 4 May 2021 14:32:44 -0500 Subject: [PATCH 212/333] Made backwards compatible content was not introduced until 3.16.0, this change switched to (cherry picked from commit b7d684cc40944c9fc0d82772dad25581099d3753) --- examples/inherit_from.cf | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/examples/inherit_from.cf b/examples/inherit_from.cf index 2882b7189a..6ba58378a7 100644 --- a/examples/inherit_from.cf +++ b/examples/inherit_from.cf @@ -6,9 +6,10 @@ bundle agent __main__ { files: "$(this.promise_filename).txt" - content => "Hello World$(const.n)2$(const.n)3$(const.n)4$(const.n)half-way -6$(const.n)7$(const.n)8$(const.n)9$(const.n)Byeeeeeee", - create => "true"; + create => "true", + template_method => "inline_mustache", + edit_template_string => string_mustache( "Hello World$(const.n)2$(const.n)3$(const.n)4$(const.n)half-way +6$(const.n)7$(const.n)8$(const.n)9$(const.n)Byeeeeeee", '{}'); reports: "The first 3 lines of this file are:" From 7f025423809aad089205bcf1a9d2106d661b4e8a Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 12 May 2021 00:21:02 +0200 Subject: [PATCH 213/333] ENT-7051: Removed codecov (3.15) Signed-off-by: Ole Herman Schumacher Elgesem --- README.md | 1 - codecov.yaml | 23 ----------------------- libntech | 2 +- travis-scripts/after_success.sh | 5 +---- travis-scripts/before_install.sh | 2 +- 5 files changed, 3 insertions(+), 30 deletions(-) delete mode 100644 codecov.yaml diff --git a/README.md b/README.md index 4f27270b26..2a0207bde8 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,6 @@ | 3.12.x LTS | [![Core Build Status](https://travis-ci.org/cfengine/core.svg?branch=3.12.x)](https://travis-ci.org/cfengine/core) | [![MPF Build Status](https://travis-ci.org/cfengine/masterfiles.svg?branch=3.12.x)](https://travis-ci.org/cfengine/masterfiles) | | 3.10.x LTS | [![Core Build Status](https://travis-ci.org/cfengine/core.svg?branch=3.10.x)](https://travis-ci.org/cfengine/core) | [![MPF Build Status](https://travis-ci.org/cfengine/masterfiles.svg?branch=3.10.x)](https://travis-ci.org/cfengine/masterfiles) | -[![codecov](https://codecov.io/gh/cfengine/core/branch/master/graph/badge.svg)](https://codecov.io/gh/cfengine/core) [![Language grade: C](https://img.shields.io/lgtm/grade/cpp/g/cfengine/core.svg?logo=lgtm&logoWidth=18&label=code%20quality)](https://lgtm.com/projects/g/cfengine/core/) # CFEngine 3 diff --git a/codecov.yaml b/codecov.yaml deleted file mode 100644 index bc2cb8e90e..0000000000 --- a/codecov.yaml +++ /dev/null @@ -1,23 +0,0 @@ -codecov: - notify: - require_ci_to_pass: yes - -coverage: - precision: 2 - round: down - range: 50..75 - - status: - project: yes - patch: yes - changes: yes - -# comment: -# layout: "header, diff, tree, changes, reach, footer" -# behavior: default - -comment: off - -# Ignore source files connected to tests. -ignore: - - "tests/" diff --git a/libntech b/libntech index 4e9efcb841..874405f95f 160000 --- a/libntech +++ b/libntech @@ -1 +1 @@ -Subproject commit 4e9efcb84172110fa92742836b8d34688983c2e7 +Subproject commit 874405f95f300f04fd0d0a018262a0f046226012 diff --git a/travis-scripts/after_success.sh b/travis-scripts/after_success.sh index 8ad32da643..6515e76fd1 100755 --- a/travis-scripts/after_success.sh +++ b/travis-scripts/after_success.sh @@ -1,8 +1,5 @@ #!/bin/sh -# For systems with sudo access, make all files readable so that codecovarage +# For systems with sudo access, make all files readable so that code covarage # data is readable even for tests that were run as root. sudo chown -R $USER . || true - -# Code coverage by codecov.io -curl -s https://codecov.io/bash | bash diff --git a/travis-scripts/before_install.sh b/travis-scripts/before_install.sh index 2230a0d0c2..41a34d73c8 100755 --- a/travis-scripts/before_install.sh +++ b/travis-scripts/before_install.sh @@ -26,7 +26,7 @@ else sudo apt-get install -y fakeroot # Optional sudo apt-get install -y libxml2-dev libacl1-dev - # codecov.io dependency + # Code coverage dependency sudo apt-get install -y lcov # Ensure traditional yacc compatibility sudo apt-get purge -y bison From 170cafcecffb55e4bc2365681d06b1e33925a403 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 12 May 2021 18:40:19 +0200 Subject: [PATCH 214/333] Reverted incompatible libntech update Back to how it was before: https://github.com/cfengine/core/pull/4644/commits/7f025423809aad089205bcf1a9d2106d661b4e8a Signed-off-by: Ole Herman Schumacher Elgesem --- libntech | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libntech b/libntech index 874405f95f..4e9efcb841 160000 --- a/libntech +++ b/libntech @@ -1 +1 @@ -Subproject commit 874405f95f300f04fd0d0a018262a0f046226012 +Subproject commit 4e9efcb84172110fa92742836b8d34688983c2e7 From 3f78eb53f8a52c4b0a724ea7eeae205a814e1a79 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Wed, 28 Apr 2021 16:47:11 -0500 Subject: [PATCH 215/333] Marked acceptance tests flakey for appropriate platforms 00_basics/ifelapsed_and_expireafter/locks.cf for solaris11sparc 00_basics/ifelapsed_and_expireafter/redmine_6334.cf for solaris11sparc 01_vars/01_basic/os_version_major.cf for centos5 05_processes/01_matching/inverse_ttime_inverse_found.cf for aix 7.1 Changelog: none Ticket: ENT-7048 (cherry picked from commit 001130f3facb2c6a0df331d05b2c745fbcc7cbc4) Conflicts: tests/acceptance/00_basics/ifelapsed_and_expireafter/locks.cf tests/acceptance/01_vars/01_basic/os_version_major.cf --- .../00_basics/ifelapsed_and_expireafter/redmine_6334.cf | 4 ++++ .../05_processes/01_matching/inverse_ttime_inverse_found.cf | 3 +++ 2 files changed, 7 insertions(+) diff --git a/tests/acceptance/00_basics/ifelapsed_and_expireafter/redmine_6334.cf b/tests/acceptance/00_basics/ifelapsed_and_expireafter/redmine_6334.cf index 392060461f..e6dace6ae4 100644 --- a/tests/acceptance/00_basics/ifelapsed_and_expireafter/redmine_6334.cf +++ b/tests/acceptance/00_basics/ifelapsed_and_expireafter/redmine_6334.cf @@ -7,6 +7,10 @@ body common control bundle agent test { + meta: + "test_flakey_fail" string => "sunos_5_11.sparc", + meta => { "ENT-7068" }; + vars: "first" string => diff --git a/tests/acceptance/05_processes/01_matching/inverse_ttime_inverse_found.cf b/tests/acceptance/05_processes/01_matching/inverse_ttime_inverse_found.cf index 6b80570a97..f7896b9e47 100644 --- a/tests/acceptance/05_processes/01_matching/inverse_ttime_inverse_found.cf +++ b/tests/acceptance/05_processes/01_matching/inverse_ttime_inverse_found.cf @@ -28,6 +28,9 @@ bundle agent test "test_suppress_fail" string => "windows", meta => { "redmine4747" }; + "test_flakey_fail" string => "aix_7_1", + meta => { "CFE-3313" }; + processes: ".*" process_count => test_range, From 4082c80ed0498856ce1063900526c51ab7e2a82f Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 29 Apr 2021 15:20:15 +0200 Subject: [PATCH 216/333] Explain why indexless var references are looked up sometimes One less '/* why? TODO comment */' comments in the code. Ticket: ENT-7029 Changelog: None (cherry picked from commit 39293f1198141299f9cbde58d1ad646601e3e158) --- libpromises/eval_context.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/libpromises/eval_context.c b/libpromises/eval_context.c index 36b3208bef..001dc3ec1c 100644 --- a/libpromises/eval_context.c +++ b/libpromises/eval_context.c @@ -2208,8 +2208,12 @@ static Variable *VariableResolve2(const EvalContext *ctx, const VarRef *ref) { return var; } - else if (ref->num_indices > 0) /* why? TODO comment */ + else if (ref->num_indices > 0) { + /* If the lookup with indices (the [idx1][idx2]... part of the + * variable reference) fails, there might still be a container + * variable where the indices actually refer to child objects inside + * the container structure. */ VarRef *base_ref = VarRefCopyIndexless(ref); var = VariableTableGet(table, base_ref); VarRefDestroy(base_ref); From 0295f8b96523a29b181255381a36d3733746f2a4 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 29 Apr 2021 16:13:23 +0200 Subject: [PATCH 217/333] Only use one 'Variable *' variable in VariableResolve Using multiple variables just makes the code confusing because it looks like they might be used further in the code, but only one is always eventually used, actually. Ticket: ENT-7029 Changelog: None (cherry picked from commit 3c08f6e629f088271bde1df99c9db907a29de04e) --- libpromises/eval_context.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/libpromises/eval_context.c b/libpromises/eval_context.c index 001dc3ec1c..11abac5772 100644 --- a/libpromises/eval_context.c +++ b/libpromises/eval_context.c @@ -2244,10 +2244,10 @@ static Variable *VariableResolve(const EvalContext *ctx, const VarRef *ref) /* We will make a first lookup that works in almost all cases: will look * for local or global variables, depending of the current scope. */ - Variable *var1 = VariableResolve2(ctx, ref); - if (var1) + Variable *ret_var = VariableResolve2(ctx, ref); + if (ret_var != NULL) { - return var1; + return ret_var; } /* Try to qualify non-scoped vars to the scope: @@ -2257,11 +2257,11 @@ static Variable *VariableResolve(const EvalContext *ctx, const VarRef *ref) { scoped_ref = VarRefCopy(ref); VarRefStackQualify(ctx, scoped_ref); - Variable *var2 = VariableResolve2(ctx, scoped_ref); - if (var2) + ret_var = VariableResolve2(ctx, scoped_ref); + if (ret_var != NULL) { VarRefDestroy(scoped_ref); - return var2; + return ret_var; } ref = scoped_ref; /* continue with the scoped variable */ } From 778bc66661b24f8df0d0dcac72f6deac440c6bb4 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 29 Apr 2021 10:11:22 +0200 Subject: [PATCH 218/333] Do not create iteration variables in foreign bundles When iterating over slists, temporary variables with the slist reference replaced by the values from the slist are created in the 'this.' scope. For example for the following slist: "list" slist => { "item1", "item2" } and the following variable expression: "$(some_data[$(list)]" the following temporary helper variables are created: "$(this.some_data[item1])" "$(this.some_data[item2])" with the respective values, which are then used as replacements for the variable references in the individual iterations. That introduces a problem if the variable reference actually contains a scope as well, e.g. "config.data[$(list)]", because there cannot be a variable with two scopes (like "this.config.data[item1]"). Instead of creating variables in the foreign scope (bundle), like "config.data[item1]" and "config.data[item2]", let's mangle the variable reference to "config___data[$(list)]" and then create and lookup variables "this.config___data[item1]" and "this.config___data[item1]". Variables from the 'this' scope are automatically cleaned when leaving the current scope (bundle, body,...). Ticket: ENT-7029 Changelog: "source=promise_iteration" variables are no longer created in foreign bundles (cherry picked from commit dceef83b6336efaf1e3b2ef55505f25dda14665d) Conflicts: libpromises/eval_context.c --- libpromises/eval_context.c | 106 ++++++++++++++++++++++++++++++++++--- 1 file changed, 100 insertions(+), 6 deletions(-) diff --git a/libpromises/eval_context.c b/libpromises/eval_context.c index 11abac5772..de4aafe68b 100644 --- a/libpromises/eval_context.c +++ b/libpromises/eval_context.c @@ -52,6 +52,10 @@ #include /* DataTypeIsIterable */ #include +/* If we need to put a scoped variable into a special scope, use the string + * below to replace the original scope separator. + * (e.g. "config.var" -> "this.config___var" ) */ +#define NESTED_SCOPE_SEP "___" static const char *STACK_FRAME_TYPE_STR[STACK_FRAME_TYPE_MAX] = { "BUNDLE", @@ -1941,24 +1945,65 @@ StringSet *EvalContextStackPromisees(const EvalContext *ctx) return promisees; } +/** + * We cannot have double-scoped variables (e.g. "this.config.var1"), so if we + * want to put a scoped variable into a special scope, we need to mangle the + * name like this: + * "config.var1" -> "config___var1" + */ +static inline char *MangleScopedVarNameIntoSpecialScopeName(const char *scope, const char *var_name) +{ + const size_t var_name_len = strlen(var_name); + + /* Replace '.' with NESTED_SCOPE_SEP */ + char *new_var_name = xmalloc(var_name_len + sizeof(NESTED_SCOPE_SEP)); + memcpy(new_var_name, var_name, var_name_len + 1 /* including '\0' */); + + /* Make sure we only replace the "scope." string, not all dots. */ + char *scope_with_dot = StringConcatenate(2, scope, "."); + char *scope_with_underscores = StringConcatenate(2, scope, NESTED_SCOPE_SEP); + + ssize_t ret = StringReplace(new_var_name, var_name_len + sizeof(NESTED_SCOPE_SEP), + scope_with_dot, scope_with_underscores); + assert(ret == (var_name_len + sizeof(NESTED_SCOPE_SEP) - 2)); + + free(scope_with_dot); + free(scope_with_underscores); + + return new_var_name; +} + /* * Copies value, so you need to free your own copy afterwards. */ bool EvalContextVariablePutSpecial(EvalContext *ctx, SpecialScope scope, const char *lval, const void *value, DataType type, const char *tags) { + char *new_lval = NULL; + if (strchr(lval, '.') != NULL) + { + VarRef *ref = VarRefParse(lval); + if (ref->scope != NULL) + { + new_lval = MangleScopedVarNameIntoSpecialScopeName(ref->scope, lval); + } + VarRefDestroy(ref); + } if (strchr(lval, '[')) { // dealing with (legacy) array reference in lval, must parse - VarRef *ref = VarRefParseFromScope(lval, SpecialScopeToString(scope)); + VarRef *ref = VarRefParseFromScope(new_lval ? new_lval : lval, SpecialScopeToString(scope)); bool ret = EvalContextVariablePut(ctx, ref, value, type, tags); + free(new_lval); VarRefDestroy(ref); return ret; } else { // plain lval, skip parsing - const VarRef ref = VarRefConst(NULL, SpecialScopeToString(scope), lval); - return EvalContextVariablePut(ctx, &ref, value, type, tags); + const VarRef ref = VarRefConst(NULL, SpecialScopeToString(scope), new_lval ? new_lval : lval); + bool ret = EvalContextVariablePut(ctx, &ref, value, type, tags); + free(new_lval); + return ret; } } @@ -2195,8 +2240,29 @@ bool EvalContextVariablePut(EvalContext *ctx, return true; } +/** + * Change ref for e.g. 'config.var1' to 'this.config___var1' + * + * @see MangleScopedVarNameIntoSpecialScopeName() + */ +static inline VarRef *MangledThisScopedRef(const VarRef *ref) +{ + VarRef *mangled_this_ref = VarRefCopy(ref); + char *scope_underscores_lval = StringConcatenate(3, mangled_this_ref->scope, + NESTED_SCOPE_SEP, + mangled_this_ref->lval); + free(mangled_this_ref->lval); + mangled_this_ref->lval = scope_underscores_lval; + free(mangled_this_ref->scope); + mangled_this_ref->scope = xstrdup("this"); + + return mangled_this_ref; +} + static Variable *VariableResolve2(const EvalContext *ctx, const VarRef *ref) { + assert(ref != NULL); + // Get the variable table associated to the scope VariableTable *table = GetVariableTableForScope(ctx, ref->ns, ref->scope); @@ -2210,6 +2276,34 @@ static Variable *VariableResolve2(const EvalContext *ctx, const VarRef *ref) } else if (ref->num_indices > 0) { + /* Iteration over slists creates special variables in the 'this.' + * scope with the slist variable replaced by the individual + * values. However, if a scoped variable is part of the variable + * reference, e.g. 'config.data[$(list)]', the special iteration + * variables use mangled names to avoid having two scopes + * (e.g. 'this.config___data[list_item1]' instead of + * 'this.config.data[list_item1]'). + * + * If the ref we are looking for has indices and it has a scope, it + * might be the case described above. Let's give it a try before + * falling back to the indexless container lookup described below + * (which will not have the list-iteration variables expanded). */ + if (ref->scope != NULL) + { + VariableTable *this_table = GetVariableTableForScope(ctx, ref->ns, + SpecialScopeToString(SPECIAL_SCOPE_THIS)); + if (this_table != NULL) + { + VarRef *mangled_this_ref = MangledThisScopedRef(ref); + var = VariableTableGet(this_table, mangled_this_ref); + VarRefDestroy(mangled_this_ref); + if (var != NULL) + { + return var; + } + } + } + /* If the lookup with indices (the [idx1][idx2]... part of the * variable reference) fails, there might still be a container * variable where the indices actually refer to child objects inside @@ -2278,14 +2372,14 @@ static Variable *VariableResolve(const EvalContext *ctx, const VarRef *ref) { VarRef *ref2 = VarRefCopy(ref); VarRefQualify(ref2, last_bundle->ns, last_bundle->name); - Variable *var3 = VariableResolve2(ctx, ref2); + ret_var = VariableResolve2(ctx, ref2); VarRefDestroy(scoped_ref); VarRefDestroy(ref2); - return var3; + return ret_var; } - VarRefDestroy(scoped_ref); + return NULL; } From cf818d76446a13297fad69d52de5229cdc1fddd0 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 14 May 2021 11:59:04 +0200 Subject: [PATCH 219/333] Add a new 'a_entry_required' parameter to DeleteDigestFromLastSeen() The lastseen local DB has entries of multiple types. In particular, there are k-entries mapping hostkeys to IP addresses and a-entries for the reverse mapping of IP addresses to hostkeys. The ideal state is to have the entries of those two types coherent (bijective), but it's not always possible. For this reason, we need to be able to delete k-entries even if there is no matching a-entry. Do not require a-entries when deleting digests in RemoveKeysFromLastSeen() if must_be_coherent is false. Ticket: ENT-6490 Changelog: None (cherry picked from commit 93b778ca3aa60e130fc5d063dee9744d398a683d) --- libpromises/lastseen.c | 8 +++++--- libpromises/lastseen.h | 2 +- tests/unit/lastseen_test.c | 25 ++++++++++++++++++++++++- 3 files changed, 30 insertions(+), 5 deletions(-) diff --git a/libpromises/lastseen.c b/libpromises/lastseen.c index 8182276d18..0327c76fcb 100644 --- a/libpromises/lastseen.c +++ b/libpromises/lastseen.c @@ -510,9 +510,11 @@ bool DeleteIpFromLastSeen(const char *ip, char *digest, size_t digest_size) * @param[in,out] ip : return the key corresponding host. * If NULL, return nothing * @param[in] ip_size : length of ip parameter + * @param[in] a_entry_required : whether 'aIP_ADDR' entry is required for + * the 'kHOSTKEY' entry deletion * @retval true if entry was deleted, false otherwise */ -bool DeleteDigestFromLastSeen(const char *key, char *ip, size_t ip_size) +bool DeleteDigestFromLastSeen(const char *key, char *ip, size_t ip_size, bool a_entry_required) { DBHandle *db; bool res = false; @@ -535,7 +537,7 @@ bool DeleteDigestFromLastSeen(const char *key, char *ip, size_t ip_size) { strcpy(bufhost, "a"); strlcat(bufhost, host, CF_BUFSIZE); - if (HasKeyDB(db, bufhost, strlen(bufhost) + 1) == false) + if (a_entry_required && !HasKeyDB(db, bufhost, strlen(bufhost) + 1)) { res = false; goto clean; @@ -710,7 +712,7 @@ int RemoveKeysFromLastSeen(const char *input, bool must_be_coherent, if (is_digest == true) { Log(LOG_LEVEL_VERBOSE, "Removing digest '%s' from lastseen database\n", input); - if (DeleteDigestFromLastSeen(input, equivalent, equivalent_size) == false) + if (!DeleteDigestFromLastSeen(input, equivalent, equivalent_size, must_be_coherent)) { Log(LOG_LEVEL_ERR, "Unable to remove digest from lastseen database."); return 252; diff --git a/libpromises/lastseen.h b/libpromises/lastseen.h index 6460ce311e..75f697ba11 100644 --- a/libpromises/lastseen.h +++ b/libpromises/lastseen.h @@ -46,7 +46,7 @@ void LastSaw1(const char *ipaddress, const char *hashstr, LastSeenRole role); void LastSaw(const char *ipaddress, const char *digest, LastSeenRole role); bool DeleteIpFromLastSeen(const char *ip, char *digest, size_t digest_size); -bool DeleteDigestFromLastSeen(const char *key, char *ip, size_t ip_size); +bool DeleteDigestFromLastSeen(const char *key, char *ip, size_t ip_size, bool a_entry_required); /* * Return false in order to stop iteration diff --git a/tests/unit/lastseen_test.c b/tests/unit/lastseen_test.c index c6a0d548e8..1aa108223b 100644 --- a/tests/unit/lastseen_test.c +++ b/tests/unit/lastseen_test.c @@ -178,7 +178,7 @@ static void test_remove(void) UpdateLastSawHost("SHA-12345", "127.0.0.64", false, 556); //RemoveHostFromLastSeen("SHA-12345"); - DeleteDigestFromLastSeen("SHA-12345", NULL, 0); + DeleteDigestFromLastSeen("SHA-12345", NULL, 0, true); DBHandle *db; OpenDB(&db, dbid_lastseen); @@ -191,6 +191,28 @@ static void test_remove(void) CloseDB(db); } +static void test_remove_no_a_entry(void) +{ + setup(); + + UpdateLastSawHost("SHA-12345", "127.0.0.64", true, 555); + UpdateLastSawHost("SHA-12345", "127.0.0.64", false, 556); + + DBHandle *db; + OpenDB(&db, dbid_lastseen); + + assert_true(DeleteDB(db, "a127.0.0.64")); + assert_false(DeleteDigestFromLastSeen("SHA-12345", NULL, 0, true)); + assert_true(DeleteDigestFromLastSeen("SHA-12345", NULL, 0, false)); + + assert_false(HasKeyDB(db, "qiSHA-12345", strlen("qiSHA-12345") + 1)); + assert_false(HasKeyDB(db, "qoSHA-12345", strlen("qoSHA-12345") + 1)); + assert_false(HasKeyDB(db, "kSHA-12345", strlen("kSHA-12345") + 1)); + assert_false(HasKeyDB(db, "a127.0.0.64", strlen("a127.0.0.64") + 1)); + + CloseDB(db); +} + static void test_remove_ip(void) { setup(); @@ -588,6 +610,7 @@ int main() unit_test(test_reverse_conflict), unit_test(test_reverse_missing_forward), unit_test(test_remove), + unit_test(test_remove_no_a_entry), unit_test(test_remove_ip), unit_test_setup_teardown(test_consistent_1a, begin, end), From 2aaa453f2de6e410e2c03c6151b475a29c02fe88 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Mon, 24 May 2021 15:36:39 +0200 Subject: [PATCH 220/333] Do not assert namespace is not specified for 'def.' variables 'def' is a pretty common name for a bundle and so using it outside of the default namespace should be allowed. It doesn't produce an error in PolicyCheckBundle() and referring to variables from a 'def' bundle in a non-default namespace should just work. Ticket: CFE-3668 Changelog: None (cherry picked from commit cfb55d6dd4f5f26ca6a9b89b9b5afe630aad7646) --- libpromises/eval_context.c | 7 ++- .../00_basics/04_bundles/non_default_def.cf | 45 +++++++++++++++++++ 2 files changed, 51 insertions(+), 1 deletion(-) create mode 100644 tests/acceptance/00_basics/04_bundles/non_default_def.cf diff --git a/libpromises/eval_context.c b/libpromises/eval_context.c index de4aafe68b..c5c4adad76 100644 --- a/libpromises/eval_context.c +++ b/libpromises/eval_context.c @@ -2071,10 +2071,15 @@ static VariableTable *GetVariableTableForScope(const EvalContext *ctx, const char *ns, const char *scope) { + assert(ctx != NULL); + switch (SpecialScopeFromString(scope)) { - case SPECIAL_SCOPE_SYS: case SPECIAL_SCOPE_DEF: + /* 'def.' is not as special as the other scopes below. (CFE-3668) */ + return ctx->global_variables; + + case SPECIAL_SCOPE_SYS: case SPECIAL_SCOPE_MON: case SPECIAL_SCOPE_CONST: assert(!ns || strcmp("default", ns) == 0); diff --git a/tests/acceptance/00_basics/04_bundles/non_default_def.cf b/tests/acceptance/00_basics/04_bundles/non_default_def.cf new file mode 100644 index 0000000000..59f99b5e41 --- /dev/null +++ b/tests/acceptance/00_basics/04_bundles/non_default_def.cf @@ -0,0 +1,45 @@ +# Test that 'def' bundle in a non-default namespace works + +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; +} + +body file control +{ + namespace => "my_ns"; +} + +bundle common def +{ + vars: + "some_var" string => "some value"; +} + +body file control +{ + namespace => "default"; +} + +bundle agent init +{ +} + +bundle agent test +{ +} + +bundle agent check +{ + classes: + "ok" expression => strcmp("$(my_ns:def.some_var)", "some value"); + + reports: + ok:: + "$(this.promise_filename) Pass"; + !ok.DEBUG:: + "my_ns:def.some_var: $(my_ns:def.some_var)"; + !ok:: + "$(this.promise_filename) FAIL"; +} From 00daf4aab3e6adea0eb76477d12a235d8b3b1405 Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Mon, 31 May 2021 15:00:38 +0200 Subject: [PATCH 221/333] Added changelog for 3.15.4 --- ChangeLog | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/ChangeLog b/ChangeLog index b624eb5c32..785e789d3e 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,41 @@ +3.15.4: + - "source=promise_iteration" variables are no longer created in foreign bundles + (ENT-7029) + - Added flakey test result handling to testall (ENT-6935) + - Added flakey test result handling to testall (ENT-6935) + - Added warning when trying to use {{.}} to expand containers in mustache templates + (CFE-3457, CFE-3489) + - Body + Ticket: (CFE-3525) + - Changed log message about whitespace in class expressions to be error + (CFE-3560) + - Clarified error log message about untrusted state directory not being private + (CFE-3599) + - Disable testall selftest (ENT-6935) + - Exit code from remote agent run is now sent to cf-runagent (CFE-3594) + - New observations of root owned SETUID programs moved from WARN to NOTICE + (ENT-6519) + - Policy function format() no longer truncates strings lager than 4KiB + (CFE-2686) + - Policy function storejson() no longer truncates strings lager than 4096 bytes + (CFE-2507) + - Removed codecov (3.15) (ENT-7051) + - Restored log level for failure to state remote files without missing_ok to ERROR + (ENT-5879) + - Set Filedescriptor Limit to a more practial Size (CFE-3625) + - Stopped emitting warning and recording result when observing new SETGID files + (ENT-6750) + - Value of the 'files_single_copy' body control attribute is now logged in verbose logging mode + (CFE-3622) + - cf-runagent now exits with a code reflecting remote agent run status(es) + (CFE-3594) + - disable defaults.cf on all platforms (ENT-6939) + - disable package_lock.cf on all platforms (ENT-6939) + - files_single_copy body agent control attribute can now be an empty list + (CFE-3622) + - files_single_copy no longer treats paths of copied files as regular expressions + (CFE-3621) + 3.15.3: - cf-secret binary for managing secrets was added (CFE-2613) - Added sys.cf_secret component variable From e7fb678da0e734f5a9756c189bc165399958300a Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Mon, 31 May 2021 16:57:25 +0200 Subject: [PATCH 222/333] Update ChangeLog --- ChangeLog | 41 +++++++++++++++++------------------------ 1 file changed, 17 insertions(+), 24 deletions(-) diff --git a/ChangeLog b/ChangeLog index 785e789d3e..991fd89ae3 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,40 +1,33 @@ 3.15.4: - - "source=promise_iteration" variables are no longer created in foreign bundles - (ENT-7029) - - Added flakey test result handling to testall (ENT-6935) - - Added flakey test result handling to testall (ENT-6935) + - cf-runagent now exits with a code reflecting remote agent run status(es) + (CFE-3594) + - Increased file descriptor limit from 128 to 2048 (CFE-3625) - Added warning when trying to use {{.}} to expand containers in mustache templates (CFE-3457, CFE-3489) - - Body - Ticket: (CFE-3525) - - Changed log message about whitespace in class expressions to be error + - Changed log message about whitespace in class expressions to be errors (CFE-3560) - - Clarified error log message about untrusted state directory not being private + - Clarified error message about untrusted state directory not being private (CFE-3599) - - Disable testall selftest (ENT-6935) - - Exit code from remote agent run is now sent to cf-runagent (CFE-3594) - New observations of root owned SETUID programs moved from WARN to NOTICE (ENT-6519) - - Policy function format() no longer truncates strings lager than 4KiB - (CFE-2686) - - Policy function storejson() no longer truncates strings lager than 4096 bytes - (CFE-2507) - - Removed codecov (3.15) (ENT-7051) - - Restored log level for failure to state remote files without missing_ok to ERROR - (ENT-5879) - - Set Filedescriptor Limit to a more practial Size (CFE-3625) - Stopped emitting warning and recording result when observing new SETGID files (ENT-6750) + - Restored log level for failure to state remote files without missing_ok to ERROR + (ENT-5879) - Value of the 'files_single_copy' body control attribute is now logged in verbose logging mode (CFE-3622) - - cf-runagent now exits with a code reflecting remote agent run status(es) - (CFE-3594) - - disable defaults.cf on all platforms (ENT-6939) - - disable package_lock.cf on all platforms (ENT-6939) - - files_single_copy body agent control attribute can now be an empty list + - Value of the 'files_single_copy' body control attribute can now be an empty list (CFE-3622) - - files_single_copy no longer treats paths of copied files as regular expressions + - 'files_single_copy' no longer treats paths of copied files as regular expressions (CFE-3621) + - Fixed vulnerabillity where malformed input could trigger buffer overflow in + format() policy function (CFE-3525) + - Policy function format() no longer truncates strings lager than 4096 bytes + (CFE-2686) + - Policy function storejson() no longer truncates strings lager than 4096 bytes + (CFE-2507) + - "source=promise_iteration" variables are no longer created in foreign bundles + (ENT-7029) 3.15.3: - cf-secret binary for managing secrets was added (CFE-2613) From 5cd3e470d85aba773b9ecb32ce6d71c3bee530b6 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 26 May 2021 01:15:31 +0200 Subject: [PATCH 223/333] Small refactorings done to package module code Discovered while looking for a memory leak. Changelog: None Ticket: ENT-5752 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 982edcd6cdbd8227736561362a9cca77c6734d47) --- libpromises/eval_context.c | 13 ++++++++----- libpromises/expand.c | 2 ++ 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/libpromises/eval_context.c b/libpromises/eval_context.c index c5c4adad76..d36074d1d9 100644 --- a/libpromises/eval_context.c +++ b/libpromises/eval_context.c @@ -225,15 +225,18 @@ int PackageManagerSeqCompare(const void *a, const void *b, ARG_UNUSED void *data void AddPackageModuleToContext(const EvalContext *ctx, PackageModuleBody *pm) { + assert(ctx != NULL); + assert(pm != NULL); + /* First check if the body is there added from previous pre-evaluation * iteration. If it is there update it as we can have new expanded variables. */ - ssize_t pm_seq_index; - if ((pm_seq_index = SeqIndexOf(ctx->package_promise_context->package_modules_bodies, - pm->name, PackageManagerSeqCompare)) != -1) + Seq *const bodies = ctx->package_promise_context->package_modules_bodies; + ssize_t index = SeqIndexOf(bodies, pm->name, PackageManagerSeqCompare); + if (index != -1) { - SeqRemove(ctx->package_promise_context->package_modules_bodies, pm_seq_index); + SeqRemove(bodies, index); } - SeqAppend(ctx->package_promise_context->package_modules_bodies, pm); + SeqAppend(bodies, pm); } PackageModuleBody *GetPackageModuleFromContext(const EvalContext *ctx, diff --git a/libpromises/expand.c b/libpromises/expand.c index 0827f76dff..2f03b46c3e 100644 --- a/libpromises/expand.c +++ b/libpromises/expand.c @@ -1044,10 +1044,12 @@ static void ResolvePackageManagerBody(EvalContext *ctx, const Body *pm_body) } else if (strcmp(cp->lval, "interpreter") == 0) { + assert(new_manager->interpreter == NULL); new_manager->interpreter = SafeStringDuplicate(RvalScalarValue(returnval)); } else if (strcmp(cp->lval, "module_path") == 0) { + assert(new_manager->module_path == NULL); new_manager->module_path = SafeStringDuplicate(RvalScalarValue(returnval)); } else From 7d6f906197813300465db336a9483b089b1118aa Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 26 May 2021 01:16:33 +0200 Subject: [PATCH 224/333] Fixed memory leak in package module code Changelog: Title Ticket: ENT-5752 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit 5297f7c409e290ce389a74988e156dda85ce7a4a) --- libpromises/eval_context.c | 4 ++++ tests/unit/new_packages_promise_test.c | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/libpromises/eval_context.c b/libpromises/eval_context.c index d36074d1d9..1ab5a1ff10 100644 --- a/libpromises/eval_context.c +++ b/libpromises/eval_context.c @@ -950,7 +950,11 @@ static void StackFrameDestroy(StackFrame *frame) static void FreePackageManager(PackageModuleBody *manager) { + assert(manager != NULL); + free(manager->name); + free(manager->interpreter); + free(manager->module_path); RlistDestroy(manager->options); free(manager); } diff --git a/tests/unit/new_packages_promise_test.c b/tests/unit/new_packages_promise_test.c index 0a6dd30bab..926597471e 100644 --- a/tests/unit/new_packages_promise_test.c +++ b/tests/unit/new_packages_promise_test.c @@ -6,7 +6,7 @@ static inline PackageModuleBody *make_mock_package_module(const char *name, int updates_ifel, int installed_ifel, Rlist *options) { - PackageModuleBody *pm = xmalloc(sizeof(PackageModuleBody)); + PackageModuleBody *pm = xcalloc(1, sizeof(PackageModuleBody)); pm->name = SafeStringDuplicate(name); pm->installed_ifelapsed = installed_ifel; pm->updates_ifelapsed = updates_ifel; From fd636f37c461bbba2d0d4f8c1173a7b81b97e442 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Tue, 1 Jun 2021 16:01:30 +0200 Subject: [PATCH 225/333] Pass program name to openlog() As man:openlog(3) says: The string pointed to by 'ident' is prepended to every message, and is typically set to the program name. If 'ident' is NULL, the program name is used. (POSIX.1-2008 does not specify the behavior when ident is NULL.) And of course not all systems we support are nice and use program name if we pass NULL as the 'ident' argument. Ticket: ENT-7100 Changelog: CFEngine processes are now properly identified in syslog on non-GNU/Linux systems (cherry picked from commit 5147f4a93a87b4ba651c6f2c3fbe1853ad1b0a86) --- cf-agent/cf-agent.c | 5 ++++- cf-execd/cf-execd.c | 7 +++++-- cf-key/cf-key.c | 6 +++++- cf-monitord/cf-monitord.c | 7 ++++++- cf-promises/cf-promises.c | 7 ++++++- cf-runagent/cf-runagent.c | 6 +++++- cf-serverd/cf-serverd-functions.c | 2 +- cf-serverd/cf-serverd.c | 5 ++++- libpromises/generic_agent.c | 12 ++++++++++-- libpromises/generic_agent.h | 2 +- 10 files changed, 47 insertions(+), 12 deletions(-) diff --git a/cf-agent/cf-agent.c b/cf-agent/cf-agent.c index ff999a3621..73f102c6af 100644 --- a/cf-agent/cf-agent.c +++ b/cf-agent/cf-agent.c @@ -258,7 +258,10 @@ int main(int argc, char *argv[]) GenericAgentConfigApply(ctx, config); - GenericAgentDiscoverContext(ctx, config); + const char *program_invocation_name = argv[0]; + const char *last_dir_sep = strrchr(program_invocation_name, FILE_SEPARATOR); + const char *program_name = (last_dir_sep != NULL ? last_dir_sep + 1 : program_invocation_name); + GenericAgentDiscoverContext(ctx, config, program_name); /* FIXME: (CFE-2709) ALWAYS_VALIDATE will always be false here, since it can * only change in KeepPromises(), five lines later on. */ diff --git a/cf-execd/cf-execd.c b/cf-execd/cf-execd.c index 8f524267fb..d59422f1f4 100644 --- a/cf-execd/cf-execd.c +++ b/cf-execd/cf-execd.c @@ -145,7 +145,10 @@ int main(int argc, char *argv[]) EvalContext *ctx = EvalContextNew(); GenericAgentConfigApply(ctx, config); - GenericAgentDiscoverContext(ctx, config); + const char *program_invocation_name = argv[0]; + const char *last_dir_sep = strrchr(program_invocation_name, FILE_SEPARATOR); + const char *program_name = (last_dir_sep != NULL ? last_dir_sep + 1 : program_invocation_name); + GenericAgentDiscoverContext(ctx, config, program_name); Policy *policy = SelectAndLoadPolicy(config, ctx, false, false); @@ -671,7 +674,7 @@ static bool ScheduleRun(EvalContext *ctx, Policy **policy, GenericAgentConfig *c UpdateLastPolicyUpdateTime(ctx); DetectEnvironment(ctx); - GenericAgentDiscoverContext(ctx, config); + GenericAgentDiscoverContext(ctx, config, NULL); EvalContextClassPutHard(ctx, CF_AGENTTYPES[AGENT_TYPE_EXECUTOR], "cfe_internal,source=agent"); diff --git a/cf-key/cf-key.c b/cf-key/cf-key.c index 9c43226498..07bc22eeef 100644 --- a/cf-key/cf-key.c +++ b/cf-key/cf-key.c @@ -38,6 +38,7 @@ #include #include #include +#include /* FILE_SEPARATOR */ #include #include @@ -145,7 +146,10 @@ int main(int argc, char *argv[]) EvalContext *ctx = EvalContextNew(); GenericAgentConfigApply(ctx, config); - GenericAgentDiscoverContext(ctx, config); + const char *program_invocation_name = argv[0]; + const char *last_dir_sep = strrchr(program_invocation_name, FILE_SEPARATOR); + const char *program_name = (last_dir_sep != NULL ? last_dir_sep + 1 : program_invocation_name); + GenericAgentDiscoverContext(ctx, config, program_name); if (SHOWHOSTS) { diff --git a/cf-monitord/cf-monitord.c b/cf-monitord/cf-monitord.c index b38e708e83..cd078cd2da 100644 --- a/cf-monitord/cf-monitord.c +++ b/cf-monitord/cf-monitord.c @@ -38,6 +38,7 @@ #include #include #include +#include /* FILE_SEPARATOR */ typedef enum { @@ -120,7 +121,11 @@ int main(int argc, char *argv[]) EvalContext *ctx = EvalContextNew(); GenericAgentConfigApply(ctx, config); - GenericAgentDiscoverContext(ctx, config); + const char *program_invocation_name = argv[0]; + const char *last_dir_sep = strrchr(program_invocation_name, FILE_SEPARATOR); + const char *program_name = (last_dir_sep != NULL ? last_dir_sep + 1 : program_invocation_name); + GenericAgentDiscoverContext(ctx, config, program_name); + Policy *policy = LoadPolicy(ctx, config); GenericAgentPostLoadInit(ctx); diff --git a/cf-promises/cf-promises.c b/cf-promises/cf-promises.c index 118d7005e7..70810122e4 100644 --- a/cf-promises/cf-promises.c +++ b/cf-promises/cf-promises.c @@ -33,6 +33,7 @@ #include #include #include +#include /* FILE_SEPARATOR */ #include #include /* CompileRegex */ #include @@ -135,7 +136,11 @@ int main(int argc, char *argv[]) EvalContext *ctx = EvalContextNew(); GenericAgentConfigApply(ctx, config); - GenericAgentDiscoverContext(ctx, config); + const char *program_invocation_name = argv[0]; + const char *last_dir_sep = strrchr(program_invocation_name, FILE_SEPARATOR); + const char *program_name = (last_dir_sep != NULL ? last_dir_sep + 1 : program_invocation_name); + GenericAgentDiscoverContext(ctx, config, program_name); + Policy *policy = LoadPolicy(ctx, config); if (!policy) { diff --git a/cf-runagent/cf-runagent.c b/cf-runagent/cf-runagent.c index d4fbcf2466..c41459ac73 100644 --- a/cf-runagent/cf-runagent.c +++ b/cf-runagent/cf-runagent.c @@ -207,7 +207,11 @@ int main(int argc, char *argv[]) EvalContext *ctx = EvalContextNew(); GenericAgentConfigApply(ctx, config); - GenericAgentDiscoverContext(ctx, config); + const char *program_invocation_name = argv[0]; + const char *last_dir_sep = strrchr(program_invocation_name, FILE_SEPARATOR); + const char *program_name = (last_dir_sep != NULL ? last_dir_sep + 1 : program_invocation_name); + GenericAgentDiscoverContext(ctx, config, program_name); + Policy *policy = LoadPolicy(ctx, config); GenericAgentPostLoadInit(ctx); diff --git a/cf-serverd/cf-serverd-functions.c b/cf-serverd/cf-serverd-functions.c index 85092b8656..4d774fb80f 100644 --- a/cf-serverd/cf-serverd-functions.c +++ b/cf-serverd/cf-serverd-functions.c @@ -439,7 +439,7 @@ static void CheckFileChanges(EvalContext *ctx, Policy **policy, GenericAgentConf UpdateLastPolicyUpdateTime(ctx); DetectEnvironment(ctx); - GenericAgentDiscoverContext(ctx, config); + GenericAgentDiscoverContext(ctx, config, NULL); /* During startup this is done in GenericAgentDiscoverContext(). */ EvalContextClassPutHard(ctx, CF_AGENTTYPES[AGENT_TYPE_SERVER], "cfe_internal,source=agent"); diff --git a/cf-serverd/cf-serverd.c b/cf-serverd/cf-serverd.c index 603ee6713f..ffa16ccb09 100644 --- a/cf-serverd/cf-serverd.c +++ b/cf-serverd/cf-serverd.c @@ -56,7 +56,10 @@ int main(int argc, char *argv[]) EvalContext *ctx = EvalContextNew(); GenericAgentConfigApply(ctx, config); - GenericAgentDiscoverContext(ctx, config); + const char *program_invocation_name = argv[0]; + const char *last_dir_sep = strrchr(program_invocation_name, FILE_SEPARATOR); + const char *program_name = (last_dir_sep != NULL ? last_dir_sep + 1 : program_invocation_name); + GenericAgentDiscoverContext(ctx, config, program_name); Policy *policy = SelectAndLoadPolicy(config, ctx, false, false); diff --git a/libpromises/generic_agent.c b/libpromises/generic_agent.c index 7e6862ccbf..5e03559d97 100644 --- a/libpromises/generic_agent.c +++ b/libpromises/generic_agent.c @@ -77,6 +77,9 @@ static pthread_once_t pid_cleanup_once = PTHREAD_ONCE_INIT; /* GLOBAL_T */ static char PIDFILE[CF_BUFSIZE] = ""; /* GLOBAL_C */ +/* Used for 'ident' argument to openlog() */ +static char CF_PROGRAM_NAME[256] = ""; + static void CheckWorkingDirectories(EvalContext *ctx); static void GetAutotagDir(char *dirname, size_t max_size, const char *maybe_dirname); @@ -516,9 +519,14 @@ static void AddPolicyEntryVariables (EvalContext *ctx, const GenericAgentConfig free(basename_path); } -void GenericAgentDiscoverContext(EvalContext *ctx, GenericAgentConfig *config) +void GenericAgentDiscoverContext(EvalContext *ctx, GenericAgentConfig *config, + const char *program_name) { strcpy(VPREFIX, ""); + if (program_name != NULL) + { + strncpy(CF_PROGRAM_NAME, program_name, sizeof(CF_PROGRAM_NAME)); + } Log(LOG_LEVEL_VERBOSE, " %s", NameVersion()); Banner("Initialization preamble"); @@ -979,7 +987,7 @@ bool GenericAgentArePromisesValid(const GenericAgentConfig *config) #if !defined(__MINGW32__) static void OpenLog(int facility) { - openlog(NULL, LOG_PID | LOG_NOWAIT | LOG_ODELAY, facility); + openlog(CF_PROGRAM_NAME, LOG_PID | LOG_NOWAIT | LOG_ODELAY, facility); } #endif diff --git a/libpromises/generic_agent.h b/libpromises/generic_agent.h index 4ccc122bc6..db510ce456 100644 --- a/libpromises/generic_agent.h +++ b/libpromises/generic_agent.h @@ -106,7 +106,7 @@ typedef struct ENTERPRISE_VOID_FUNC_2ARG_DECLARE(void, GenericAgentSetDefaultDigest, HashMethod *, digest, int *, digest_len); const char *GenericAgentResolveInputPath(const GenericAgentConfig *config, const char *input_file); void MarkAsPolicyServer(EvalContext *ctx); -void GenericAgentDiscoverContext(EvalContext *ctx, GenericAgentConfig *config); +void GenericAgentDiscoverContext(EvalContext *ctx, GenericAgentConfig *config, const char *program_name); bool GenericAgentCheckPolicy(GenericAgentConfig *config, bool force_validation, bool write_validated_file); ENTERPRISE_VOID_FUNC_1ARG_DECLARE(void, GenericAgentAddEditionClasses, EvalContext *, ctx); From 8a616939ac1eb4253c13d5d09b69a720fc5e208e Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 21 Jan 2021 11:50:43 +0100 Subject: [PATCH 226/333] Add missing SELinux rules for PID files manipulation cf-hub and PostgreSQL (systemd) need to check, remove and create their PID files. (cherry picked from commit 208da1b7758b626afa43a56d83c30d3ea83c41db) --- misc/selinux/cfengine-enterprise.te | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index f10cb7b842..499ab84f83 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -349,7 +349,7 @@ allow cfengine_hub_t unreserved_port_t:tcp_socket name_connect; allow cfengine_hub_t cfengine_log_t:dir getattr; allow cfengine_hub_t cfengine_var_lib_t:dir { add_name getattr open read search write remove_name }; -allow cfengine_hub_t cfengine_var_lib_t:file { create ioctl lock write }; +allow cfengine_hub_t cfengine_var_lib_t:file { create ioctl lock write unlink }; allow cfengine_hub_t cfengine_var_lib_t:lnk_file { getattr read }; allow cfengine_hub_t cfengine_var_lib_t:sock_file { create unlink }; @@ -460,8 +460,8 @@ allow cfengine_postgres_t tmpfs_t:filesystem getattr; allow cfengine_postgres_t var_log_t:file { append open }; # Needed for systemd to be able to check PostgreSQL's PID file -allow init_t cfengine_var_lib_t:dir read; -allow init_t cfengine_var_lib_t:file { getattr open read }; +allow init_t cfengine_var_lib_t:dir { read remove_name write }; +allow init_t cfengine_var_lib_t:file { getattr open read unlink }; # TODO: these should not be needed allow cfengine_postgres_t shell_exec_t:file map; From 0ca4ffe08b1aaa73a571a36a6aca6064a98c8a31 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 3 Jun 2021 18:09:24 +0200 Subject: [PATCH 227/333] Two additional SELinux allow rules Needed for our apache process to work properly. (cherry picked from commit ab1ad1c94da4ae7904defbc38cedba74341b7d51) --- misc/selinux/cfengine-enterprise.te | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 499ab84f83..f581535332 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -137,7 +137,7 @@ require { class dccp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; class ib_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; class mpls_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; - class process { setrlimit transition dyntransition execstack execheap execmem }; + class process { setrlimit transition dyntransition execstack execheap execmem signull }; class file { execute execute_no_trans getattr ioctl map open read unlink write entrypoint lock link rename append setattr create relabelfrom relabelto }; class fifo_file { create open getattr setattr read write append rename link unlink ioctl lock relabelfrom relabelto }; class dir { getattr read search open write add_name remove_name lock ioctl create }; @@ -496,7 +496,7 @@ allow cfengine_httpd_t cfengine_postgres_t:unix_stream_socket connectto; # allow httpd to use our custom compiled module allow cfengine_httpd_t cfengine_var_lib_t:file map; -allow cfengine_httpd_t cfengine_var_lib_t:file { append create execute getattr ioctl lock open read setattr unlink write }; +allow cfengine_httpd_t cfengine_var_lib_t:file { append create execute getattr ioctl lock open read setattr unlink write rename }; allow cfengine_httpd_t cfengine_var_lib_t:dir { add_name getattr open read remove_name search write create }; allow cfengine_httpd_t cfengine_var_lib_t:lnk_file read; @@ -513,6 +513,7 @@ allow cfengine_httpd_t node_t:tcp_socket node_bind; allow cfengine_httpd_t self:capability { dac_override dac_read_search kill net_bind_service setgid setuid }; allow cfengine_httpd_t self:netlink_route_socket { bind create getattr nlmsg_read }; allow cfengine_httpd_t self:process execmem; +allow cfengine_httpd_t unconfined_t:process signull; allow cfengine_httpd_t self:tcp_socket { accept bind connect create getattr getopt listen setopt shutdown }; allow cfengine_httpd_t self:udp_socket { connect create getattr }; allow cfengine_httpd_t self:unix_dgram_socket { connect create }; From b9702ba6d65a66292f726058b86aa44df04de122 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Tue, 8 Jun 2021 14:52:07 +0200 Subject: [PATCH 228/333] Bumped .CFVERSION number to 3.15.5 Signed-off-by: Ole Herman Schumacher Elgesem --- .CFVERSION | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.CFVERSION b/.CFVERSION index 8f6e357b33..9658bd085c 100644 --- a/.CFVERSION +++ b/.CFVERSION @@ -1 +1 @@ -3.15.4 +3.15.5 From 5d0677e82228c52123ee6362d818c763cfbccc24 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 10 Jun 2021 09:07:07 +0200 Subject: [PATCH 229/333] Do openlog() before any logging happens Ticket: ENT-6955 Changelog: None (cherry picked from commit 08fc8fa922a078b077fd3c9468efdb2db9796906) Conflicts: libpromises/generic_agent.c --- libpromises/generic_agent.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/libpromises/generic_agent.c b/libpromises/generic_agent.c index 5e03559d97..f344064d40 100644 --- a/libpromises/generic_agent.c +++ b/libpromises/generic_agent.c @@ -1035,6 +1035,9 @@ void GenericAgentInitialize(EvalContext *ctx, GenericAgentConfig *config) DetermineCfenginePort(); + OpenLog(LOG_USER); + SetSyslogFacility(LOG_USER); + EvalContextClassPutHard(ctx, "any", "source=agent"); GenericAgentAddEditionClasses(ctx); @@ -1049,9 +1052,6 @@ void GenericAgentInitialize(EvalContext *ctx, GenericAgentConfig *config) FatalError(ctx, "Error determining working directory"); } - OpenLog(LOG_USER); - SetSyslogFacility(LOG_USER); - Log(LOG_LEVEL_VERBOSE, "Work directory is %s", workdir); snprintf(vbuff, CF_BUFSIZE, "%s%cupdate.conf", GetInputDir(), FILE_SEPARATOR); From ba09eab18605e66c2fe519ed7e383c8200c1ed0c Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 10 Jun 2021 10:35:46 +0200 Subject: [PATCH 230/333] Store log facility for processes and use it as defautl on startup CFEngine processes can use various log facilities based on the policy (body control attributes). However, some logging happens even before the policy is loaded and processed. To avoid using wrong facility before the policy is loaded, we store the previous log facility for the given process and then load it very early during process initialization to ensure even the early logging happens with the (previously) desired facility. Ticket: ENT-6955 Changelog: CFEngine processes now reuse log facility from previous run for early logging before policy is loaded (cherry picked from commit 0e062bbca3dd089710c882cf745fde5a59fc2e4a) Conflicts: libpromises/generic_agent.c --- libpromises/generic_agent.c | 82 ++++++++++++++++++++++++++++++++++++- libpromises/syslog_client.c | 5 +++ libpromises/syslog_client.h | 1 + 3 files changed, 86 insertions(+), 2 deletions(-) diff --git a/libpromises/generic_agent.c b/libpromises/generic_agent.c index f344064d40..774fe04826 100644 --- a/libpromises/generic_agent.c +++ b/libpromises/generic_agent.c @@ -95,6 +95,9 @@ static bool MissingInputFile(const char *input_file); static bool LoadAugmentsFiles(EvalContext *ctx, const char* filename); +static int ParseFacility(const char *name); +static inline const char *LogFacilityToString(int facility); + #if !defined(__MINGW32__) static void OpenLog(int facility); #endif @@ -1005,6 +1008,58 @@ ENTERPRISE_VOID_FUNC_1ARG_DEFINE_STUB(void, GenericAgentAddEditionClasses, EvalC EvalContextClassPutHard(ctx, "community_edition", "inventory,attribute_name=none,source=agent"); } +static int GetDefaultLogFacility() +{ + char log_facility_file[PATH_MAX]; + int written = snprintf(log_facility_file, sizeof(log_facility_file) - 1, + "%s%c%s_log_facility.dat", GetStateDir(), + FILE_SEPARATOR, CF_PROGRAM_NAME); + assert(written < PATH_MAX); + if (access(log_facility_file, R_OK) != 0) + { + return LOG_USER; + } + + FILE *f = fopen(log_facility_file, "r"); + if (f == NULL) + { + return LOG_USER; + } + char facility_str[16] = {0}; /* at most "LOG_DAEMON\n" */ + size_t n_read = fread(facility_str, 1, sizeof(facility_str) - 1, f); + fclose(f); + if (n_read == 0) + { + return LOG_USER; + } + if (facility_str[n_read - 1] == '\n') + { + facility_str[n_read - 1] = '\0'; + } + return ParseFacility(facility_str); +} + +static bool StoreDefaultLogFacility() +{ + char log_facility_file[PATH_MAX]; + int written = snprintf(log_facility_file, sizeof(log_facility_file) - 1, + "%s%c%s_log_facility.dat", GetStateDir(), + FILE_SEPARATOR, CF_PROGRAM_NAME); + assert(written < PATH_MAX); + + FILE *f = fopen(log_facility_file, "w"); + if (f == NULL) + { + return false; + } + const char *facility_str = LogFacilityToString(GetSyslogFacility()); + int printed = fprintf(f, "%s\n", facility_str); + assert(printed > 0); + + fclose(f); + return true; +} + void GenericAgentInitialize(EvalContext *ctx, GenericAgentConfig *config) { int force = false; @@ -1035,8 +1090,9 @@ void GenericAgentInitialize(EvalContext *ctx, GenericAgentConfig *config) DetermineCfenginePort(); - OpenLog(LOG_USER); - SetSyslogFacility(LOG_USER); + int default_facility = GetDefaultLogFacility(); + OpenLog(default_facility); + SetSyslogFacility(default_facility); EvalContextClassPutHard(ctx, "any", "source=agent"); @@ -1592,6 +1648,24 @@ static int ParseFacility(const char *name) return -1; } +static inline const char *LogFacilityToString(int facility) +{ + switch(facility) + { + case LOG_LOCAL0: return "LOG_LOCAL0"; + case LOG_LOCAL1: return "LOG_LOCAL1"; + case LOG_LOCAL2: return "LOG_LOCAL2"; + case LOG_LOCAL3: return "LOG_LOCAL3"; + case LOG_LOCAL4: return "LOG_LOCAL4"; + case LOG_LOCAL5: return "LOG_LOCAL5"; + case LOG_LOCAL6: return "LOG_LOCAL6"; + case LOG_LOCAL7: return "LOG_LOCAL7"; + case LOG_USER: return "LOG_USER"; + case LOG_DAEMON: return "LOG_DAEMON"; + default: return "UNKNOWN"; + } +} + void SetFacility(const char *retval) { Log(LOG_LEVEL_VERBOSE, "SET Syslog FACILITY = %s", retval); @@ -1599,6 +1673,10 @@ void SetFacility(const char *retval) CloseLog(); OpenLog(ParseFacility(retval)); SetSyslogFacility(ParseFacility(retval)); + if (!StoreDefaultLogFacility()) + { + Log(LOG_LEVEL_ERR, "Failed to store default log facility"); + } } static void CheckWorkingDirectories(EvalContext *ctx) diff --git a/libpromises/syslog_client.c b/libpromises/syslog_client.c index 2f22df9181..ca6b9a8bf7 100644 --- a/libpromises/syslog_client.c +++ b/libpromises/syslog_client.c @@ -46,6 +46,11 @@ void SetSyslogFacility(int facility) SYSLOG_FACILITY = facility; } +int GetSyslogFacility() +{ + return SYSLOG_FACILITY; +} + bool SetSyslogHost(const char *host) { if (strlen(host) < sizeof(SYSLOG_HOST)) diff --git a/libpromises/syslog_client.h b/libpromises/syslog_client.h index 1d1deb374c..a81906cfe4 100644 --- a/libpromises/syslog_client.h +++ b/libpromises/syslog_client.h @@ -34,6 +34,7 @@ bool SetSyslogHost(const char *host); void SetSyslogPort(uint16_t port); void SetSyslogFacility(int facility); +int GetSyslogFacility(); void RemoteSysLog(int log_priority, const char *log_string); From c8bd7813c106ea041dc5acd594147472b4a23f70 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 23 Jun 2021 14:51:22 +0200 Subject: [PATCH 231/333] SELinux: Allow cf-execd and cf-monitord getattr access to terminal devices Not sure why exactly these two daemons need the access, but it's definitely safe. Ticket: ENT-7340 Changelog: None (cherry picked from commit 374f9d9b012d3e637910f9cde04c1b2f87219cb3) --- misc/selinux/cfengine-enterprise.te | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index f581535332..2bb0b3f9d4 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -75,6 +75,8 @@ require { type tmp_t; type tmpfs_t; role system_r; + type tty_device_t; + type user_devpts_t; class tcp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown name_connect accept listen name_bind node_bind }; class udp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown node_bind }; class sock_file { create write setattr unlink }; @@ -151,6 +153,7 @@ require { class service { start stop status reload enable disable }; class memprotect mmap_zero; class peer recv; + class chr_file { getattr }; } @@ -228,6 +231,8 @@ allow cfengine_execd_t user_cron_spool_t:dir getattr; allow cfengine_execd_t useradd_exec_t:file getattr; allow cfengine_execd_t var_t:dir read; allow cfengine_execd_t semanage_exec_t:file getattr; +allow cfengine_execd_t tty_device_t:chr_file getattr; +allow cfengine_execd_t user_devpts_t:chr_file getattr; #============= cfengine_monitord_t ============== @@ -261,6 +266,8 @@ allow cfengine_monitord_t user_cron_spool_t:dir getattr; allow cfengine_monitord_t useradd_exec_t:file getattr; allow cfengine_monitord_t var_t:dir read; allow cfengine_monitord_t semanage_exec_t:file getattr; +allow cfengine_monitord_t tty_device_t:chr_file getattr; +allow cfengine_monitord_t user_devpts_t:chr_file getattr; # TODO: this should not be needed allow cfengine_monitord_t proc_xen_t:dir search; From 2e32ee6986a8ba9a09aa130bb4227ae2f288983b Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 23 Jun 2021 15:24:26 +0200 Subject: [PATCH 232/333] SELinux: Allow httpd, apachectl and PHP more access to sockets Ticket: ENT-7340 Changelog: None (cherry picked from commit 87825c18564e68438d1601764e165622d8708235) --- misc/selinux/cfengine-enterprise.te | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 2bb0b3f9d4..0c68d77374 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -512,16 +512,16 @@ allow cfengine_httpd_t devlog_t:lnk_file read; allow cfengine_httpd_t devlog_t:sock_file write; allow cfengine_httpd_t http_port_t:tcp_socket { name_bind name_connect }; allow cfengine_httpd_t init_t:dbus send_msg; -allow cfengine_httpd_t init_t:unix_stream_socket { getattr ioctl }; +allow cfengine_httpd_t init_t:unix_stream_socket { getattr ioctl read write }; # apachectl allow cfengine_httpd_t init_var_run_t:dir search; allow cfengine_httpd_t kernel_t:unix_dgram_socket sendto; allow cfengine_httpd_t net_conf_t:file { getattr open read }; allow cfengine_httpd_t node_t:tcp_socket node_bind; -allow cfengine_httpd_t self:capability { dac_override dac_read_search kill net_bind_service setgid setuid }; -allow cfengine_httpd_t self:netlink_route_socket { bind create getattr nlmsg_read }; +allow cfengine_httpd_t self:capability { dac_override dac_read_search kill net_bind_service setgid setuid net_admin }; +allow cfengine_httpd_t self:netlink_route_socket { bind create getattr nlmsg_read read write }; allow cfengine_httpd_t self:process execmem; allow cfengine_httpd_t unconfined_t:process signull; -allow cfengine_httpd_t self:tcp_socket { accept bind connect create getattr getopt listen setopt shutdown }; +allow cfengine_httpd_t self:tcp_socket { accept bind connect create getattr getopt listen setopt shutdown read write }; allow cfengine_httpd_t self:udp_socket { connect create getattr }; allow cfengine_httpd_t self:unix_dgram_socket { connect create }; allow cfengine_httpd_t sssd_public_t:dir search; From df8f2af02abe91ee8b1edca33b620843c93e14ee Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 23 Jun 2021 15:32:55 +0200 Subject: [PATCH 233/333] SELinux: Allow cf-hub more access to sockets Ticket: ENT-7340 Changelog: None (cherry picked from commit cc9d3ef009b28901b15cb1563f72cd242716716e) --- misc/selinux/cfengine-enterprise.te | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 0c68d77374..3e6455f346 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -374,7 +374,7 @@ allow cfengine_hub_t hostname_exec_t:file getattr; allow cfengine_hub_t init_exec_t:file getattr; allow cfengine_hub_t init_t:dir { getattr open read search }; allow cfengine_hub_t init_t:file { getattr open read }; -allow cfengine_hub_t init_t:unix_stream_socket ioctl; +allow cfengine_hub_t init_t:unix_stream_socket { ioctl getattr read write }; # systemd, PAM? allow cfengine_hub_t init_var_run_t:dir search; allow cfengine_hub_t journalctl_exec_t:file getattr; allow cfengine_hub_t kernel_t:unix_dgram_socket sendto; @@ -385,10 +385,10 @@ allow cfengine_hub_t proc_net_t:file { getattr open read }; allow cfengine_hub_t proc_t:dir read; allow cfengine_hub_t rpm_exec_t:file getattr; allow cfengine_hub_t self:capability dac_override; -allow cfengine_hub_t self:tcp_socket { connect create getopt setopt }; -allow cfengine_hub_t self:udp_socket { connect create getattr ioctl setopt }; +allow cfengine_hub_t self:tcp_socket { connect create getopt setopt read write }; +allow cfengine_hub_t self:udp_socket { connect create getattr ioctl setopt read write }; allow cfengine_hub_t self:netlink_route_socket { create getopt setopt bind getattr }; -allow cfengine_hub_t self:unix_dgram_socket { create connect }; +allow cfengine_hub_t self:unix_dgram_socket { create connect read write }; allow cfengine_hub_t semanage_exec_t:file getattr; allow cfengine_hub_t shadow_t:file getattr; allow cfengine_hub_t sssd_public_t:dir search; From 0a0468475f3a4c0e2a4fd5f6114139dc5186b837 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 23 Jun 2021 15:33:33 +0200 Subject: [PATCH 234/333] SELinux: Allow cf-monitord dir-read access to /etc/sysctl* Ticket: ENT-7340 Changelog: None (cherry picked from commit 9be7734810d18ec0e4ddb8f732fc88004b3ec4e1) --- misc/selinux/cfengine-enterprise.te | 2 ++ 1 file changed, 2 insertions(+) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 3e6455f346..49d8b6286e 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -77,6 +77,7 @@ require { role system_r; type tty_device_t; type user_devpts_t; + type sysctl_t; class tcp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown name_connect accept listen name_bind node_bind }; class udp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown node_bind }; class sock_file { create write setattr unlink }; @@ -268,6 +269,7 @@ allow cfengine_monitord_t var_t:dir read; allow cfengine_monitord_t semanage_exec_t:file getattr; allow cfengine_monitord_t tty_device_t:chr_file getattr; allow cfengine_monitord_t user_devpts_t:chr_file getattr; +allow cfengine_monitord_t sysctl_t:dir read; # TODO: this should not be needed allow cfengine_monitord_t proc_xen_t:dir search; From 59b7cbafb8b2c6fb873a963074645728d50da882 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 23 Jun 2021 15:36:23 +0200 Subject: [PATCH 235/333] SELinux: Allow postgres, pg_ctl and others more access to sockets Ticket: ENT-7340 Changelog: None (cherry picked from commit 95b6acec2168f4d915e42a04f206959132d89222) --- misc/selinux/cfengine-enterprise.te | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 49d8b6286e..d235369823 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -446,14 +446,14 @@ allow cfengine_postgres_t postgresql_port_t:tcp_socket name_bind; allow cfengine_postgres_t hugetlbfs_t:file map; allow cfengine_postgres_t hugetlbfs_t:file { read write }; -allow cfengine_postgres_t init_t:unix_stream_socket { getattr ioctl }; +allow cfengine_postgres_t init_t:unix_stream_socket { getattr ioctl read write }; # pg_ctl, systemd, PAM? allow cfengine_postgres_t net_conf_t:file { getattr open read }; allow cfengine_postgres_t node_t:tcp_socket node_bind; allow cfengine_postgres_t node_t:udp_socket node_bind; allow cfengine_postgres_t proc_t:file { getattr open read }; -allow cfengine_postgres_t self:netlink_route_socket { bind create getattr nlmsg_read }; -allow cfengine_postgres_t self:tcp_socket { bind create listen setopt }; -allow cfengine_postgres_t self:udp_socket { bind connect create getattr getopt }; +allow cfengine_postgres_t self:netlink_route_socket { bind create getattr nlmsg_read read write }; +allow cfengine_postgres_t self:tcp_socket { bind create listen setopt read write }; +allow cfengine_postgres_t self:udp_socket { bind connect create getattr getopt read write }; allow cfengine_postgres_t sssd_public_t:dir search; allow cfengine_postgres_t sssd_public_t:file map; allow cfengine_postgres_t sssd_public_t:file { getattr open read }; From 7a259c835eec4f3097026f77cbe0373027a81eb9 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 23 Jun 2021 15:40:57 +0200 Subject: [PATCH 236/333] SELinux: Allow cf-serverd access to /proc/1 Ticket: ENT-7340 Changelog: None (cherry picked from commit 986d45ac43ed75349361f4141111f6f07d7b34f7) --- misc/selinux/cfengine-enterprise.te | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index d235369823..4a264f1cdf 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -306,7 +306,7 @@ allow cfengine_serverd_t fs_t:filesystem getattr; allow cfengine_serverd_t groupadd_exec_t:file getattr; allow cfengine_serverd_t hostname_exec_t:file getattr; allow cfengine_serverd_t init_exec_t:file getattr; -allow cfengine_serverd_t init_t:dir read; +allow cfengine_serverd_t init_t:dir { getattr open search read } ; # /proc/1 analysis allow cfengine_serverd_t init_t:file { getattr open read }; allow cfengine_serverd_t journalctl_exec_t:file getattr; allow cfengine_serverd_t ping_exec_t:file getattr; From 68f35dd7982d92ef73a26a52fa6f4d91a91ad285 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 23 Jun 2021 15:41:26 +0200 Subject: [PATCH 237/333] SELinux: Allow our daemons to inherit signal handlers from systemd Ticket: ENT-7340 Changelog: None (cherry picked from commit 7085fa238c9b897d45b87c18249ceb3a681b13b0) --- misc/selinux/cfengine-enterprise.te | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 4a264f1cdf..6668a27205 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -140,7 +140,7 @@ require { class dccp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; class ib_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; class mpls_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; - class process { setrlimit transition dyntransition execstack execheap execmem signull }; + class process { setrlimit transition dyntransition execstack execheap execmem signull siginh }; class file { execute execute_no_trans getattr ioctl map open read unlink write entrypoint lock link rename append setattr create relabelfrom relabelto }; class fifo_file { create open getattr setattr read write append rename link unlink ioctl lock relabelfrom relabelto }; class dir { getattr read search open write add_name remove_name lock ioctl create }; @@ -339,6 +339,7 @@ role object_r types cfengine_hub_exec_t; type_transition init_t cfengine_hub_exec_t:process cfengine_hub_t; allow init_t cfengine_hub_t:process transition; allow init_t cfengine_hub_exec_t:file { execute open read }; +allow init_t cfengine_hub_t:process siginh; allow cfengine_hub_t cfengine_hub_exec_t:file entrypoint; allow cfengine_hub_t cfengine_hub_exec_t:file { ioctl read getattr lock map execute open }; @@ -432,6 +433,7 @@ role object_r types cfengine_postgres_exec_t; type_transition init_t cfengine_postgres_exec_t:process cfengine_postgres_t; allow init_t cfengine_postgres_t:process transition; allow init_t cfengine_postgres_exec_t:file { execute open read }; +allow init_t cfengine_postgres_t:process siginh; allow cfengine_postgres_t cfengine_postgres_exec_t:file entrypoint; allow cfengine_postgres_t cfengine_postgres_exec_t:file { ioctl read getattr lock map execute open }; @@ -493,6 +495,7 @@ role object_r types cfengine_httpd_exec_t; type_transition init_t cfengine_httpd_exec_t:process cfengine_httpd_t; allow init_t cfengine_httpd_t:process transition; allow init_t cfengine_httpd_exec_t:file { execute getattr open read }; +allow init_t cfengine_httpd_t:process siginh; allow cfengine_httpd_t cfengine_httpd_exec_t:file entrypoint; allow cfengine_httpd_t cfengine_httpd_exec_t:file { ioctl read getattr lock map execute open }; From 3101d717210b2d4f4c84aa10636be03e353d10ee Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Tue, 29 Jun 2021 20:55:58 +0200 Subject: [PATCH 238/333] Updated libntech to latest Signed-off-by: Ole Herman Schumacher Elgesem --- libntech | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libntech b/libntech index 4e9efcb841..10bd6288fd 160000 --- a/libntech +++ b/libntech @@ -1 +1 @@ -Subproject commit 4e9efcb84172110fa92742836b8d34688983c2e7 +Subproject commit 10bd6288fdef879f24d69032910cd9f002b624f2 From fb2f95e4fb4b03b3d99ab2f248c4329bf68e3ce3 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 17 Mar 2021 13:21:18 +0100 Subject: [PATCH 239/333] Add definitions for data directory Unfortunately, part of the definitions has to be duplicated here and in libntech. Ticket: ENT-6789 Changelog: None (cherry picked from commit 00bb3304ff871349eb1cfe2e4378f62e326b02f0) --- configure.ac | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/configure.ac b/configure.ac index 98f207f252..af3cc24885 100644 --- a/configure.ac +++ b/configure.ac @@ -190,6 +190,7 @@ AS_IF([test x"$enable_fhs" = xyes], [ WORKDIR='${localstatedir}/lib/${PACKAGE}' MASTERDIR='default' INPUTDIR='default' + DATADIR='default' LOGDIR='${localstatedir}/log/${PACKAGE}' PIDDIR='${runstatedir:-${localstatedir}/run}/${PACKAGE}' STATEDIR='default' @@ -201,6 +202,7 @@ AS_IF([test x"$enable_fhs" = xyes], [ WORKDIR=$(cmd /c "echo %PROGRAMFILES%\\Cfengine" | sed 's/\\/\\\\/g') MASTERDIR=default INPUTDIR=default + DATADIR=default LOGDIR=$(cmd /c "echo %PROGRAMFILES%\\Cfengine" | sed 's/\\/\\\\/g') PIDDIR=$(cmd /c "echo %PROGRAMFILES%\\Cfengine" | sed 's/\\/\\\\/g') STATEDIR=default @@ -209,6 +211,7 @@ AS_IF([test x"$enable_fhs" = xyes], [ WORKDIR=/var/cfengine MASTERDIR=default INPUTDIR=default + DATADIR=default LOGDIR=/var/cfengine PIDDIR=/var/cfengine STATEDIR=default @@ -218,6 +221,7 @@ AS_IF([test x"$enable_fhs" = xyes], [ WORKDIR="${localstatedir}/${PACKAGE}" MASTERDIR="default" INPUTDIR="default" + DATADIR="default" LOGDIR="${localstatedir}/${PACKAGE}" PIDDIR="${localstatedir}/${PACKAGE}" STATEDIR="default" @@ -1370,6 +1374,22 @@ adl_RECURSIVE_EVAL("${INPUTDIR}", INPUTDIR) AC_DEFINE_UNQUOTED(INPUTDIR, "${INPUTDIR}", [Inputs directory location]) AC_SUBST(inputdir, "${INPUTDIR}") +AC_ARG_WITH(datadir, + [ --with-datadir=DATADIR default for internal data directory ], + [ + if test "x$withval" != x ; then + DATADIR="$withval" + fi + ], +) + +dnl Expand ${prefix} and whatnot in DATADIR + +adl_RECURSIVE_EVAL("${DATADIR}", DATADIR) + +AC_DEFINE_UNQUOTED(DATADIR, "${DATADIR}", [Datadir location]) +AC_SUBST(datadir, "${DATADIR}") + AC_ARG_WITH(logdir, [ --with-logdir=LOGDIR default for internal log directory ], [ @@ -1746,6 +1766,7 @@ m4_indir(incstart[]incend, [nova/config.m4]) AC_MSG_RESULT([-> Workdir: $WORKDIR]) AC_MSG_RESULT([-> Masterdir: $MASTERDIR]) AC_MSG_RESULT([-> Inputdir: $INPUTDIR]) +AC_MSG_RESULT([-> Datadir: $DATADIR]) AC_MSG_RESULT([-> Logdir: $LOGDIR]) AC_MSG_RESULT([-> Piddir: $PIDDIR]) AC_MSG_RESULT([-> Statedir: $STATEDIR]) From 4981c261f251c7a914a6c9aaef5d4c9266fe6660 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 18 Mar 2021 15:52:37 +0100 Subject: [PATCH 240/333] Use a different name for the DATADIR macro Unfortunately, on mingw builds, 'DATADIR' is a name of an enum type. So we cannot use the same string as a name for our own macro. Let's use CF_DATADIR as the macro name which is also what libntech uses (changed in cfengine/libntech@ad5eeaa5969). Ticket: ENT-6789 Changelog: None (cherry picked from commit 932806f16c0d16bdb4602afaf3bd12dde8c341e2) --- configure.ac | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/configure.ac b/configure.ac index af3cc24885..35ffbcda5f 100644 --- a/configure.ac +++ b/configure.ac @@ -1387,7 +1387,8 @@ dnl Expand ${prefix} and whatnot in DATADIR adl_RECURSIVE_EVAL("${DATADIR}", DATADIR) -AC_DEFINE_UNQUOTED(DATADIR, "${DATADIR}", [Datadir location]) +dnl There's a conflict on mingw where they have a type called DATADIR! +AC_DEFINE_UNQUOTED(CF_DATADIR, "${DATADIR}", [Datadir location]) AC_SUBST(datadir, "${DATADIR}") AC_ARG_WITH(logdir, From 8ffe1768271b8540b0d743249d6fbafe82d27a58 Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Sun, 10 Jan 2021 16:32:51 +0100 Subject: [PATCH 241/333] Fixed some sign-compare warnings Ticket: CFE-3415 Changelog: None Signed-off-by: Lars Erik Wik (cherry picked from commit d926a2ce7b19261629f20b3ff029d051ee7b5620) --- cf-agent/verify_files.c | 2 +- cf-check/backup.c | 4 ++-- cf-check/diagnose.c | 4 ++-- cf-check/dump.c | 4 ++-- cf-check/lmdump.c | 2 +- cf-check/repair.c | 2 +- cf-check/utilities.c | 3 ++- libcfnet/client_code.c | 18 +++++++++++++++--- libcfnet/stat_cache.c | 6 +++++- libcfnet/tls_client.c | 12 ++++++++---- libenv/sysinfo.c | 4 ++-- libpromises/crypto.c | 32 +++++++++++++++++++++++++++----- libpromises/evalfunction.c | 2 +- libpromises/ornaments.c | 8 +++++++- libpromises/pipes.c | 17 +++++++++++++++-- libpromises/processes_select.c | 2 +- libpromises/rlist.c | 8 ++++---- libpromises/syntax.c | 2 +- libpromises/var_expressions.c | 2 +- libpromises/vars.c | 2 +- libpromises/verify_classes.c | 3 ++- 21 files changed, 101 insertions(+), 38 deletions(-) diff --git a/cf-agent/verify_files.c b/cf-agent/verify_files.c index aea86d211e..69b6b2c809 100644 --- a/cf-agent/verify_files.c +++ b/cf-agent/verify_files.c @@ -585,7 +585,7 @@ static bool SaveBufferCallback(const char *dest_filename, void *param, NewLineMo if (bytes_written != BufferSize(output_buffer)) { Log(LOG_LEVEL_ERR, - "Error writing to output file '%s' when writing. %zu bytes written but expected %u. (fclose: %s)", + "Error writing to output file '%s' when writing. %zu bytes written but expected %zu. (fclose: %s)", dest_filename, bytes_written, BufferSize(output_buffer), GetErrorStr()); fclose(fp); return false; diff --git a/cf-check/backup.c b/cf-check/backup.c index 02e8561eff..f37936a57b 100644 --- a/cf-check/backup.c +++ b/cf-check/backup.c @@ -106,7 +106,7 @@ int backup_files_copy(Seq *filenames) Log(LOG_LEVEL_INFO, "Backing up to '%s'", backup_dir); int ret = 0; - for (int i = 0; i < length; ++i) + for (size_t i = 0; i < length; ++i) { const char *file = SeqAt(filenames, i); if (!File_CopyToDir(file, backup_dir)) @@ -137,7 +137,7 @@ static int backup_files_replicate(const Seq *files) Log(LOG_LEVEL_INFO, "Backing up to '%s' using data replication", backup_dir); size_t corrupted = 0; - for (int i = 0; i < length; ++i) + for (size_t i = 0; i < length; ++i) { const char *file = SeqAt(files, i); assert(StringEndsWith(backup_dir, "/")); diff --git a/cf-check/diagnose.c b/cf-check/diagnose.c index 0c6020511b..7689bca611 100644 --- a/cf-check/diagnose.c +++ b/cf-check/diagnose.c @@ -515,7 +515,7 @@ static char *follow_symlink(const char *path) { return NULL; } - if (r >= sizeof(target_buf)) + if ((size_t) r >= sizeof(target_buf)) { Log(LOG_LEVEL_ERR, "Symlink target path too long: %s", path); return NULL; @@ -548,7 +548,7 @@ size_t diagnose_files( *corrupt = SeqNew(length, free); } - for (int i = 0; i < length; ++i) + for (size_t i = 0; i < length; ++i) { const char *filename = SeqAt(filenames, i); const char *symlink = NULL; // Only initialized because of gcc warning diff --git a/cf-check/dump.c b/cf-check/dump.c index bd3ffa2056..bbea0e289d 100644 --- a/cf-check/dump.c +++ b/cf-check/dump.c @@ -458,7 +458,7 @@ int dump_main(int argc, const char *const *const argv) dump_mode mode = DUMP_NICE; size_t offset = 1; - if (argc > offset && argv[offset] != NULL && argv[offset][0] == '-') + if ((size_t) argc > offset && argv[offset] != NULL && argv[offset][0] == '-') { const char *const option = argv[offset]; offset += 1; @@ -491,7 +491,7 @@ int dump_main(int argc, const char *const *const argv) } } - if (argc > offset && argv[offset] != NULL && argv[offset][0] == '-') + if ((size_t) argc > offset && argv[offset] != NULL && argv[offset][0] == '-') { print_usage(); printf("Only one option supported!\n"); diff --git a/cf-check/lmdump.c b/cf-check/lmdump.c index 65b528d1aa..8170103a13 100644 --- a/cf-check/lmdump.c +++ b/cf-check/lmdump.c @@ -10,7 +10,7 @@ static void lmdump_print_hex(const char *s, size_t len) { - for (int i = 0; i < len; i++) + for (size_t i = 0; i < len; i++) { printf("%02x", s[i]); } diff --git a/cf-check/repair.c b/cf-check/repair.c index c51f3145af..d2ad3409d6 100644 --- a/cf-check/repair.c +++ b/cf-check/repair.c @@ -249,7 +249,7 @@ int repair_lmdb_files(Seq *files, bool force) const size_t length = SeqLength(corrupt); assert(length > 0); backup_files_copy(corrupt); - for (int i = 0; i < length; ++i) + for (size_t i = 0; i < length; ++i) { const char *file = SeqAt(corrupt, i); if (repair_lmdb_file(file, -1) == -1) diff --git a/cf-check/utilities.c b/cf-check/utilities.c index 1ccf777bd5..add3cd848d 100644 --- a/cf-check/utilities.c +++ b/cf-check/utilities.c @@ -22,7 +22,8 @@ Seq *default_lmdb_files() Seq *argv_to_lmdb_files( const int argc, const char *const *const argv, const size_t offset) { - if (offset >= argc) + assert(argc >= 0); + if (offset >= (size_t) argc) { Log(LOG_LEVEL_INFO, "No filenames specified, defaulting to .lmdb files in %s", diff --git a/libcfnet/client_code.c b/libcfnet/client_code.c index d1a544d83d..e1d88865de 100644 --- a/libcfnet/client_code.c +++ b/libcfnet/client_code.c @@ -359,7 +359,11 @@ Item *RemoteDirList(const char *dirname, bool encrypt, AgentConnection *conn) tosend = cipherlen + CF_PROTO_OFFSET; - if(tosend > sizeof(sendbuffer)) + if (tosend < 0) + { + ProgrammingError("RemoteDirList: tosend (%d) < 0", tosend); + } + else if ((unsigned long) tosend > sizeof(sendbuffer)) { ProgrammingError("RemoteDirList: tosend (%d) > sendbuffer (%zd)", tosend, sizeof(sendbuffer)); @@ -495,7 +499,11 @@ bool CompareHashNet(const char *file1, const char *file2, bool encrypt, AgentCon tosend = cipherlen + CF_PROTO_OFFSET; - if(tosend > sizeof(sendbuffer)) + if (tosend < 0) + { + ProgrammingError("CompareHashNet: tosend (%d) < 0", tosend); + } + else if ((unsigned long) tosend > sizeof(sendbuffer)) { ProgrammingError("CompareHashNet: tosend (%d) > sendbuffer (%zd)", tosend, sizeof(sendbuffer)); @@ -589,7 +597,11 @@ static bool EncryptCopyRegularFileNet(const char *source, const char *dest, off_ tosend = cipherlen + CF_PROTO_OFFSET; - if(tosend > sizeof(workbuf)) + if (tosend < 0) + { + ProgrammingError("EncryptCopyRegularFileNet: tosend (%d) < 0", tosend); + } + else if ((unsigned long) tosend > sizeof(workbuf)) { ProgrammingError("EncryptCopyRegularFileNet: tosend (%d) > workbuf (%zd)", tosend, sizeof(workbuf)); diff --git a/libcfnet/stat_cache.c b/libcfnet/stat_cache.c index a3f2c0b013..21fa53dce8 100644 --- a/libcfnet/stat_cache.c +++ b/libcfnet/stat_cache.c @@ -165,7 +165,11 @@ int cf_remote_stat(AgentConnection *conn, bool encrypt, const char *file, tosend = cipherlen + CF_PROTO_OFFSET; - if(tosend > sizeof(sendbuffer)) + if (tosend < 0) + { + ProgrammingError("cf_remote_stat: tosend (%d) < 0", tosend); + } + else if((unsigned int) tosend > sizeof(sendbuffer)) { ProgrammingError("cf_remote_stat: tosend (%d) > sendbuffer (%zd)", tosend, sizeof(sendbuffer)); diff --git a/libcfnet/tls_client.c b/libcfnet/tls_client.c index 040239306a..16c6126496 100644 --- a/libcfnet/tls_client.c +++ b/libcfnet/tls_client.c @@ -250,7 +250,12 @@ int TLSClientIdentificationDialog(ConnectionInfo *conn_info, { ret = snprintf(&line[line_len], sizeof(line) - line_len, " USERNAME=%s", username); - if (ret >= sizeof(line) - line_len) + if (ret < 0) + { + Log(LOG_LEVEL_ERR, "snprintf failed: %s", GetErrorStr()); + return -1; + } + else if ((unsigned int) ret >= sizeof(line) - line_len) { Log(LOG_LEVEL_ERR, "Sending IDENTITY truncated: %s", line); return -1; @@ -275,14 +280,13 @@ int TLSClientIdentificationDialog(ConnectionInfo *conn_info, static const char OK[] = "OK WELCOME"; size_t OK_len = sizeof(OK) - 1; ret = TLSRecvLines(conn_info->ssl, line, sizeof(line)); - if (ret == -1) + if (ret < 0) { Log(LOG_LEVEL_ERR, "Connection was hung up during identification! (3)"); return -1; } - - if (ret < OK_len || strncmp(line, OK, OK_len) != 0) + else if ((size_t) ret < OK_len || strncmp(line, OK, OK_len) != 0) { Log(LOG_LEVEL_ERR, "Peer did not accept our identity! Responded: %s", diff --git a/libenv/sysinfo.c b/libenv/sysinfo.c index a3b251e6b1..6e10d62558 100644 --- a/libenv/sysinfo.c +++ b/libenv/sysinfo.c @@ -1928,7 +1928,7 @@ static int Linux_Redhat_Version(EvalContext *ctx) Scientific Linux don't fall through the cracks. */ - for (int i = 0; i < strlen(relstring); i++) + for (size_t i = 0; i < strlen(relstring); i++) { relstring[i] = tolower(relstring[i]); } @@ -2079,7 +2079,7 @@ static int Linux_Suse_Version(EvalContext *ctx) * SUSE with SUSE 10.0. */ - for (int i = 0; i < strlen(relstring); i++) + for (size_t i = 0; i < strlen(relstring); i++) { relstring[i] = tolower(relstring[i]); } diff --git a/libpromises/crypto.c b/libpromises/crypto.c index 29680bd715..3143ca0d0d 100644 --- a/libpromises/crypto.c +++ b/libpromises/crypto.c @@ -433,7 +433,12 @@ bool SavePublicKey(const char *user, const char *digest, const RSA *key) int ret; ret = snprintf(keyname, sizeof(keyname), "%s-%s", user, digest); - if (ret >= sizeof(keyname)) + if (ret < 0) + { + Log(LOG_LEVEL_ERR, "snprintf failed: %s", GetErrorStr()); + return false; + } + else if ((unsigned long) ret >= sizeof(keyname)) { Log(LOG_LEVEL_ERR, "USERNAME-KEY (%s-%s) string too long!", user, digest); @@ -442,7 +447,12 @@ bool SavePublicKey(const char *user, const char *digest, const RSA *key) ret = snprintf(filename, sizeof(filename), "%s/ppkeys/%s.pub", GetWorkDir(), keyname); - if (ret >= sizeof(filename)) + if (ret < 0) + { + Log(LOG_LEVEL_ERR, "snprintf failed: %s", GetErrorStr()); + return false; + } + else if ((unsigned long) ret >= sizeof(filename)) { Log(LOG_LEVEL_ERR, "Filename too long!"); return false; @@ -625,7 +635,11 @@ int EncryptString(char *out, size_t out_size, const char *in, int plainlen, cipherlen += tmplen; - if(cipherlen > max_ciphertext_size) + if (cipherlen < 0) + { + ProgrammingError("EncryptString: chipherlen (%d) < 0", cipherlen); + } + else if ((size_t) cipherlen > max_ciphertext_size) { ProgrammingError("EncryptString: too large ciphertext written: cipherlen (%d) > max_ciphertext_size (%zd)", cipherlen, max_ciphertext_size); @@ -711,7 +725,11 @@ int DecryptString(char *out, size_t out_size, const char *in, int cipherlen, plainlen += tmplen; - if(plainlen > max_plaintext_size) + if (plainlen < 0) + { + ProgrammingError("DecryptString: plainlen (%d) < 0", plainlen); + } + if ((size_t) plainlen > max_plaintext_size) { ProgrammingError("DecryptString: too large plaintext written: plainlen (%d) > max_plaintext_size (%zd)", plainlen, max_plaintext_size); @@ -729,7 +747,11 @@ void DebugBinOut(char *buffer, int len, char *comment) char buf[CF_BUFSIZE]; char hexStr[3]; // one byte as hex - if (len >= (sizeof(buf) / 2)) // hex uses two chars per byte + if (len < 0) + { + Log(LOG_LEVEL_ERR, "Debug binary print negative len param (len = %d)", len); + } + else if ((unsigned long) len >= (sizeof(buf) / 2)) // hex uses two chars per byte { Log(LOG_LEVEL_DEBUG, "Debug binary print is too large (len = %d)", len); return; diff --git a/libpromises/evalfunction.c b/libpromises/evalfunction.c index 57546eb014..21010bf11e 100644 --- a/libpromises/evalfunction.c +++ b/libpromises/evalfunction.c @@ -7768,7 +7768,7 @@ static char *StripPatterns(char *file_buffer, const char *pattern, const char *f return file_buffer; } - int start, end, count = 0; + size_t start, end, count = 0; const size_t original_length = strlen(file_buffer); while (StringMatchWithPrecompiledRegex(rx, file_buffer, &start, &end)) { diff --git a/libpromises/ornaments.c b/libpromises/ornaments.c index a3fd9a70e4..e6bf9ad850 100644 --- a/libpromises/ornaments.c +++ b/libpromises/ornaments.c @@ -39,7 +39,13 @@ */ static bool StringAppendPromise(char *dst, const char *src, size_t n) { - int i, j; + size_t i, j; + + if (n == 0) + { + return false; + } + n--; for (i = 0; i < n && dst[i]; i++) { diff --git a/libpromises/pipes.c b/libpromises/pipes.c index 4f7265a274..2e0255d974 100644 --- a/libpromises/pipes.c +++ b/libpromises/pipes.c @@ -176,7 +176,14 @@ int PipeWriteData(const char *base_cmd, const char *args, const char *data) io.read_fd, io.write_fd, args); int res = 0; - if (PipeWrite(&io, data) != strlen(data)) + int written = PipeWrite(&io, data); + if (written < 0) + { + Log(LOG_LEVEL_ERR, "Failed to write to pipe (fd %d): %s", + io.write_fd, GetErrorStr()); + res = -1; + } + else if ((size_t) written != strlen(data)) { Log(LOG_LEVEL_VERBOSE, "Was not able to send whole data to application '%s'.", @@ -217,7 +224,13 @@ int PipeReadWriteData(const char *base_cmd, const char *args, const char *reques Log(LOG_LEVEL_DEBUG, "Opened fds %d and %d for command '%s'.", io.read_fd, io.write_fd, command); - if (PipeWrite(&io, request) != strlen(request)) + int written = PipeWrite(&io, request); + if (written < 0) { + Log(LOG_LEVEL_ERR, "Failed to write to pipe (fd %d): %s", + io.write_fd, GetErrorStr()); + return -1; + } + else if ((size_t) written != strlen(request)) { Log(LOG_LEVEL_VERBOSE, "Couldn't send whole data to application '%s'.", base_cmd); diff --git a/libpromises/processes_select.c b/libpromises/processes_select.c index b5b59aa669..1dbaab05a0 100644 --- a/libpromises/processes_select.c +++ b/libpromises/processes_select.c @@ -594,7 +594,7 @@ static bool SelectProcRegexMatch(const char *name1, const char *name2, } else { - int s, e; + size_t s, e; return StringMatch(regex, line[i], &s, &e); } } diff --git a/libpromises/rlist.c b/libpromises/rlist.c index 66891cc054..1cd66bfefd 100644 --- a/libpromises/rlist.c +++ b/libpromises/rlist.c @@ -1146,8 +1146,8 @@ Rlist *RlistFromSplitRegex(const char *string, const char *regex, size_t max_ent const char *sp = string; size_t entry_count = 0; - int start = 0; - int end = 0; + size_t start = 0; + size_t end = 0; Rlist *result = NULL; Buffer *buffer = BufferNewWithCapacity(CF_MAXVARSIZE); @@ -1206,7 +1206,7 @@ Rlist *RlistFromRegexSplitNoOverflow(const char *string, const char *regex, int { Rlist *liststart = NULL; char node[CF_MAXVARSIZE]; - int start, end; + size_t start, end; int count = 0; assert(max > 0); // ensured by FnCallStringSplit() before calling us @@ -1230,7 +1230,7 @@ Rlist *RlistFromRegexSplitNoOverflow(const char *string, const char *regex, int { len = CF_MAXVARSIZE - 1; Log(LOG_LEVEL_WARNING, - "Segment in string_split() is %d bytes and will be truncated to %zu bytes", + "Segment in string_split() is %zu bytes and will be truncated to %zu bytes", start, len); } diff --git a/libpromises/syntax.c b/libpromises/syntax.c index d8d4735fda..8edaa9a4b0 100644 --- a/libpromises/syntax.c +++ b/libpromises/syntax.c @@ -1065,7 +1065,7 @@ static JsonElement *BundleTypesToJson(void) while ((bundle_type = JsonIteratorNextKey(&it))) { JsonElement *promise_types = JsonObjectGetAsArray(JsonObjectGetAsObject(bundle_types, bundle_type), "promiseTypes"); - for (int i = 0; i < SeqLength(common_promise_types); i++) + for (size_t i = 0; i < SeqLength(common_promise_types); i++) { const char *common_promise_type = SeqAt(common_promise_types, i); JsonArrayAppendString(promise_types, common_promise_type); diff --git a/libpromises/var_expressions.c b/libpromises/var_expressions.c index 6b17ac8570..108090db57 100644 --- a/libpromises/var_expressions.c +++ b/libpromises/var_expressions.c @@ -416,7 +416,7 @@ void VarRefDestroy(VarRef *ref) free(ref->lval); if (ref->num_indices > 0) { - for (int i = 0; i < ref->num_indices; ++i) + for (size_t i = 0; i < ref->num_indices; ++i) { free(ref->indices[i]); } diff --git a/libpromises/vars.c b/libpromises/vars.c index d927a168a4..62faf39c02 100644 --- a/libpromises/vars.c +++ b/libpromises/vars.c @@ -384,7 +384,7 @@ bool ExtractScalarReference(Buffer *out, const char *str, size_t len, bool extra } const char *dollar_point = memchr(str, '$', len); - if (!dollar_point || (dollar_point - str) == len) + if (!dollar_point || (size_t) (dollar_point - str) == len) { return false; } diff --git a/libpromises/verify_classes.c b/libpromises/verify_classes.c index 1dfcc98f53..2657f337a1 100644 --- a/libpromises/verify_classes.c +++ b/libpromises/verify_classes.c @@ -48,7 +48,8 @@ static bool ValidClassName(const char *str) FreeExpression(res.result); } - return res.result && res.position == strlen(str); + assert(res.position >= 0); + return res.result && (size_t) res.position == strlen(str); } PromiseResult VerifyClassPromise(EvalContext *ctx, const Promise *pp, ARG_UNUSED void *param) From a5821fb93dddbd0928a3d7651db0c647fd509ef9 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Fri, 8 Jan 2021 14:41:58 +0100 Subject: [PATCH 242/333] Fixed memory leak in logging_test Changelog: None Ticket: CFE-3538 Signed-off-by: Ole Herman Schumacher Elgesem (cherry picked from commit f5911649440dbb7bd3219e7427efb6bbb4c74824) --- tests/unit/logging_test.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tests/unit/logging_test.c b/tests/unit/logging_test.c index cc342de44c..dd2f087ae5 100644 --- a/tests/unit/logging_test.c +++ b/tests/unit/logging_test.c @@ -18,6 +18,7 @@ ssize_t sendto(ARG_UNUSED int sockfd, ARG_UNUSED const void *buf, const struct sockaddr *dest_addr, ARG_UNUSED socklen_t addrlen) { + free(got_address); // RemoteSysLog can call this multiple times got_address = xmemdup(dest_addr, sizeof(struct sockaddr_in)); return len; } @@ -32,6 +33,7 @@ int sendto(ARG_UNUSED int sockfd, ARG_UNUSED const void *buf, const void *dest_addr, ARG_UNUSED int addrlen) { + free(got_address); // RemoteSysLog can call this multiple times got_address = xmemdup(dest_addr, sizeof(struct sockaddr_in)); return len; } @@ -52,6 +54,7 @@ static void test_set_port(void) } free(got_address); + got_address = NULL; // Safe to free(NULL) in another test } static void test_set_host(void) @@ -62,6 +65,8 @@ static void test_set_host(void) assert_int_equal(got_address->sa_family, AF_INET); assert_int_equal(ntohl(((struct sockaddr_in *) got_address)->sin_addr.s_addr), 0x7f000037); + free(got_address); + got_address = NULL; // Safe to free(NULL) in another test } #define check_level(str, lvl) \ From 964a08e3734fa12be3ca6c81f24abb0b7f520294 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Fri, 11 Sep 2020 16:16:54 -0500 Subject: [PATCH 243/333] Removed "Enterprise" from cf-execd systemd unit description (cherry picked from commit 29586f371328789c1529e919f0cebc65cbd8586e) --- misc/systemd/cf-execd.service.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/misc/systemd/cf-execd.service.in b/misc/systemd/cf-execd.service.in index 83e5fe0e12..95567bcfe0 100644 --- a/misc/systemd/cf-execd.service.in +++ b/misc/systemd/cf-execd.service.in @@ -1,5 +1,5 @@ [Unit] -Description=CFEngine Enterprise Execution Scheduler +Description=CFEngine Execution Scheduler After=syslog.target ConditionPathExists=@bindir@/cf-execd ConditionPathExists=@workdir@/inputs/promises.cf From e0a28f63fc9fff69e704c3a6da2e09d475e266de Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Thu, 1 Jul 2021 15:17:16 -0500 Subject: [PATCH 244/333] Revert "Added cf-postgres requirement to cf-apache and cf-hub systemd units" This reverts commit ce2862bfc3f8c864c5335d25dd9a6ff15cfe120a. Due to HA managing cf-postgres outside of systemd we must use Wants instead of Requires. Ticket: ENT-5125 Changelog: title (cherry picked from commit 2cf1a86fb0eb9a504aa265a61f9b52313f3d21b5) --- misc/systemd/cf-apache.service.in | 1 - misc/systemd/cf-hub.service.in | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/misc/systemd/cf-apache.service.in b/misc/systemd/cf-apache.service.in index ac41ac49f8..4e0e7b53d0 100644 --- a/misc/systemd/cf-apache.service.in +++ b/misc/systemd/cf-apache.service.in @@ -2,7 +2,6 @@ Description=CFEngine Enterprise Webserver After=syslog.target After=cf-postgres.service -Requires=cf-postgres.service ConditionPathExists=@workdir@/httpd/bin/apachectl PartOf=cfengine3.service diff --git a/misc/systemd/cf-hub.service.in b/misc/systemd/cf-hub.service.in index 1d6a879035..1c8c62aa46 100644 --- a/misc/systemd/cf-hub.service.in +++ b/misc/systemd/cf-hub.service.in @@ -6,8 +6,8 @@ ConditionPathExists=@workdir@/inputs/promises.cf After=syslog.target After=network.target +Wants=cf-postgres.service After=cf-postgres.service -Requires=cf-postgres.service [Service] Type=simple From cd7c3496c0412659f2664383655e5e36d623aeb2 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Thu, 1 Jul 2021 15:17:54 -0500 Subject: [PATCH 245/333] Add Wants cf-postgres.service to cf-apache.service in systemd configuration Ticket: ENT-5125 Changelog: title (cherry picked from commit 23358cac14388b1d418608bf07ac9f100f71d10b) --- misc/systemd/cf-apache.service.in | 1 + 1 file changed, 1 insertion(+) diff --git a/misc/systemd/cf-apache.service.in b/misc/systemd/cf-apache.service.in index 4e0e7b53d0..0ca6e5405f 100644 --- a/misc/systemd/cf-apache.service.in +++ b/misc/systemd/cf-apache.service.in @@ -1,6 +1,7 @@ [Unit] Description=CFEngine Enterprise Webserver After=syslog.target +Wants=cf-postgres.service After=cf-postgres.service ConditionPathExists=@workdir@/httpd/bin/apachectl PartOf=cfengine3.service From 78249db657a2023e395ce5f15e47482753e4ffa7 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Fri, 23 Jul 2021 17:56:55 +0200 Subject: [PATCH 246/333] Updated libntech Travis is failing, so interested to see if this helps. Signed-off-by: Ole Herman Schumacher Elgesem --- libntech | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libntech b/libntech index 10bd6288fd..04917ec6e2 160000 --- a/libntech +++ b/libntech @@ -1 +1 @@ -Subproject commit 10bd6288fdef879f24d69032910cd9f002b624f2 +Subproject commit 04917ec6e20b86457da6d6e10304f4948c8c9d09 From 267d4fcc933b0147e219ed88719a39c0914068fa Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Mon, 26 Jul 2021 09:21:58 -0500 Subject: [PATCH 247/333] Added example illustrating rxdirs perms body attribute Ticket: CFE-3718 Changelog: None (cherry picked from commit 42227b172c13c3c561ae0ed1dcf63e42185e0cec) --- examples/rxdirs.cf | 59 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 examples/rxdirs.cf diff --git a/examples/rxdirs.cf b/examples/rxdirs.cf new file mode 100644 index 0000000000..8c2abbaf59 --- /dev/null +++ b/examples/rxdirs.cf @@ -0,0 +1,59 @@ +############################################################################### +#+begin_src cfengine3 +bundle agent __main__ +# @brief Illustrating the behavior of rxdirs in perms bodies. Note, by default, when read is requested for a directory, execute is given as well. +{ + vars: + "example_dirs" slist => { + "/tmp/rxdirs_example/rxdirs=default(true)-r", + "/tmp/rxdirs_example/rxdirs=default(true)-rx", + "/tmp/rxdirs_example/rxdirs=false-r", + "/tmp/rxdirs_example/rxdirs=false-rx", + }; + + files: + "$(example_dirs)/." + create => "true"; + + "/tmp/rxdirs_example/rxdirs=default(true)-r" + perms => example:m( 600 ); + + "/tmp/rxdirs_example/rxdirs=default(true)-rx" + perms => example:m( 700 ); + + "/tmp/rxdirs_example/rxdirs=false-r" + perms => example:m_rxdirs_off( 600 ); + + "/tmp/rxdirs_example/rxdirs=false-rx" + perms => example:m_rxdirs_off( 700 ); + + reports: + "$(example_dirs) modeoct='$(with)'" + with => filestat( $(example_dirs), modeoct ); +} + +body file control{ namespace => "example"; } + +body perms m(mode) +# @brief Set the file mode +# @param mode The new mode +{ + mode => "$(mode)"; +} +body perms m_rxdirs_off(mode) +# @brief Set the file mode, don't be clever and try to set +x on directory when +r is desired +# @param mode The new mode +{ + inherit_from => m( $(mode) ); + rxdirs => "false"; +} +#+end_src +############################################################################### +#+begin_src example_output +#@ ``` +#@ R: /tmp/rxdirs_example/rxdirs=default(true)-r modeoct='40700' +#@ R: /tmp/rxdirs_example/rxdirs=default(true)-rx modeoct='40700' +#@ R: /tmp/rxdirs_example/rxdirs=false-r modeoct='40600' +#@ R: /tmp/rxdirs_example/rxdirs=false-rx modeoct='40700' +#@ ``` +#+end_example From bcf6ca27e76d168ec8416f013c8924136d3f8559 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Tue, 20 Jul 2021 15:36:23 -0500 Subject: [PATCH 248/333] Use ts_key file for measurement names in cf-check dump Without using ts_key we can't dump custom measurements. Ticket: ENT-7452 Changelog: title (cherry picked from commit 418aa12f13e3fa5f4edd07c5fea55806e39b3f14) --- cf-check/Makefile.am | 3 +- cf-check/dump.c | 81 +++++++++++++++++++------- cf-check/observables.c | 105 ++++++++++++++++++++++++++++++++++ cf-check/observables.h | 11 ++++ libpromises/monitoring_read.c | 4 ++ 5 files changed, 182 insertions(+), 22 deletions(-) create mode 100644 cf-check/observables.c create mode 100644 cf-check/observables.h diff --git a/cf-check/Makefile.am b/cf-check/Makefile.am index a7ebe4a2c0..deb1a9a515 100644 --- a/cf-check/Makefile.am +++ b/cf-check/Makefile.am @@ -61,7 +61,8 @@ libcf_check_la_SOURCES = \ utilities.c utilities.h \ repair.c repair.h \ replicate_lmdb.c replicate_lmdb.h \ - validate.c validate.h + validate.c validate.h \ + observables.c observables.h if !BUILTIN_EXTENSIONS bin_PROGRAMS = cf-check diff --git a/cf-check/dump.c b/cf-check/dump.c index bbea0e289d..a9ea155ba7 100644 --- a/cf-check/dump.c +++ b/cf-check/dump.c @@ -7,6 +7,9 @@ #include #include #include +#include // GetStateDir() for usage printout +#include // FILE_SEPARATOR +#include typedef enum { @@ -19,7 +22,18 @@ typedef enum static void print_usage(void) { - printf("Usage: cf-check dump [-k|-v|-n|-s|-p] [FILE ...]\n"); + printf("Usage: cf-check dump [-k|-v|-n|-s|-p|-t FILE] [FILE ...]\n"); + printf("\n"); + printf("\t-k|--keys print only keys\n"); + printf("\t-v|--values print only values\n"); + printf("\t-n|--nice print strings in a nice way and with database specific awareness\n"); + printf("\t-s|--simple print as simple escaped binary data\n"); + printf("\t-p|--portable print unambiguously with structs and raw strings\n"); + printf("\t-t|--tskey FILE use FILE as list of observables\n"); + printf("\tWill use '%s%cts_key' if not specified, or built-in list if no ts_key file is found.\n", + GetStateDir(), + FILE_SEPARATOR); + printf("\n"); printf("Example: cf-check dump /var/cfengine/state/cf_lastseen.lmdb\n"); } @@ -54,7 +68,7 @@ static void print_json_string( bool known_data = ((size == 1) || (len == (size - 1)) || (data[size - 1] == '\n')); if (!known_data) { - printf("\nError: This database contains unknown binary data - use --simple to print anyway\n"); + printf("\nError: This database contains unknown binary data - use --simple to print anyway or try on the same OS/architecture the lmdb file was generated on\n"); exit(1); } @@ -143,7 +157,7 @@ static void print_struct_lock_data( // Used to print values in /var/cfengine/state/cf_observations.lmdb: static void print_struct_averages( - const MDB_val value, const bool strip_strings) + const MDB_val value, const bool strip_strings, const char *tskey_filename) { assert(sizeof(Averages) == value.mv_size); if (sizeof(Averages) != value.mv_size) @@ -154,25 +168,30 @@ static void print_struct_averages( else { // TODO: clean up Averages + char **obnames = NULL; const Averages *const averages = value.mv_data; const time_t last_seen = averages->last_seen; - JsonElement *all_observables = JsonObjectCreate(observables_max); - assert(observables_max <= CF_OBSERVABLES); + obnames = GetObservableNames(tskey_filename); + JsonElement *all_observables = JsonObjectCreate(CF_OBSERVABLES); - for (Observable i = 0; i < observables_max; ++i) + for (Observable i = 0; i < CF_OBSERVABLES; ++i) { + char *name = obnames[i]; JsonElement *observable = JsonObjectCreate(4); QPoint Q = averages->Q[i]; - const char *const name = observable_strings[i]; JsonObjectAppendReal(observable, "q", Q.q); JsonObjectAppendReal(observable, "expect", Q.expect); JsonObjectAppendReal(observable, "var", Q.var); JsonObjectAppendReal(observable, "dq", Q.dq); JsonObjectAppendObject(all_observables, name, observable); + + free(obnames[i]); } + free(obnames); + JsonElement *top_json = JsonObjectCreate(2); JsonObjectAppendInteger(top_json, "last_seen", last_seen); JsonObjectAppendObject(top_json, "Q", all_observables); @@ -254,7 +273,8 @@ static void print_struct_or_string( const MDB_val value, const char *const file, const bool strip_strings, - const bool structs) + const bool structs, + const char *tskey_filename) { if (structs) { @@ -277,12 +297,12 @@ static void print_struct_or_string( } else { - print_struct_averages(value, strip_strings); + print_struct_averages(value, strip_strings, tskey_filename); } } else if (StringEndsWith(file, "history.lmdb")) { - print_struct_averages(value, strip_strings); + print_struct_averages(value, strip_strings, tskey_filename); } else if (StringEndsWith(file, "cf_state.lmdb")) { @@ -323,12 +343,13 @@ static void print_line_key_value( const MDB_val value, const char *const file, const bool strip_strings, - const bool structs) + const bool structs, + const char *tskey_filename) { printf("\t"); print_json_string(key.mv_data, key.mv_size, strip_strings); printf(": "); - print_struct_or_string(key, value, file, strip_strings, structs); + print_struct_or_string(key, value, file, strip_strings, structs, tskey_filename); printf(",\n"); } @@ -364,7 +385,7 @@ static void print_closing_bracket(const dump_mode mode) } } -static int dump_db(const char *file, const dump_mode mode) +static int dump_db(const char *file, const dump_mode mode, const char *tskey_filename) { assert(file != NULL); @@ -372,10 +393,10 @@ static int dump_db(const char *file, const dump_mode mode) const bool structs = (mode == DUMP_NICE || mode == DUMP_PORTABLE); int r; - MDB_env *env; - MDB_txn *txn; + MDB_env *env = NULL; + MDB_txn *txn = NULL; MDB_dbi dbi; - MDB_cursor *cursor; + MDB_cursor *cursor = NULL; if (0 != (r = mdb_env_create(&env)) || 0 != (r = mdb_env_open(env, file, MDB_NOSUBDIR | MDB_RDONLY, 0644)) @@ -383,6 +404,18 @@ static int dump_db(const char *file, const dump_mode mode) || 0 != (r = mdb_open(txn, NULL, 0, &dbi)) || 0 != (r = mdb_cursor_open(txn, dbi, &cursor))) { + if (env != NULL) + { + if (txn != NULL) + { + if (cursor != NULL) + { + mdb_cursor_close(cursor); + } + mdb_txn_abort(txn); + } + mdb_env_close(env); + } return dump_report_error(r); } @@ -395,7 +428,7 @@ static int dump_db(const char *file, const dump_mode mode) case DUMP_NICE: case DUMP_PORTABLE: case DUMP_SIMPLE: - print_line_key_value(key, value, file, strip_strings, structs); + print_line_key_value(key, value, file, strip_strings, structs, tskey_filename); break; case DUMP_KEYS: print_line_array_element(key, strip_strings); @@ -424,7 +457,7 @@ static int dump_db(const char *file, const dump_mode mode) return 0; } -static int dump_dbs(Seq *const files, const dump_mode mode) +static int dump_dbs(Seq *const files, const dump_mode mode, const char *tskey_filename) { assert(files != NULL); const size_t length = SeqLength(files); @@ -432,7 +465,7 @@ static int dump_dbs(Seq *const files, const dump_mode mode) if (length == 1) { - return dump_db(SeqAt(files, 0), mode); + return dump_db(SeqAt(files, 0), mode, tskey_filename); } int ret = 0; @@ -441,7 +474,7 @@ static int dump_dbs(Seq *const files, const dump_mode mode) { const char *const filename = SeqAt(files, i); printf("%s:\n", filename); - const int r = dump_db(filename, mode); + const int r = dump_db(filename, mode, tskey_filename); if (r != 0) { ret = r; @@ -457,6 +490,7 @@ int dump_main(int argc, const char *const *const argv) dump_mode mode = DUMP_NICE; size_t offset = 1; + const char *tskey_filename = NULL; if ((size_t) argc > offset && argv[offset] != NULL && argv[offset][0] == '-') { @@ -483,6 +517,11 @@ int dump_main(int argc, const char *const *const argv) { mode = DUMP_PORTABLE; } + else if (StringMatchesOption(option, "--tskey", "-t")) + { + tskey_filename = argv[offset]; + offset += 1; + } else { print_usage(); @@ -499,7 +538,7 @@ int dump_main(int argc, const char *const *const argv) } Seq *files = argv_to_lmdb_files(argc, argv, offset); - const int ret = dump_dbs(files, mode); + const int ret = dump_dbs(files, mode, tskey_filename); SeqDestroy(files); return ret; } diff --git a/cf-check/observables.c b/cf-check/observables.c new file mode 100644 index 0000000000..857c6a192d --- /dev/null +++ b/cf-check/observables.c @@ -0,0 +1,105 @@ +#include +#include +#include // CF_BUFSIZE +#include // GetStateDir() +#include // FILE_SEPARATOR +#include // xstrdup() +#include // observable_strings +#include // StringEqual +#include + +/** + * GetObservableNames guarantees to return an array of CF_OBSERVABLES size with + * non-NULL strings if using built-in, will generate "observable[n]" name, if + * reading ts_key file and the item is "spare" then generating "spare[n]" where + * 'n' is the id number which could be helpful in debugging raw observables + * data. + * + * This function is similar to Nova_LoadSlots() in + * libpromises/monitoring_read.c If we refactor for dynamic observables instead + * of hard coded and limited to 100 then we likely should change here and there + * or refactor to have this ts_key read/parse code in one shared place. + */ +char **GetObservableNames(const char *ts_key_path) +{ + char buf[CF_BUFSIZE]; + const char *filename; + char **temp = xmalloc(CF_OBSERVABLES * sizeof(char *)); + + if (ts_key_path == NULL) + { + snprintf( + buf, CF_BUFSIZE - 1, "%s%cts_key", GetStateDir(), FILE_SEPARATOR); + filename = buf; + } + else + { + filename = ts_key_path; + } + + FILE *f = safe_fopen(filename, "r"); + if (f == NULL) + { + for (int i = 0; i < CF_OBSERVABLES; ++i) + { + if (i < observables_max) + { + temp[i] = xstrdup(observable_strings[i]); + } + else + { + snprintf(buf, CF_MAXVARSIZE, "observable[%d]", i); + temp[i] = xstrdup(buf); + } + } + } + else + { + for (int i = 0; i < CF_OBSERVABLES; ++i) + { + char line[CF_MAXVARSIZE]; + + char name[CF_MAXVARSIZE], desc[CF_MAXVARSIZE]; + char units[CF_MAXVARSIZE] = "unknown"; + double expected_min = 0.0; + double expected_max = 100.0; + int consolidable = true; + + if (fgets(line, CF_MAXVARSIZE, f) == NULL) + { + Log(LOG_LEVEL_ERR, + "Error trying to read ts_key from file '%s'. (fgets: %s)", + filename, + GetErrorStr()); + break; + } + + const int fields = sscanf( + line, + "%*d,%1023[^,],%1023[^,],%1023[^,],%lf,%lf,%d", + name, + desc, + units, + &expected_min, + &expected_max, + &consolidable); + + if ((fields != 2) && (fields != 6)) + { + Log(LOG_LEVEL_ERR, "Wrong line format in ts_key: %s", line); + } + + if (StringEqual(name, "spare")) + { + temp[i] = xstrdup(name); + } + else + { + snprintf(buf, CF_MAXVARSIZE, "spare[%d]", i); + temp[i] = xstrdup(buf); + } + } + fclose(f); + } + return temp; +} diff --git a/cf-check/observables.h b/cf-check/observables.h new file mode 100644 index 0000000000..1c8d932b96 --- /dev/null +++ b/cf-check/observables.h @@ -0,0 +1,11 @@ +#ifndef CF_CHECK_TS_KEY_READ_H +#define CF_CHECK_TS_KEY_READ_H + +#include + +// copy of libpromises/cf3.defs.h, TODO refactor +#define CF_OBSERVABLES 100 + +char **GetObservableNames(const char *ts_key_path); + +#endif diff --git a/libpromises/monitoring_read.c b/libpromises/monitoring_read.c index ddec36e206..8ecba8a48d 100644 --- a/libpromises/monitoring_read.c +++ b/libpromises/monitoring_read.c @@ -133,6 +133,10 @@ MonitoringSlot *Nova_MakeSlot(const char *name, const char *description, return slot; } +// This function is similar to cf-check/observables.c function GetObservableNames +// If we refactor for dynamic observables instead of hard coded and limited to 100 +// then we likely should change here and there or refactor to have +// this ts_key read/parse code in one shared place. void Nova_LoadSlots(void) { char filename[CF_BUFSIZE]; From fff7826ee00e03f4adfa1ba211d5a17a1178cc90 Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Wed, 28 Jul 2021 14:02:26 +0200 Subject: [PATCH 249/333] Fixed some memory leaks in cf-key Ticket: CFE-3719 Changelog: None Signed-off-by: Lars Erik Wik (cherry picked from commit b76d3599097ad862eb673ae50258c8ed2e86dd23) --- cf-key/cf-key.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/cf-key/cf-key.c b/cf-key/cf-key.c index 07bc22eeef..b5808f26c2 100644 --- a/cf-key/cf-key.c +++ b/cf-key/cf-key.c @@ -155,12 +155,14 @@ int main(int argc, char *argv[]) { SetupSignalsForCfKey(handleShowKeysSignal); ShowLastSeenHosts(!NO_TRUNCATE); + GenericAgentFinalize(ctx, config); CallCleanupFunctions(); return 0; } if (print_digest_arg) { + GenericAgentFinalize(ctx, config); CallCleanupFunctions(); return PrintDigest(print_digest_arg); } @@ -190,10 +192,10 @@ int main(int argc, char *argv[]) Log (LOG_LEVEL_VERBOSE, "Forced removal of entry '%s' was successful", remove_keys_host); - CallCleanupFunctions(); - return 0; + status = 0; } } + GenericAgentFinalize(ctx, config); CallCleanupFunctions(); return status; } @@ -201,6 +203,7 @@ int main(int argc, char *argv[]) if(LICENSE_INSTALL) { bool success = LicenseInstall(LICENSE_SOURCE); + GenericAgentFinalize(ctx, config); CallCleanupFunctions(); return success ? 0 : 1; } @@ -223,6 +226,7 @@ int main(int argc, char *argv[]) bool ret = TrustKey(filename, ipaddr, username); free(arg); + GenericAgentFinalize(ctx, config); CallCleanupFunctions(); return ret ? EXIT_SUCCESS : EXIT_FAILURE; } @@ -292,6 +296,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) GenericAgentWriteVersion(w); FileWriterDetach(w); } + GenericAgentConfigDestroy(config); DoCleanupAndExit(EXIT_SUCCESS); case 'v': @@ -338,6 +343,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) case 'h': PrintHelp(); + GenericAgentConfigDestroy(config); DoCleanupAndExit(EXIT_SUCCESS); case 'M': @@ -350,12 +356,14 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) NULL, false, false); FileWriterDetach(out); + GenericAgentConfigDestroy(config); DoCleanupAndExit(EXIT_SUCCESS); } case 'C': if (!GenericAgentConfigParseColor(config, optarg)) { + GenericAgentConfigDestroy(config); DoCleanupAndExit(EXIT_FAILURE); } break; @@ -370,6 +378,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) default: PrintHelp(); + GenericAgentConfigDestroy(config); DoCleanupAndExit(EXIT_FAILURE); } @@ -379,6 +388,7 @@ static GenericAgentConfig *CheckOpts(int argc, char **argv) { PrintHelp(); Log(LOG_LEVEL_ERR, "--no-truncate / -N option is only for --show-hosts / -s"); + GenericAgentConfigDestroy(config); DoCleanupAndExit(EXIT_FAILURE); } From 550c1522f636ac09c7fa71fab2819d957717a63a Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Wed, 28 Jul 2021 14:06:35 +0200 Subject: [PATCH 250/333] Changed from hard coded return codes to macros Ticket: None Changelog: None Signed-off-by: Lars Erik Wik (cherry picked from commit 29b1fee62029c6c901cf332d0d9ca7cbe6220286) --- cf-key/cf-key.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cf-key/cf-key.c b/cf-key/cf-key.c index b5808f26c2..6aa0d1a5f0 100644 --- a/cf-key/cf-key.c +++ b/cf-key/cf-key.c @@ -157,7 +157,7 @@ int main(int argc, char *argv[]) ShowLastSeenHosts(!NO_TRUNCATE); GenericAgentFinalize(ctx, config); CallCleanupFunctions(); - return 0; + return EXIT_SUCCESS; } if (print_digest_arg) @@ -192,7 +192,7 @@ int main(int argc, char *argv[]) Log (LOG_LEVEL_VERBOSE, "Forced removal of entry '%s' was successful", remove_keys_host); - status = 0; + status = EXIT_SUCCESS; } } GenericAgentFinalize(ctx, config); @@ -205,7 +205,7 @@ int main(int argc, char *argv[]) bool success = LicenseInstall(LICENSE_SOURCE); GenericAgentFinalize(ctx, config); CallCleanupFunctions(); - return success ? 0 : 1; + return success ? EXIT_SUCCESS : EXIT_FAILURE; } if (trust_key_arg != NULL) From 08940d06df4cf9991100a13d49a38da0da925bbd Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Tue, 10 Aug 2021 15:35:57 +0200 Subject: [PATCH 251/333] Update libntech to latest for 3.15.x Ticket: None Changelog: None Signed-off-by: Lars Erik Wik --- libntech | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libntech b/libntech index 04917ec6e2..1ddc193972 160000 --- a/libntech +++ b/libntech @@ -1 +1 @@ -Subproject commit 04917ec6e20b86457da6d6e10304f4948c8c9d09 +Subproject commit 1ddc193972340d5c0dd2e49f613ded00aece9497 From f6a83b708ca295cc31e251fdfcc81ecf0d2b7742 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Tue, 10 Aug 2021 09:16:28 -0500 Subject: [PATCH 252/333] Skip update policy run with --skip-bootstrap-policy-run Workaround for lmdb invalid argument issues in CI (ENT-7500) Ticket: ENT-7511 Changelog: title (cherry picked from commit ee529a5ca34e537ed00044b3bc7909b7bf9763d4) --- libpromises/failsafe.cf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libpromises/failsafe.cf b/libpromises/failsafe.cf index 134a5ca1ea..ea60da2c24 100644 --- a/libpromises/failsafe.cf +++ b/libpromises/failsafe.cf @@ -383,7 +383,7 @@ bundle agent failsafe_cfe_internal_call_update # On Windows we need cf-execd to call update.cf, otherwise the daemons will # not run under the SYSTEM account. - !windows:: + !windows.!skip_policy_on_bootstrap:: "$(sys.cf_agent) -f $(sys.update_policy_path) --define $(mode)" handle => "failsafe_cfe_internal_call_update_commands_call_update_cf", if => fileexists( $(sys.update_policy_path) ), From b545b805c7632ffabc74700ee206120dda3391c2 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Tue, 17 Aug 2021 13:38:56 -0500 Subject: [PATCH 253/333] Added example illustrating that the last key wins with mergedata() Ticket: CFE-3733 Changelog: None (cherry picked from commit 62a2a95c246fd5123d56dd5999a8dc4ad25d88d6) --- examples/mergedata-last-key-wins.cf | 30 +++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 examples/mergedata-last-key-wins.cf diff --git a/examples/mergedata-last-key-wins.cf b/examples/mergedata-last-key-wins.cf new file mode 100644 index 0000000000..fc75d4702c --- /dev/null +++ b/examples/mergedata-last-key-wins.cf @@ -0,0 +1,30 @@ +#+begin_src cfengine3 +bundle agent mergedata_last_key_wins +# @brief Example illustrating how the last key wins when merging data containers with conflicting keys +{ + vars: + + "one" data => '{ "color": "red", "stuff": [ "one", "two" ], "thing": "one" }'; + "two" data => '{ "color": "blue", "stuff": [ "three" ] }'; + + reports: + "$(with)" with => storejson( mergedata( one, two ) ); + +} +bundle agent __main__ +{ + methods: "mergedata_last_key_wins"; +} +#+end_src +############################################################################### +#+begin_src example_output +#@ ``` +#@ R: { +#@ "color": "blue", +#@ "stuff": [ +#@ "three" +#@ ], +#@ "thing": "one" +#@ } +#@ ``` +#+end_src From c2c04f3f7f909e181621070b226acc1eb2483b71 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Fri, 20 Aug 2021 11:58:44 -0500 Subject: [PATCH 254/333] Fix dbm_tokyocab.c DBPrivRead() argument type Ticket: CFE-3737 Changelog: title (cherry picked from commit 36ac4af02c36f778ea69b9e10471d899f5b4b15f) --- libpromises/dbm_tokyocab.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libpromises/dbm_tokyocab.c b/libpromises/dbm_tokyocab.c index 81be2442a7..c6280e1550 100644 --- a/libpromises/dbm_tokyocab.c +++ b/libpromises/dbm_tokyocab.c @@ -257,7 +257,7 @@ int DBPrivGetValueSize(DBPriv *db, const void *key, int key_size) return tchdbvsiz(db->hdb, key, key_size); } -bool DBPrivRead(DBPriv *db, const void *key, int key_size, void *dest, int dest_size) +bool DBPrivRead(DBPriv *db, const void *key, int key_size, void *dest, size_t dest_size) { if (tchdbget3(db->hdb, key, key_size, dest, dest_size) == -1) { From ca636110ce7f16dbf869c184b70b08179d863f85 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Fri, 20 Aug 2021 12:14:14 -0500 Subject: [PATCH 255/333] Fix dbm_quick.c DBPrivRead() argument type Ticket: CFE-3737 Changelog: title (cherry picked from commit c431b10b975b1ab3c2083f9b365d9244dd8de204) --- configure.ac | 2 +- libpromises/dbm_quick.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/configure.ac b/configure.ac index 35ffbcda5f..00d4452fb2 100644 --- a/configure.ac +++ b/configure.ac @@ -390,7 +390,7 @@ AS_IF([test -n "$with_qdbm" && test "x$with_qdbm" != "xno"], [WITH_QDBM=1], [WIT if test $WITH_QDBM = 1; then CF3_WITH_LIBRARY(qdbm, [ AC_CHECK_LIB(qdbm, dpopen, [], [AC_MSG_ERROR(Cannot find QDBM)]) - AC_CHECK_HEADERS(depot.h, [], [AC_MSG_ERROR(Cannot find QDBM)]) + AC_CHECK_HEADERS(qdbm/depot.h, [], [AC_MSG_ERROR(Cannot find QDBM)]) AC_DEFINE(QDB, 1, [Define if QDBM is available]) ]) fi diff --git a/libpromises/dbm_quick.c b/libpromises/dbm_quick.c index 57823dc729..be8ade490c 100644 --- a/libpromises/dbm_quick.c +++ b/libpromises/dbm_quick.c @@ -33,7 +33,7 @@ #include #ifdef QDB -# include +# include struct DBPriv_ { @@ -206,7 +206,7 @@ bool DBPrivClean(DBPriv *db) return true; } -bool DBPrivRead(DBPriv *db, const void *key, int key_size, void *dest, int dest_size) +bool DBPrivRead(DBPriv *db, const void *key, int key_size, void *dest, size_t dest_size) { if (!Lock(db)) { From f53395f63a921cc6133ae187582d1924ffcb44e1 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Fri, 2 Jul 2021 15:10:00 -0500 Subject: [PATCH 256/333] ifelse() not falling back in case of undefined vars The existing test case apparently wasn't sufficient to catch a regression that may have occurred. Ticket: ENT-4653 Changelog: none (cherry picked from commit 8550de2be3042e6236f292e63620123c2cc675a7) --- libpromises/fncall.c | 9 ++--- .../01_vars/02_functions/ifelse_undefined.cf | 37 +++++++++++++++++++ 2 files changed, 41 insertions(+), 5 deletions(-) create mode 100644 tests/acceptance/01_vars/02_functions/ifelse_undefined.cf diff --git a/libpromises/fncall.c b/libpromises/fncall.c index fbe80c40ae..f2fb02a1e6 100644 --- a/libpromises/fncall.c +++ b/libpromises/fncall.c @@ -352,12 +352,11 @@ FnCallResult FnCallEvaluate(EvalContext *ctx, const Policy *policy, FnCall *fp, if ( ! (fp_type->options & FNCALL_OPTION_DELAYED_EVALUATION) && RlistIsUnresolved(expargs)) { - // Special case: ifelse(isvariable("x"), $(x), "default") - // (the first argument will come down expanded as "!any") + // Special case where a three argument ifelse call must + // be allowed to have undefined variables. if (strcmp(fp->name, "ifelse") == 0 && - RlistLen(expargs) == 3 && - strcmp("!any", RlistScalarValueSafe(expargs)) == 0 && - !RlistIsUnresolved(expargs->next->next)) + expargs->val.type != RVAL_TYPE_FNCALL && + RlistLen(expargs) == 3) { Log(LOG_LEVEL_DEBUG, "Allowing ifelse() function evaluation even" " though its arguments contain unresolved variables: %s", diff --git a/tests/acceptance/01_vars/02_functions/ifelse_undefined.cf b/tests/acceptance/01_vars/02_functions/ifelse_undefined.cf new file mode 100644 index 0000000000..96126464ec --- /dev/null +++ b/tests/acceptance/01_vars/02_functions/ifelse_undefined.cf @@ -0,0 +1,37 @@ +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +bundle agent test +{ + meta: + "description" -> { "ENT-4653" } + string => "Test that ifelse works with undefined variables in the second or third arguments"; + + vars: + "test_one" string => ifelse( "no_such_class", "$(no_such_var)", "test_one_expected_value" ); + "test_two" string => ifelse( "any", "test_two_expected_value", "$(no_such_var)" ); +} + +bundle agent check +{ + + reports: + '$(this.promise_filename) Pass' + if => and( + strcmp( "test_one_expected_value", $(test.test_one) ), + strcmp( "test_two_expected_value", $(test.test_two) ) ); + + '$(this.promise_filename) FAIL' + if => or( + not( isvariable( "test.test_one" ) ), + not( isvariable( "test.test_two" ) ) ); + + '$(this.propmise_filename) FAIL' + if => or( + not(strcmp( "test_one_expected_value", $(test.test_one) ) ), + not(strcmp( "test_two_expected_value", $(test.test_two) ) ) ); +} From 093bc32cbfa7ae7cc1695dcc5a21c6772ad3bd7b Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Wed, 22 Sep 2021 11:53:49 -0500 Subject: [PATCH 257/333] Changed IRC channel reference from freenode to libera.chat Ticket: ENT-7627 Changelog: None (cherry picked from commit a08082b21b835b5a607cb0706d9e0adcbec49639) --- CONTRIBUTING.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 54794a6115..926b50fa5a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,12 +2,11 @@ Thanks for considering contributing to CFEngine! We take pull-requests [on GitHub](https://github.com/cfengine/core/pulls) and we have a public -[bug-tracker](https://tracker.mender.io/projects/CFE/issues/). Discussion is taking place -on the [dev-cfengine](https://groups.google.com/forum/#!forum/dev-cfengine) -and [help-cfengine](https://groups.google.com/forum/#!forum/help-cfengine) -mailing lists. You'll find us chatting on Freenode's IRC channels -[#cfengine](https://webchat.freenode.net/?channels=cfengine&nick=) and -[#cfengine-dev](https://webchat.freenode.net/?channels=cfengine-dev&nick=). +[bug-tracker](https://tracker.mender.io/projects/CFE/issues/). Discussion +is taking place on [GitHub Discussions](https://github.com/cfengine/core/discussions) +and on the [help-cfengine](https://groups.google.com/forum/#!forum/help-cfengine) +mailing list. You'll find us chatting on the Libera.chat IRC channel +[#cfengine](https://web.libera.chat/?channel=cfengine). Normally, bug fixes have a higher chance of getting accepted than new features, but we certainly welcome feature contributions. If you have an idea From ef55226506267dd70c495106c21d9e43aea7e406 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Tue, 5 Oct 2021 12:21:30 -0500 Subject: [PATCH 258/333] Added example illustrating data_sysctlvalues() behavior Ticket: CFE-3783 Changelog: Title (cherry picked from commit 38488d41c87aef8ed74d407b5babc2eeaec08f66) --- examples/data_sysctlvalues.cf | 932 ++++++++++++++++++++++++++++++++++ 1 file changed, 932 insertions(+) create mode 100644 examples/data_sysctlvalues.cf diff --git a/examples/data_sysctlvalues.cf b/examples/data_sysctlvalues.cf new file mode 100644 index 0000000000..a9aad1e895 --- /dev/null +++ b/examples/data_sysctlvalues.cf @@ -0,0 +1,932 @@ +bundle agent inventory_sysctl +# @brief Inventory each sysctl variable +{ + vars: + # Get the complete set of sysctl variables and values as a data structure + "_sysctl_d" + data => data_sysctlvalues(), + unless => isvariable( _sysctl_d ); + + # Get the sysctl variable names + "_sysctl_i" + slist => getindices( "_sysctl_d" ); + + # Define a variable tagged for inventory for each sysctl variable + "sysctl[$(_sysctl_i)]" + string => "$(_sysctl_d[$(_sysctl_i)])", + meta => { "inventory", "attribute_name=Kernel Tunable $(_sysctl_i)" }; + + reports: + # Show what data_sysctlvalues() returned. + "data_sysctlvalues() returned:$(with)" + with => storejson( @(_sysctl_d) ); + +} +bundle agent __main__ +{ + methods: "inventory_sysctl"; + +} +#+begin_src mock_example_output +#@ ``` +#@ R: data_sysctlvalues() returned:{ +#@ "abi.vsyscall32": "1", +#@ "crypto.fips_enabled": "0", +#@ "debug.exception-trace": "1", +#@ "debug.kprobes-optimization": "1", +#@ "debug.panic_on_rcu_stall": "0", +#@ "dev.hpet.max-user-freq": "64", +#@ "dev.mac_hid.mouse_button2_keycode": "97", +#@ "dev.mac_hid.mouse_button3_keycode": "100", +#@ "dev.mac_hid.mouse_button_emulation": "0", +#@ "dev.parport.default.spintime": "500", +#@ "dev.parport.default.timeslice": "200", +#@ "dev.raid.speed_limit_max": "200000", +#@ "dev.raid.speed_limit_min": "1000", +#@ "dev.scsi.logging_level": "0", +#@ "fs.aio-max-nr": "65536", +#@ "fs.aio-nr": "0", +#@ "fs.binfmt_misc.status": "enabled", +#@ "fs.dentry-state": "43676\t27378\t45\t0\t479\t0", +#@ "fs.dir-notify-enable": "1", +#@ "fs.epoll.max_user_watches": "99020", +#@ "fs.file-max": "47380", +#@ "fs.file-nr": "1472\t0\t47380", +#@ "fs.inode-nr": "47116\t21990", +#@ "fs.inode-state": "47116\t21990\t0\t0\t0\t0\t0", +#@ "fs.inotify.max_queued_events": "16384", +#@ "fs.inotify.max_user_instances": "128", +#@ "fs.inotify.max_user_watches": "8192", +#@ "fs.lease-break-time": "45", +#@ "fs.leases-enable": "1", +#@ "fs.may_detach_mounts": "0", +#@ "fs.mount-max": "100000", +#@ "fs.mqueue.msg_default": "10", +#@ "fs.mqueue.msg_max": "10", +#@ "fs.mqueue.msgsize_default": "8192", +#@ "fs.mqueue.msgsize_max": "8192", +#@ "fs.mqueue.queues_max": "256", +#@ "fs.negative-dentry-limit": "0", +#@ "fs.nr_open": "1048576", +#@ "fs.overflowgid": "65534", +#@ "fs.overflowuid": "65534", +#@ "fs.pipe-max-size": "1048576", +#@ "fs.pipe-user-pages-hard": "0", +#@ "fs.pipe-user-pages-soft": "16384", +#@ "fs.protected_hardlinks": "1", +#@ "fs.protected_symlinks": "1", +#@ "fs.quota.allocated_dquots": "0", +#@ "fs.quota.cache_hits": "0", +#@ "fs.quota.drops": "0", +#@ "fs.quota.free_dquots": "0", +#@ "fs.quota.lookups": "0", +#@ "fs.quota.reads": "0", +#@ "fs.quota.syncs": "0", +#@ "fs.quota.warnings": "1", +#@ "fs.quota.writes": "0", +#@ "fs.suid_dumpable": "0", +#@ "fs.xfs.age_buffer_centisecs": "1500", +#@ "fs.xfs.error_level": "3", +#@ "fs.xfs.filestream_centisecs": "3000", +#@ "fs.xfs.inherit_noatime": "1", +#@ "fs.xfs.inherit_nodefrag": "1", +#@ "fs.xfs.inherit_nodump": "1", +#@ "fs.xfs.inherit_nosymlinks": "0", +#@ "fs.xfs.inherit_sync": "1", +#@ "fs.xfs.irix_sgid_inherit": "0", +#@ "fs.xfs.irix_symlink_mode": "0", +#@ "fs.xfs.panic_mask": "0", +#@ "fs.xfs.rotorstep": "1", +#@ "fs.xfs.speculative_prealloc_lifetime": "300", +#@ "fs.xfs.stats_clear": "0", +#@ "fs.xfs.xfsbufd_centisecs": "100", +#@ "fs.xfs.xfssyncd_centisecs": "3000", +#@ "kernel.acct": "4\t2\t30", +#@ "kernel.acpi_video_flags": "0", +#@ "kernel.auto_msgmni": "0", +#@ "kernel.bootloader_type": "114", +#@ "kernel.bootloader_version": "2", +#@ "kernel.cad_pid": "1", +#@ "kernel.cap_last_cap": "36", +#@ "kernel.compat-log": "1", +#@ "kernel.core_pattern": "core", +#@ "kernel.core_pipe_limit": "0", +#@ "kernel.core_uses_pid": "1", +#@ "kernel.ctrl-alt-del": "0", +#@ "kernel.dmesg_restrict": "0", +#@ "kernel.domainname": "(none)", +#@ "kernel.ftrace_dump_on_oops": "0", +#@ "kernel.ftrace_enabled": "1", +#@ "kernel.hardlockup_all_cpu_backtrace": "0", +#@ "kernel.hardlockup_panic": "1", +#@ "kernel.hostname": "hub.example.com", +#@ "kernel.hotplug": "", +#@ "kernel.hung_task_check_count": "4194304", +#@ "kernel.hung_task_panic": "0", +#@ "kernel.hung_task_timeout_secs": "120", +#@ "kernel.hung_task_warnings": "10", +#@ "kernel.io_delay_type": "0", +#@ "kernel.kexec_load_disabled": "0", +#@ "kernel.keys.gc_delay": "300", +#@ "kernel.keys.maxbytes": "20000", +#@ "kernel.keys.maxkeys": "200", +#@ "kernel.keys.persistent_keyring_expiry": "259200", +#@ "kernel.keys.root_maxbytes": "25000000", +#@ "kernel.keys.root_maxkeys": "1000000", +#@ "kernel.kptr_restrict": "0", +#@ "kernel.max_lock_depth": "1024", +#@ "kernel.modprobe": "/sbin/modprobe", +#@ "kernel.modules_disabled": "0", +#@ "kernel.msg_next_id": "-1", +#@ "kernel.msgmax": "8192", +#@ "kernel.msgmnb": "16384", +#@ "kernel.msgmni": "32000", +#@ "kernel.ngroups_max": "65536", +#@ "kernel.nmi_watchdog": "1", +#@ "kernel.ns_last_pid": "3240", +#@ "kernel.numa_balancing": "0", +#@ "kernel.numa_balancing_scan_delay_ms": "1000", +#@ "kernel.numa_balancing_scan_period_max_ms": "60000", +#@ "kernel.numa_balancing_scan_period_min_ms": "1000", +#@ "kernel.numa_balancing_scan_size_mb": "256", +#@ "kernel.numa_balancing_settle_count": "4", +#@ "kernel.osrelease": "3.10.0-1127.el7.x86_64", +#@ "kernel.ostype": "Linux", +#@ "kernel.overflowgid": "65534", +#@ "kernel.overflowuid": "65534", +#@ "kernel.panic": "0", +#@ "kernel.panic_on_io_nmi": "0", +#@ "kernel.panic_on_oops": "1", +#@ "kernel.panic_on_stackoverflow": "0", +#@ "kernel.panic_on_unrecovered_nmi": "0", +#@ "kernel.panic_on_warn": "0", +#@ "kernel.perf_cpu_time_max_percent": "25", +#@ "kernel.perf_event_max_sample_rate": "100000", +#@ "kernel.perf_event_mlock_kb": "516", +#@ "kernel.perf_event_paranoid": "2", +#@ "kernel.pid_max": "32768", +#@ "kernel.poweroff_cmd": "/sbin/poweroff", +#@ "kernel.print-fatal-signals": "0", +#@ "kernel.printk": "7\t4\t1\t7", +#@ "kernel.printk_delay": "0", +#@ "kernel.printk_ratelimit": "5", +#@ "kernel.printk_ratelimit_burst": "10", +#@ "kernel.pty.max": "4096", +#@ "kernel.pty.nr": "1", +#@ "kernel.pty.reserve": "1024", +#@ "kernel.random.boot_id": "422886c9-8219-4749-9610-981bdeb5e509", +#@ "kernel.random.entropy_avail": "3071", +#@ "kernel.random.poolsize": "4096", +#@ "kernel.random.read_wakeup_threshold": "64", +#@ "kernel.random.urandom_min_reseed_secs": "60", +#@ "kernel.random.uuid": "f9544436-b0a1-43b7-a36b-3b8da50e066a", +#@ "kernel.random.write_wakeup_threshold": "896", +#@ "kernel.randomize_va_space": "2", +#@ "kernel.real-root-dev": "0", +#@ "kernel.sched_autogroup_enabled": "0", +#@ "kernel.sched_cfs_bandwidth_slice_us": "5000", +#@ "kernel.sched_child_runs_first": "0", +#@ "kernel.sched_domain.cpu0.domain0.busy_factor": "32", +#@ "kernel.sched_domain.cpu0.domain0.busy_idx": "2", +#@ "kernel.sched_domain.cpu0.domain0.cache_nice_tries": "1", +#@ "kernel.sched_domain.cpu0.domain0.flags": "559", +#@ "kernel.sched_domain.cpu0.domain0.forkexec_idx": "0", +#@ "kernel.sched_domain.cpu0.domain0.idle_idx": "0", +#@ "kernel.sched_domain.cpu0.domain0.imbalance_pct": "117", +#@ "kernel.sched_domain.cpu0.domain0.max_interval": "4", +#@ "kernel.sched_domain.cpu0.domain0.max_newidle_lb_cost": "37143", +#@ "kernel.sched_domain.cpu0.domain0.min_interval": "2", +#@ "kernel.sched_domain.cpu0.domain0.name": "MC", +#@ "kernel.sched_domain.cpu0.domain0.newidle_idx": "0", +#@ "kernel.sched_domain.cpu0.domain0.wake_idx": "0", +#@ "kernel.sched_domain.cpu1.domain0.busy_factor": "32", +#@ "kernel.sched_domain.cpu1.domain0.busy_idx": "2", +#@ "kernel.sched_domain.cpu1.domain0.cache_nice_tries": "1", +#@ "kernel.sched_domain.cpu1.domain0.flags": "559", +#@ "kernel.sched_domain.cpu1.domain0.forkexec_idx": "0", +#@ "kernel.sched_domain.cpu1.domain0.idle_idx": "0", +#@ "kernel.sched_domain.cpu1.domain0.imbalance_pct": "117", +#@ "kernel.sched_domain.cpu1.domain0.max_interval": "4", +#@ "kernel.sched_domain.cpu1.domain0.max_newidle_lb_cost": "16770", +#@ "kernel.sched_domain.cpu1.domain0.min_interval": "2", +#@ "kernel.sched_domain.cpu1.domain0.name": "MC", +#@ "kernel.sched_domain.cpu1.domain0.newidle_idx": "0", +#@ "kernel.sched_domain.cpu1.domain0.wake_idx": "0", +#@ "kernel.sched_latency_ns": "12000000", +#@ "kernel.sched_migration_cost_ns": "500000", +#@ "kernel.sched_min_granularity_ns": "10000000", +#@ "kernel.sched_nr_migrate": "32", +#@ "kernel.sched_rr_timeslice_ms": "100", +#@ "kernel.sched_rt_period_us": "1000000", +#@ "kernel.sched_rt_runtime_us": "950000", +#@ "kernel.sched_schedstats": "0", +#@ "kernel.sched_shares_window_ns": "10000000", +#@ "kernel.sched_time_avg_ms": "1000", +#@ "kernel.sched_tunable_scaling": "1", +#@ "kernel.sched_wakeup_granularity_ns": "15000000", +#@ "kernel.seccomp.actions_avail": "kill trap errno trace allow", +#@ "kernel.seccomp.actions_logged": "kill trap errno trace", +#@ "kernel.sem": "250\t32000\t32\t128", +#@ "kernel.sem_next_id": "-1", +#@ "kernel.shm_next_id": "-1", +#@ "kernel.shm_rmid_forced": "0", +#@ "kernel.shmall": "18446744073692774399", +#@ "kernel.shmmax": "18446744073692774399", +#@ "kernel.shmmni": "4096", +#@ "kernel.softlockup_all_cpu_backtrace": "0", +#@ "kernel.softlockup_panic": "0", +#@ "kernel.stack_tracer_enabled": "0", +#@ "kernel.sysctl_writes_strict": "1", +#@ "kernel.sysrq": "16", +#@ "kernel.tainted": "0", +#@ "kernel.threads-max": "3777", +#@ "kernel.timer_migration": "1", +#@ "kernel.traceoff_on_warning": "0", +#@ "kernel.unknown_nmi_panic": "0", +#@ "kernel.usermodehelper.bset": "4294967295\t31", +#@ "kernel.usermodehelper.inheritable": "4294967295\t31", +#@ "kernel.version": "#1 SMP Tue Mar 31 23:36:51 UTC 2020", +#@ "kernel.watchdog": "1", +#@ "kernel.watchdog_cpumask": "0-1", +#@ "kernel.watchdog_thresh": "10", +#@ "kernel.yama.ptrace_scope": "0", +#@ "net.core.bpf_jit_enable": "1", +#@ "net.core.bpf_jit_harden": "1", +#@ "net.core.bpf_jit_kallsyms": "0", +#@ "net.core.busy_poll": "0", +#@ "net.core.busy_read": "0", +#@ "net.core.default_qdisc": "pfifo_fast", +#@ "net.core.dev_weight": "64", +#@ "net.core.dev_weight_rx_bias": "1", +#@ "net.core.dev_weight_tx_bias": "1", +#@ "net.core.message_burst": "10", +#@ "net.core.message_cost": "5", +#@ "net.core.netdev_budget": "300", +#@ "net.core.netdev_max_backlog": "1000", +#@ "net.core.netdev_rss_key": "00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00", +#@ "net.core.netdev_tstamp_prequeue": "1", +#@ "net.core.optmem_max": "20480", +#@ "net.core.rmem_default": "212992", +#@ "net.core.rmem_max": "212992", +#@ "net.core.rps_sock_flow_entries": "0", +#@ "net.core.somaxconn": "128", +#@ "net.core.warnings": "1", +#@ "net.core.wmem_default": "212992", +#@ "net.core.wmem_max": "212992", +#@ "net.core.xfrm_acq_expires": "30", +#@ "net.core.xfrm_aevent_etime": "10", +#@ "net.core.xfrm_aevent_rseqth": "2", +#@ "net.core.xfrm_larval_drop": "1", +#@ "net.ipv4.cipso_cache_bucket_size": "10", +#@ "net.ipv4.cipso_cache_enable": "1", +#@ "net.ipv4.cipso_rbm_optfmt": "0", +#@ "net.ipv4.cipso_rbm_strictvalid": "1", +#@ "net.ipv4.conf.all.accept_local": "0", +#@ "net.ipv4.conf.all.accept_redirects": "1", +#@ "net.ipv4.conf.all.accept_source_route": "0", +#@ "net.ipv4.conf.all.arp_accept": "0", +#@ "net.ipv4.conf.all.arp_announce": "0", +#@ "net.ipv4.conf.all.arp_filter": "0", +#@ "net.ipv4.conf.all.arp_ignore": "0", +#@ "net.ipv4.conf.all.arp_notify": "0", +#@ "net.ipv4.conf.all.bootp_relay": "0", +#@ "net.ipv4.conf.all.disable_policy": "0", +#@ "net.ipv4.conf.all.disable_xfrm": "0", +#@ "net.ipv4.conf.all.force_igmp_version": "0", +#@ "net.ipv4.conf.all.forwarding": "0", +#@ "net.ipv4.conf.all.igmpv2_unsolicited_report_interval": "10000", +#@ "net.ipv4.conf.all.igmpv3_unsolicited_report_interval": "1000", +#@ "net.ipv4.conf.all.log_martians": "0", +#@ "net.ipv4.conf.all.mc_forwarding": "0", +#@ "net.ipv4.conf.all.medium_id": "0", +#@ "net.ipv4.conf.all.promote_secondaries": "1", +#@ "net.ipv4.conf.all.proxy_arp": "0", +#@ "net.ipv4.conf.all.proxy_arp_pvlan": "0", +#@ "net.ipv4.conf.all.route_localnet": "0", +#@ "net.ipv4.conf.all.rp_filter": "1", +#@ "net.ipv4.conf.all.secure_redirects": "1", +#@ "net.ipv4.conf.all.send_redirects": "1", +#@ "net.ipv4.conf.all.shared_media": "1", +#@ "net.ipv4.conf.all.src_valid_mark": "0", +#@ "net.ipv4.conf.all.tag": "0", +#@ "net.ipv4.conf.default.accept_local": "0", +#@ "net.ipv4.conf.default.accept_redirects": "1", +#@ "net.ipv4.conf.default.accept_source_route": "0", +#@ "net.ipv4.conf.default.arp_accept": "0", +#@ "net.ipv4.conf.default.arp_announce": "0", +#@ "net.ipv4.conf.default.arp_filter": "0", +#@ "net.ipv4.conf.default.arp_ignore": "0", +#@ "net.ipv4.conf.default.arp_notify": "0", +#@ "net.ipv4.conf.default.bootp_relay": "0", +#@ "net.ipv4.conf.default.disable_policy": "0", +#@ "net.ipv4.conf.default.disable_xfrm": "0", +#@ "net.ipv4.conf.default.force_igmp_version": "0", +#@ "net.ipv4.conf.default.forwarding": "0", +#@ "net.ipv4.conf.default.igmpv2_unsolicited_report_interval": "10000", +#@ "net.ipv4.conf.default.igmpv3_unsolicited_report_interval": "1000", +#@ "net.ipv4.conf.default.log_martians": "0", +#@ "net.ipv4.conf.default.mc_forwarding": "0", +#@ "net.ipv4.conf.default.medium_id": "0", +#@ "net.ipv4.conf.default.promote_secondaries": "1", +#@ "net.ipv4.conf.default.proxy_arp": "0", +#@ "net.ipv4.conf.default.proxy_arp_pvlan": "0", +#@ "net.ipv4.conf.default.route_localnet": "0", +#@ "net.ipv4.conf.default.rp_filter": "1", +#@ "net.ipv4.conf.default.secure_redirects": "1", +#@ "net.ipv4.conf.default.send_redirects": "1", +#@ "net.ipv4.conf.default.shared_media": "1", +#@ "net.ipv4.conf.default.src_valid_mark": "0", +#@ "net.ipv4.conf.default.tag": "0", +#@ "net.ipv4.conf.eth0.accept_local": "0", +#@ "net.ipv4.conf.eth0.accept_redirects": "1", +#@ "net.ipv4.conf.eth0.accept_source_route": "0", +#@ "net.ipv4.conf.eth0.arp_accept": "0", +#@ "net.ipv4.conf.eth0.arp_announce": "0", +#@ "net.ipv4.conf.eth0.arp_filter": "0", +#@ "net.ipv4.conf.eth0.arp_ignore": "0", +#@ "net.ipv4.conf.eth0.arp_notify": "0", +#@ "net.ipv4.conf.eth0.bootp_relay": "0", +#@ "net.ipv4.conf.eth0.disable_policy": "0", +#@ "net.ipv4.conf.eth0.disable_xfrm": "0", +#@ "net.ipv4.conf.eth0.force_igmp_version": "0", +#@ "net.ipv4.conf.eth0.forwarding": "0", +#@ "net.ipv4.conf.eth0.igmpv2_unsolicited_report_interval": "10000", +#@ "net.ipv4.conf.eth0.igmpv3_unsolicited_report_interval": "1000", +#@ "net.ipv4.conf.eth0.log_martians": "0", +#@ "net.ipv4.conf.eth0.mc_forwarding": "0", +#@ "net.ipv4.conf.eth0.medium_id": "0", +#@ "net.ipv4.conf.eth0.promote_secondaries": "1", +#@ "net.ipv4.conf.eth0.proxy_arp": "0", +#@ "net.ipv4.conf.eth0.proxy_arp_pvlan": "0", +#@ "net.ipv4.conf.eth0.route_localnet": "0", +#@ "net.ipv4.conf.eth0.rp_filter": "1", +#@ "net.ipv4.conf.eth0.secure_redirects": "1", +#@ "net.ipv4.conf.eth0.send_redirects": "1", +#@ "net.ipv4.conf.eth0.shared_media": "1", +#@ "net.ipv4.conf.eth0.src_valid_mark": "0", +#@ "net.ipv4.conf.eth0.tag": "0", +#@ "net.ipv4.conf.eth1.accept_local": "0", +#@ "net.ipv4.conf.eth1.accept_redirects": "1", +#@ "net.ipv4.conf.eth1.accept_source_route": "0", +#@ "net.ipv4.conf.eth1.arp_accept": "0", +#@ "net.ipv4.conf.eth1.arp_announce": "0", +#@ "net.ipv4.conf.eth1.arp_filter": "0", +#@ "net.ipv4.conf.eth1.arp_ignore": "0", +#@ "net.ipv4.conf.eth1.arp_notify": "0", +#@ "net.ipv4.conf.eth1.bootp_relay": "0", +#@ "net.ipv4.conf.eth1.disable_policy": "0", +#@ "net.ipv4.conf.eth1.disable_xfrm": "0", +#@ "net.ipv4.conf.eth1.force_igmp_version": "0", +#@ "net.ipv4.conf.eth1.forwarding": "0", +#@ "net.ipv4.conf.eth1.igmpv2_unsolicited_report_interval": "10000", +#@ "net.ipv4.conf.eth1.igmpv3_unsolicited_report_interval": "1000", +#@ "net.ipv4.conf.eth1.log_martians": "0", +#@ "net.ipv4.conf.eth1.mc_forwarding": "0", +#@ "net.ipv4.conf.eth1.medium_id": "0", +#@ "net.ipv4.conf.eth1.promote_secondaries": "1", +#@ "net.ipv4.conf.eth1.proxy_arp": "0", +#@ "net.ipv4.conf.eth1.proxy_arp_pvlan": "0", +#@ "net.ipv4.conf.eth1.route_localnet": "0", +#@ "net.ipv4.conf.eth1.rp_filter": "1", +#@ "net.ipv4.conf.eth1.secure_redirects": "1", +#@ "net.ipv4.conf.eth1.send_redirects": "1", +#@ "net.ipv4.conf.eth1.shared_media": "1", +#@ "net.ipv4.conf.eth1.src_valid_mark": "0", +#@ "net.ipv4.conf.eth1.tag": "0", +#@ "net.ipv4.conf.lo.accept_local": "0", +#@ "net.ipv4.conf.lo.accept_redirects": "1", +#@ "net.ipv4.conf.lo.accept_source_route": "1", +#@ "net.ipv4.conf.lo.arp_accept": "0", +#@ "net.ipv4.conf.lo.arp_announce": "0", +#@ "net.ipv4.conf.lo.arp_filter": "0", +#@ "net.ipv4.conf.lo.arp_ignore": "0", +#@ "net.ipv4.conf.lo.arp_notify": "0", +#@ "net.ipv4.conf.lo.bootp_relay": "0", +#@ "net.ipv4.conf.lo.disable_policy": "1", +#@ "net.ipv4.conf.lo.disable_xfrm": "1", +#@ "net.ipv4.conf.lo.force_igmp_version": "0", +#@ "net.ipv4.conf.lo.forwarding": "0", +#@ "net.ipv4.conf.lo.igmpv2_unsolicited_report_interval": "10000", +#@ "net.ipv4.conf.lo.igmpv3_unsolicited_report_interval": "1000", +#@ "net.ipv4.conf.lo.log_martians": "0", +#@ "net.ipv4.conf.lo.mc_forwarding": "0", +#@ "net.ipv4.conf.lo.medium_id": "0", +#@ "net.ipv4.conf.lo.promote_secondaries": "0", +#@ "net.ipv4.conf.lo.proxy_arp": "0", +#@ "net.ipv4.conf.lo.proxy_arp_pvlan": "0", +#@ "net.ipv4.conf.lo.route_localnet": "0", +#@ "net.ipv4.conf.lo.rp_filter": "0", +#@ "net.ipv4.conf.lo.secure_redirects": "1", +#@ "net.ipv4.conf.lo.send_redirects": "1", +#@ "net.ipv4.conf.lo.shared_media": "1", +#@ "net.ipv4.conf.lo.src_valid_mark": "0", +#@ "net.ipv4.conf.lo.tag": "0", +#@ "net.ipv4.fib_multipath_hash_policy": "0", +#@ "net.ipv4.fwmark_reflect": "0", +#@ "net.ipv4.icmp_echo_ignore_all": "0", +#@ "net.ipv4.icmp_echo_ignore_broadcasts": "1", +#@ "net.ipv4.icmp_errors_use_inbound_ifaddr": "0", +#@ "net.ipv4.icmp_ignore_bogus_error_responses": "1", +#@ "net.ipv4.icmp_msgs_burst": "50", +#@ "net.ipv4.icmp_msgs_per_sec": "1000", +#@ "net.ipv4.icmp_ratelimit": "1000", +#@ "net.ipv4.icmp_ratemask": "6168", +#@ "net.ipv4.igmp_max_memberships": "20", +#@ "net.ipv4.igmp_max_msf": "10", +#@ "net.ipv4.igmp_qrv": "2", +#@ "net.ipv4.inet_peer_maxttl": "600", +#@ "net.ipv4.inet_peer_minttl": "120", +#@ "net.ipv4.inet_peer_threshold": "65664", +#@ "net.ipv4.ip_default_ttl": "64", +#@ "net.ipv4.ip_dynaddr": "0", +#@ "net.ipv4.ip_early_demux": "1", +#@ "net.ipv4.ip_forward": "0", +#@ "net.ipv4.ip_forward_use_pmtu": "0", +#@ "net.ipv4.ip_local_port_range": "32768\t60999", +#@ "net.ipv4.ip_local_reserved_ports": "", +#@ "net.ipv4.ip_no_pmtu_disc": "0", +#@ "net.ipv4.ip_nonlocal_bind": "0", +#@ "net.ipv4.ipfrag_high_thresh": "4194304", +#@ "net.ipv4.ipfrag_low_thresh": "3145728", +#@ "net.ipv4.ipfrag_max_dist": "64", +#@ "net.ipv4.ipfrag_secret_interval": "600", +#@ "net.ipv4.ipfrag_time": "30", +#@ "net.ipv4.neigh.default.anycast_delay": "100", +#@ "net.ipv4.neigh.default.app_solicit": "0", +#@ "net.ipv4.neigh.default.base_reachable_time": "30", +#@ "net.ipv4.neigh.default.base_reachable_time_ms": "30000", +#@ "net.ipv4.neigh.default.delay_first_probe_time": "5", +#@ "net.ipv4.neigh.default.gc_interval": "30", +#@ "net.ipv4.neigh.default.gc_stale_time": "60", +#@ "net.ipv4.neigh.default.gc_thresh1": "128", +#@ "net.ipv4.neigh.default.gc_thresh2": "512", +#@ "net.ipv4.neigh.default.gc_thresh3": "1024", +#@ "net.ipv4.neigh.default.locktime": "100", +#@ "net.ipv4.neigh.default.mcast_solicit": "3", +#@ "net.ipv4.neigh.default.proxy_delay": "80", +#@ "net.ipv4.neigh.default.proxy_qlen": "64", +#@ "net.ipv4.neigh.default.retrans_time": "100", +#@ "net.ipv4.neigh.default.retrans_time_ms": "1000", +#@ "net.ipv4.neigh.default.ucast_solicit": "3", +#@ "net.ipv4.neigh.default.unres_qlen": "31", +#@ "net.ipv4.neigh.default.unres_qlen_bytes": "65536", +#@ "net.ipv4.neigh.eth0.anycast_delay": "100", +#@ "net.ipv4.neigh.eth0.app_solicit": "0", +#@ "net.ipv4.neigh.eth0.base_reachable_time": "30", +#@ "net.ipv4.neigh.eth0.base_reachable_time_ms": "30000", +#@ "net.ipv4.neigh.eth0.delay_first_probe_time": "5", +#@ "net.ipv4.neigh.eth0.gc_stale_time": "60", +#@ "net.ipv4.neigh.eth0.locktime": "100", +#@ "net.ipv4.neigh.eth0.mcast_solicit": "3", +#@ "net.ipv4.neigh.eth0.proxy_delay": "80", +#@ "net.ipv4.neigh.eth0.proxy_qlen": "64", +#@ "net.ipv4.neigh.eth0.retrans_time": "100", +#@ "net.ipv4.neigh.eth0.retrans_time_ms": "1000", +#@ "net.ipv4.neigh.eth0.ucast_solicit": "3", +#@ "net.ipv4.neigh.eth0.unres_qlen": "31", +#@ "net.ipv4.neigh.eth0.unres_qlen_bytes": "65536", +#@ "net.ipv4.neigh.eth1.anycast_delay": "100", +#@ "net.ipv4.neigh.eth1.app_solicit": "0", +#@ "net.ipv4.neigh.eth1.base_reachable_time": "30", +#@ "net.ipv4.neigh.eth1.base_reachable_time_ms": "30000", +#@ "net.ipv4.neigh.eth1.delay_first_probe_time": "5", +#@ "net.ipv4.neigh.eth1.gc_stale_time": "60", +#@ "net.ipv4.neigh.eth1.locktime": "100", +#@ "net.ipv4.neigh.eth1.mcast_solicit": "3", +#@ "net.ipv4.neigh.eth1.proxy_delay": "80", +#@ "net.ipv4.neigh.eth1.proxy_qlen": "64", +#@ "net.ipv4.neigh.eth1.retrans_time": "100", +#@ "net.ipv4.neigh.eth1.retrans_time_ms": "1000", +#@ "net.ipv4.neigh.eth1.ucast_solicit": "3", +#@ "net.ipv4.neigh.eth1.unres_qlen": "31", +#@ "net.ipv4.neigh.eth1.unres_qlen_bytes": "65536", +#@ "net.ipv4.neigh.lo.anycast_delay": "100", +#@ "net.ipv4.neigh.lo.app_solicit": "0", +#@ "net.ipv4.neigh.lo.base_reachable_time": "30", +#@ "net.ipv4.neigh.lo.base_reachable_time_ms": "30000", +#@ "net.ipv4.neigh.lo.delay_first_probe_time": "5", +#@ "net.ipv4.neigh.lo.gc_stale_time": "60", +#@ "net.ipv4.neigh.lo.locktime": "100", +#@ "net.ipv4.neigh.lo.mcast_solicit": "3", +#@ "net.ipv4.neigh.lo.proxy_delay": "80", +#@ "net.ipv4.neigh.lo.proxy_qlen": "64", +#@ "net.ipv4.neigh.lo.retrans_time": "100", +#@ "net.ipv4.neigh.lo.retrans_time_ms": "1000", +#@ "net.ipv4.neigh.lo.ucast_solicit": "3", +#@ "net.ipv4.neigh.lo.unres_qlen": "31", +#@ "net.ipv4.neigh.lo.unres_qlen_bytes": "65536", +#@ "net.ipv4.ping_group_range": "1\t0", +#@ "net.ipv4.route.error_burst": "5000", +#@ "net.ipv4.route.error_cost": "1000", +#@ "net.ipv4.route.gc_elasticity": "8", +#@ "net.ipv4.route.gc_interval": "60", +#@ "net.ipv4.route.gc_min_interval": "0", +#@ "net.ipv4.route.gc_min_interval_ms": "500", +#@ "net.ipv4.route.gc_thresh": "-1", +#@ "net.ipv4.route.gc_timeout": "300", +#@ "net.ipv4.route.max_size": "2147483647", +#@ "net.ipv4.route.min_adv_mss": "256", +#@ "net.ipv4.route.min_pmtu": "552", +#@ "net.ipv4.route.mtu_expires": "600", +#@ "net.ipv4.route.redirect_load": "20", +#@ "net.ipv4.route.redirect_number": "9", +#@ "net.ipv4.route.redirect_silence": "20480", +#@ "net.ipv4.tcp_abort_on_overflow": "0", +#@ "net.ipv4.tcp_adv_win_scale": "1", +#@ "net.ipv4.tcp_allowed_congestion_control": "cubic reno", +#@ "net.ipv4.tcp_app_win": "31", +#@ "net.ipv4.tcp_autocorking": "1", +#@ "net.ipv4.tcp_available_congestion_control": "cubic reno", +#@ "net.ipv4.tcp_base_mss": "512", +#@ "net.ipv4.tcp_challenge_ack_limit": "1000", +#@ "net.ipv4.tcp_congestion_control": "cubic", +#@ "net.ipv4.tcp_dsack": "1", +#@ "net.ipv4.tcp_early_retrans": "3", +#@ "net.ipv4.tcp_ecn": "2", +#@ "net.ipv4.tcp_fack": "1", +#@ "net.ipv4.tcp_fastopen": "0", +#@ "net.ipv4.tcp_fastopen_key": "00000000-00000000-00000000-00000000", +#@ "net.ipv4.tcp_fin_timeout": "60", +#@ "net.ipv4.tcp_frto": "2", +#@ "net.ipv4.tcp_invalid_ratelimit": "500", +#@ "net.ipv4.tcp_keepalive_intvl": "75", +#@ "net.ipv4.tcp_keepalive_probes": "9", +#@ "net.ipv4.tcp_keepalive_time": "7200", +#@ "net.ipv4.tcp_limit_output_bytes": "262144", +#@ "net.ipv4.tcp_low_latency": "0", +#@ "net.ipv4.tcp_max_orphans": "2048", +#@ "net.ipv4.tcp_max_ssthresh": "0", +#@ "net.ipv4.tcp_max_syn_backlog": "128", +#@ "net.ipv4.tcp_max_tw_buckets": "2048", +#@ "net.ipv4.tcp_mem": "11517\t15356\t23034", +#@ "net.ipv4.tcp_min_snd_mss": "48", +#@ "net.ipv4.tcp_min_tso_segs": "2", +#@ "net.ipv4.tcp_moderate_rcvbuf": "1", +#@ "net.ipv4.tcp_mtu_probing": "0", +#@ "net.ipv4.tcp_no_metrics_save": "0", +#@ "net.ipv4.tcp_notsent_lowat": "-1", +#@ "net.ipv4.tcp_orphan_retries": "0", +#@ "net.ipv4.tcp_reordering": "3", +#@ "net.ipv4.tcp_retrans_collapse": "1", +#@ "net.ipv4.tcp_retries1": "3", +#@ "net.ipv4.tcp_retries2": "15", +#@ "net.ipv4.tcp_rfc1337": "0", +#@ "net.ipv4.tcp_rmem": "4096\t87380\t3868000", +#@ "net.ipv4.tcp_sack": "1", +#@ "net.ipv4.tcp_slow_start_after_idle": "1", +#@ "net.ipv4.tcp_stdurg": "0", +#@ "net.ipv4.tcp_syn_retries": "6", +#@ "net.ipv4.tcp_synack_retries": "5", +#@ "net.ipv4.tcp_syncookies": "1", +#@ "net.ipv4.tcp_thin_dupack": "0", +#@ "net.ipv4.tcp_thin_linear_timeouts": "0", +#@ "net.ipv4.tcp_timestamps": "1", +#@ "net.ipv4.tcp_tso_win_divisor": "3", +#@ "net.ipv4.tcp_tw_recycle": "0", +#@ "net.ipv4.tcp_tw_reuse": "0", +#@ "net.ipv4.tcp_window_scaling": "1", +#@ "net.ipv4.tcp_wmem": "4096\t16384\t3868000", +#@ "net.ipv4.tcp_workaround_signed_windows": "0", +#@ "net.ipv4.udp_mem": "11331\t15109\t22662", +#@ "net.ipv4.udp_rmem_min": "4096", +#@ "net.ipv4.udp_wmem_min": "4096", +#@ "net.ipv4.xfrm4_gc_thresh": "32768", +#@ "net.ipv6.anycast_src_echo_reply": "0", +#@ "net.ipv6.bindv6only": "0", +#@ "net.ipv6.conf.all.accept_dad": "0", +#@ "net.ipv6.conf.all.accept_ra": "1", +#@ "net.ipv6.conf.all.accept_ra_defrtr": "1", +#@ "net.ipv6.conf.all.accept_ra_pinfo": "1", +#@ "net.ipv6.conf.all.accept_ra_rt_info_max_plen": "0", +#@ "net.ipv6.conf.all.accept_ra_rtr_pref": "1", +#@ "net.ipv6.conf.all.accept_redirects": "1", +#@ "net.ipv6.conf.all.accept_source_route": "0", +#@ "net.ipv6.conf.all.autoconf": "1", +#@ "net.ipv6.conf.all.dad_transmits": "1", +#@ "net.ipv6.conf.all.disable_ipv6": "0", +#@ "net.ipv6.conf.all.enhanced_dad": "1", +#@ "net.ipv6.conf.all.force_mld_version": "0", +#@ "net.ipv6.conf.all.force_tllao": "0", +#@ "net.ipv6.conf.all.forwarding": "0", +#@ "net.ipv6.conf.all.hop_limit": "64", +#@ "net.ipv6.conf.all.keep_addr_on_down": "0", +#@ "net.ipv6.conf.all.max_addresses": "16", +#@ "net.ipv6.conf.all.max_desync_factor": "600", +#@ "net.ipv6.conf.all.mc_forwarding": "0", +#@ "net.ipv6.conf.all.mldv1_unsolicited_report_interval": "10000", +#@ "net.ipv6.conf.all.mldv2_unsolicited_report_interval": "1000", +#@ "net.ipv6.conf.all.mtu": "1280", +#@ "net.ipv6.conf.all.ndisc_notify": "0", +#@ "net.ipv6.conf.all.optimistic_dad": "0", +#@ "net.ipv6.conf.all.proxy_ndp": "0", +#@ "net.ipv6.conf.all.regen_max_retry": "3", +#@ "net.ipv6.conf.all.router_probe_interval": "60", +#@ "net.ipv6.conf.all.router_solicitation_delay": "1", +#@ "net.ipv6.conf.all.router_solicitation_interval": "4", +#@ "net.ipv6.conf.all.router_solicitations": "3", +#@ "net.ipv6.conf.all.temp_prefered_lft": "86400", +#@ "net.ipv6.conf.all.temp_valid_lft": "604800", +#@ "net.ipv6.conf.all.use_optimistic": "0", +#@ "net.ipv6.conf.all.use_tempaddr": "0", +#@ "net.ipv6.conf.default.accept_dad": "1", +#@ "net.ipv6.conf.default.accept_ra": "1", +#@ "net.ipv6.conf.default.accept_ra_defrtr": "1", +#@ "net.ipv6.conf.default.accept_ra_pinfo": "1", +#@ "net.ipv6.conf.default.accept_ra_rt_info_max_plen": "0", +#@ "net.ipv6.conf.default.accept_ra_rtr_pref": "1", +#@ "net.ipv6.conf.default.accept_redirects": "1", +#@ "net.ipv6.conf.default.accept_source_route": "0", +#@ "net.ipv6.conf.default.autoconf": "1", +#@ "net.ipv6.conf.default.dad_transmits": "1", +#@ "net.ipv6.conf.default.disable_ipv6": "0", +#@ "net.ipv6.conf.default.enhanced_dad": "1", +#@ "net.ipv6.conf.default.force_mld_version": "0", +#@ "net.ipv6.conf.default.force_tllao": "0", +#@ "net.ipv6.conf.default.forwarding": "0", +#@ "net.ipv6.conf.default.hop_limit": "64", +#@ "net.ipv6.conf.default.keep_addr_on_down": "0", +#@ "net.ipv6.conf.default.max_addresses": "16", +#@ "net.ipv6.conf.default.max_desync_factor": "600", +#@ "net.ipv6.conf.default.mc_forwarding": "0", +#@ "net.ipv6.conf.default.mldv1_unsolicited_report_interval": "10000", +#@ "net.ipv6.conf.default.mldv2_unsolicited_report_interval": "1000", +#@ "net.ipv6.conf.default.mtu": "1280", +#@ "net.ipv6.conf.default.ndisc_notify": "0", +#@ "net.ipv6.conf.default.optimistic_dad": "0", +#@ "net.ipv6.conf.default.proxy_ndp": "0", +#@ "net.ipv6.conf.default.regen_max_retry": "3", +#@ "net.ipv6.conf.default.router_probe_interval": "60", +#@ "net.ipv6.conf.default.router_solicitation_delay": "1", +#@ "net.ipv6.conf.default.router_solicitation_interval": "4", +#@ "net.ipv6.conf.default.router_solicitations": "3", +#@ "net.ipv6.conf.default.temp_prefered_lft": "86400", +#@ "net.ipv6.conf.default.temp_valid_lft": "604800", +#@ "net.ipv6.conf.default.use_optimistic": "0", +#@ "net.ipv6.conf.default.use_tempaddr": "0", +#@ "net.ipv6.conf.eth0.accept_dad": "1", +#@ "net.ipv6.conf.eth0.accept_ra": "1", +#@ "net.ipv6.conf.eth0.accept_ra_defrtr": "1", +#@ "net.ipv6.conf.eth0.accept_ra_pinfo": "1", +#@ "net.ipv6.conf.eth0.accept_ra_rt_info_max_plen": "0", +#@ "net.ipv6.conf.eth0.accept_ra_rtr_pref": "1", +#@ "net.ipv6.conf.eth0.accept_redirects": "1", +#@ "net.ipv6.conf.eth0.accept_source_route": "0", +#@ "net.ipv6.conf.eth0.autoconf": "1", +#@ "net.ipv6.conf.eth0.dad_transmits": "1", +#@ "net.ipv6.conf.eth0.disable_ipv6": "0", +#@ "net.ipv6.conf.eth0.enhanced_dad": "1", +#@ "net.ipv6.conf.eth0.force_mld_version": "0", +#@ "net.ipv6.conf.eth0.force_tllao": "0", +#@ "net.ipv6.conf.eth0.forwarding": "0", +#@ "net.ipv6.conf.eth0.hop_limit": "64", +#@ "net.ipv6.conf.eth0.keep_addr_on_down": "0", +#@ "net.ipv6.conf.eth0.max_addresses": "16", +#@ "net.ipv6.conf.eth0.max_desync_factor": "600", +#@ "net.ipv6.conf.eth0.mc_forwarding": "0", +#@ "net.ipv6.conf.eth0.mldv1_unsolicited_report_interval": "10000", +#@ "net.ipv6.conf.eth0.mldv2_unsolicited_report_interval": "1000", +#@ "net.ipv6.conf.eth0.mtu": "1500", +#@ "net.ipv6.conf.eth0.ndisc_notify": "0", +#@ "net.ipv6.conf.eth0.optimistic_dad": "0", +#@ "net.ipv6.conf.eth0.proxy_ndp": "0", +#@ "net.ipv6.conf.eth0.regen_max_retry": "3", +#@ "net.ipv6.conf.eth0.router_probe_interval": "60", +#@ "net.ipv6.conf.eth0.router_solicitation_delay": "1", +#@ "net.ipv6.conf.eth0.router_solicitation_interval": "4", +#@ "net.ipv6.conf.eth0.router_solicitations": "3", +#@ "net.ipv6.conf.eth0.temp_prefered_lft": "86400", +#@ "net.ipv6.conf.eth0.temp_valid_lft": "604800", +#@ "net.ipv6.conf.eth0.use_optimistic": "0", +#@ "net.ipv6.conf.eth0.use_tempaddr": "0", +#@ "net.ipv6.conf.eth1.accept_dad": "1", +#@ "net.ipv6.conf.eth1.accept_ra": "0", +#@ "net.ipv6.conf.eth1.accept_ra_defrtr": "0", +#@ "net.ipv6.conf.eth1.accept_ra_pinfo": "0", +#@ "net.ipv6.conf.eth1.accept_ra_rt_info_max_plen": "0", +#@ "net.ipv6.conf.eth1.accept_ra_rtr_pref": "0", +#@ "net.ipv6.conf.eth1.accept_redirects": "1", +#@ "net.ipv6.conf.eth1.accept_source_route": "0", +#@ "net.ipv6.conf.eth1.autoconf": "1", +#@ "net.ipv6.conf.eth1.dad_transmits": "1", +#@ "net.ipv6.conf.eth1.disable_ipv6": "0", +#@ "net.ipv6.conf.eth1.enhanced_dad": "1", +#@ "net.ipv6.conf.eth1.force_mld_version": "0", +#@ "net.ipv6.conf.eth1.force_tllao": "0", +#@ "net.ipv6.conf.eth1.forwarding": "0", +#@ "net.ipv6.conf.eth1.hop_limit": "64", +#@ "net.ipv6.conf.eth1.keep_addr_on_down": "0", +#@ "net.ipv6.conf.eth1.max_addresses": "16", +#@ "net.ipv6.conf.eth1.max_desync_factor": "600", +#@ "net.ipv6.conf.eth1.mc_forwarding": "0", +#@ "net.ipv6.conf.eth1.mldv1_unsolicited_report_interval": "10000", +#@ "net.ipv6.conf.eth1.mldv2_unsolicited_report_interval": "1000", +#@ "net.ipv6.conf.eth1.mtu": "1500", +#@ "net.ipv6.conf.eth1.ndisc_notify": "0", +#@ "net.ipv6.conf.eth1.optimistic_dad": "0", +#@ "net.ipv6.conf.eth1.proxy_ndp": "0", +#@ "net.ipv6.conf.eth1.regen_max_retry": "3", +#@ "net.ipv6.conf.eth1.router_probe_interval": "60", +#@ "net.ipv6.conf.eth1.router_solicitation_delay": "1", +#@ "net.ipv6.conf.eth1.router_solicitation_interval": "4", +#@ "net.ipv6.conf.eth1.router_solicitations": "3", +#@ "net.ipv6.conf.eth1.temp_prefered_lft": "86400", +#@ "net.ipv6.conf.eth1.temp_valid_lft": "604800", +#@ "net.ipv6.conf.eth1.use_optimistic": "0", +#@ "net.ipv6.conf.eth1.use_tempaddr": "0", +#@ "net.ipv6.conf.lo.accept_dad": "-1", +#@ "net.ipv6.conf.lo.accept_ra": "1", +#@ "net.ipv6.conf.lo.accept_ra_defrtr": "1", +#@ "net.ipv6.conf.lo.accept_ra_pinfo": "1", +#@ "net.ipv6.conf.lo.accept_ra_rt_info_max_plen": "0", +#@ "net.ipv6.conf.lo.accept_ra_rtr_pref": "1", +#@ "net.ipv6.conf.lo.accept_redirects": "1", +#@ "net.ipv6.conf.lo.accept_source_route": "0", +#@ "net.ipv6.conf.lo.autoconf": "1", +#@ "net.ipv6.conf.lo.dad_transmits": "1", +#@ "net.ipv6.conf.lo.disable_ipv6": "0", +#@ "net.ipv6.conf.lo.enhanced_dad": "1", +#@ "net.ipv6.conf.lo.force_mld_version": "0", +#@ "net.ipv6.conf.lo.force_tllao": "0", +#@ "net.ipv6.conf.lo.forwarding": "0", +#@ "net.ipv6.conf.lo.hop_limit": "64", +#@ "net.ipv6.conf.lo.keep_addr_on_down": "0", +#@ "net.ipv6.conf.lo.max_addresses": "16", +#@ "net.ipv6.conf.lo.max_desync_factor": "600", +#@ "net.ipv6.conf.lo.mc_forwarding": "0", +#@ "net.ipv6.conf.lo.mldv1_unsolicited_report_interval": "10000", +#@ "net.ipv6.conf.lo.mldv2_unsolicited_report_interval": "1000", +#@ "net.ipv6.conf.lo.mtu": "65536", +#@ "net.ipv6.conf.lo.ndisc_notify": "0", +#@ "net.ipv6.conf.lo.optimistic_dad": "0", +#@ "net.ipv6.conf.lo.proxy_ndp": "0", +#@ "net.ipv6.conf.lo.regen_max_retry": "3", +#@ "net.ipv6.conf.lo.router_probe_interval": "60", +#@ "net.ipv6.conf.lo.router_solicitation_delay": "1", +#@ "net.ipv6.conf.lo.router_solicitation_interval": "4", +#@ "net.ipv6.conf.lo.router_solicitations": "3", +#@ "net.ipv6.conf.lo.temp_prefered_lft": "86400", +#@ "net.ipv6.conf.lo.temp_valid_lft": "604800", +#@ "net.ipv6.conf.lo.use_optimistic": "0", +#@ "net.ipv6.conf.lo.use_tempaddr": "-1", +#@ "net.ipv6.fwmark_reflect": "0", +#@ "net.ipv6.icmp.ratelimit": "1000", +#@ "net.ipv6.idgen_delay": "1", +#@ "net.ipv6.idgen_retries": "3", +#@ "net.ipv6.ip6frag_high_thresh": "4194304", +#@ "net.ipv6.ip6frag_low_thresh": "3145728", +#@ "net.ipv6.ip6frag_secret_interval": "600", +#@ "net.ipv6.ip6frag_time": "60", +#@ "net.ipv6.ip_nonlocal_bind": "0", +#@ "net.ipv6.mld_max_msf": "64", +#@ "net.ipv6.mld_qrv": "2", +#@ "net.ipv6.neigh.default.anycast_delay": "100", +#@ "net.ipv6.neigh.default.app_solicit": "0", +#@ "net.ipv6.neigh.default.base_reachable_time": "30", +#@ "net.ipv6.neigh.default.base_reachable_time_ms": "30000", +#@ "net.ipv6.neigh.default.delay_first_probe_time": "5", +#@ "net.ipv6.neigh.default.gc_interval": "30", +#@ "net.ipv6.neigh.default.gc_stale_time": "60", +#@ "net.ipv6.neigh.default.gc_thresh1": "128", +#@ "net.ipv6.neigh.default.gc_thresh2": "512", +#@ "net.ipv6.neigh.default.gc_thresh3": "1024", +#@ "net.ipv6.neigh.default.locktime": "0", +#@ "net.ipv6.neigh.default.mcast_solicit": "3", +#@ "net.ipv6.neigh.default.proxy_delay": "80", +#@ "net.ipv6.neigh.default.proxy_qlen": "64", +#@ "net.ipv6.neigh.default.retrans_time": "1000", +#@ "net.ipv6.neigh.default.retrans_time_ms": "1000", +#@ "net.ipv6.neigh.default.ucast_solicit": "3", +#@ "net.ipv6.neigh.default.unres_qlen": "31", +#@ "net.ipv6.neigh.default.unres_qlen_bytes": "65536", +#@ "net.ipv6.neigh.eth0.anycast_delay": "100", +#@ "net.ipv6.neigh.eth0.app_solicit": "0", +#@ "net.ipv6.neigh.eth0.base_reachable_time": "30", +#@ "net.ipv6.neigh.eth0.base_reachable_time_ms": "30000", +#@ "net.ipv6.neigh.eth0.delay_first_probe_time": "5", +#@ "net.ipv6.neigh.eth0.gc_stale_time": "60", +#@ "net.ipv6.neigh.eth0.locktime": "0", +#@ "net.ipv6.neigh.eth0.mcast_solicit": "3", +#@ "net.ipv6.neigh.eth0.proxy_delay": "80", +#@ "net.ipv6.neigh.eth0.proxy_qlen": "64", +#@ "net.ipv6.neigh.eth0.retrans_time": "1000", +#@ "net.ipv6.neigh.eth0.retrans_time_ms": "1000", +#@ "net.ipv6.neigh.eth0.ucast_solicit": "3", +#@ "net.ipv6.neigh.eth0.unres_qlen": "31", +#@ "net.ipv6.neigh.eth0.unres_qlen_bytes": "65536", +#@ "net.ipv6.neigh.eth1.anycast_delay": "100", +#@ "net.ipv6.neigh.eth1.app_solicit": "0", +#@ "net.ipv6.neigh.eth1.base_reachable_time": "30", +#@ "net.ipv6.neigh.eth1.base_reachable_time_ms": "30000", +#@ "net.ipv6.neigh.eth1.delay_first_probe_time": "5", +#@ "net.ipv6.neigh.eth1.gc_stale_time": "60", +#@ "net.ipv6.neigh.eth1.locktime": "0", +#@ "net.ipv6.neigh.eth1.mcast_solicit": "3", +#@ "net.ipv6.neigh.eth1.proxy_delay": "80", +#@ "net.ipv6.neigh.eth1.proxy_qlen": "64", +#@ "net.ipv6.neigh.eth1.retrans_time": "1000", +#@ "net.ipv6.neigh.eth1.retrans_time_ms": "1000", +#@ "net.ipv6.neigh.eth1.ucast_solicit": "3", +#@ "net.ipv6.neigh.eth1.unres_qlen": "31", +#@ "net.ipv6.neigh.eth1.unres_qlen_bytes": "65536", +#@ "net.ipv6.neigh.lo.anycast_delay": "100", +#@ "net.ipv6.neigh.lo.app_solicit": "0", +#@ "net.ipv6.neigh.lo.base_reachable_time": "30", +#@ "net.ipv6.neigh.lo.base_reachable_time_ms": "30000", +#@ "net.ipv6.neigh.lo.delay_first_probe_time": "5", +#@ "net.ipv6.neigh.lo.gc_stale_time": "60", +#@ "net.ipv6.neigh.lo.locktime": "0", +#@ "net.ipv6.neigh.lo.mcast_solicit": "3", +#@ "net.ipv6.neigh.lo.proxy_delay": "80", +#@ "net.ipv6.neigh.lo.proxy_qlen": "64", +#@ "net.ipv6.neigh.lo.retrans_time": "1000", +#@ "net.ipv6.neigh.lo.retrans_time_ms": "1000", +#@ "net.ipv6.neigh.lo.ucast_solicit": "3", +#@ "net.ipv6.neigh.lo.unres_qlen": "31", +#@ "net.ipv6.neigh.lo.unres_qlen_bytes": "65536", +#@ "net.ipv6.route.gc_elasticity": "9", +#@ "net.ipv6.route.gc_interval": "30", +#@ "net.ipv6.route.gc_min_interval": "0", +#@ "net.ipv6.route.gc_min_interval_ms": "500", +#@ "net.ipv6.route.gc_thresh": "1024", +#@ "net.ipv6.route.gc_timeout": "60", +#@ "net.ipv6.route.max_size": "16384", +#@ "net.ipv6.route.min_adv_mss": "1220", +#@ "net.ipv6.route.mtu_expires": "600", +#@ "net.ipv6.xfrm6_gc_thresh": "32768", +#@ "net.netfilter.nf_log.0": "NONE", +#@ "net.netfilter.nf_log.1": "NONE", +#@ "net.netfilter.nf_log.10": "NONE", +#@ "net.netfilter.nf_log.11": "NONE", +#@ "net.netfilter.nf_log.12": "NONE", +#@ "net.netfilter.nf_log.2": "NONE", +#@ "net.netfilter.nf_log.3": "NONE", +#@ "net.netfilter.nf_log.4": "NONE", +#@ "net.netfilter.nf_log.5": "NONE", +#@ "net.netfilter.nf_log.6": "NONE", +#@ "net.netfilter.nf_log.7": "NONE", +#@ "net.netfilter.nf_log.8": "NONE", +#@ "net.netfilter.nf_log.9": "NONE", +#@ "net.netfilter.nf_log_all_netns": "0", +#@ "net.unix.max_dgram_qlen": "512", +#@ "sunrpc.max_resvport": "1023", +#@ "sunrpc.min_resvport": "665", +#@ "sunrpc.nfs_debug": "0x0000", +#@ "sunrpc.nfsd_debug": "0x0000", +#@ "sunrpc.nlm_debug": "0x0000", +#@ "sunrpc.rpc_debug": "0x0000", +#@ "sunrpc.tcp_fin_timeout": "15", +#@ "sunrpc.tcp_max_slot_table_entries": "65536", +#@ "sunrpc.tcp_slot_table_entries": "2", +#@ "sunrpc.transports": "tcp 1048576\nudp 32768", +#@ "sunrpc.udp_slot_table_entries": "16", +#@ "user.max_ipc_namespaces": "1888", +#@ "user.max_mnt_namespaces": "1888", +#@ "user.max_net_namespaces": "1888", +#@ "user.max_pid_namespaces": "1888", +#@ "user.max_user_namespaces": "0", +#@ "user.max_uts_namespaces": "1888", +#@ "vm.admin_reserve_kbytes": "8192", +#@ "vm.block_dump": "0", +#@ "vm.dirty_background_bytes": "0", +#@ "vm.dirty_background_ratio": "10", +#@ "vm.dirty_bytes": "0", +#@ "vm.dirty_expire_centisecs": "3000", +#@ "vm.dirty_ratio": "30", +#@ "vm.dirty_writeback_centisecs": "500", +#@ "vm.drop_caches": "0", +#@ "vm.extfrag_threshold": "500", +#@ "vm.hugepages_treat_as_movable": "0", +#@ "vm.hugetlb_shm_group": "0", +#@ "vm.laptop_mode": "0", +#@ "vm.legacy_va_layout": "0", +#@ "vm.lowmem_reserve_ratio": "256\t256\t32", +#@ "vm.max_map_count": "65530", +#@ "vm.memory_failure_early_kill": "0", +#@ "vm.memory_failure_recovery": "1", +#@ "vm.min_free_kbytes": "2816", +#@ "vm.min_slab_ratio": "5", +#@ "vm.min_unmapped_ratio": "1", +#@ "vm.mmap_min_addr": "4096", +#@ "vm.mmap_rnd_bits": "28", +#@ "vm.mmap_rnd_compat_bits": "8", +#@ "vm.nr_hugepages": "0", +#@ "vm.nr_hugepages_mempolicy": "0", +#@ "vm.nr_overcommit_hugepages": "0", +#@ "vm.nr_pdflush_threads": "0", +#@ "vm.numa_zonelist_order": "default", +#@ "vm.oom_dump_tasks": "1", +#@ "vm.oom_kill_allocating_task": "0", +#@ "vm.overcommit_kbytes": "0", +#@ "vm.overcommit_memory": "0", +#@ "vm.overcommit_ratio": "50", +#@ "vm.page-cluster": "3", +#@ "vm.panic_on_oom": "0", +#@ "vm.percpu_pagelist_fraction": "0", +#@ "vm.stat_interval": "1", +#@ "vm.swappiness": "30", +#@ "vm.user_reserve_kbytes": "14160", +#@ "vm.vfs_cache_pressure": "100", +#@ "vm.zone_reclaim_mode": "0" +#@ } +#@ ``` +#+end_src From 4403c4f162715dd9d5cdb7f53cfd3bd26202e233 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Tue, 5 Oct 2021 15:13:33 -0500 Subject: [PATCH 259/333] Wrapped example in src block for inclusion from documentation (cherry picked from commit b9cb2172cbe391d79215876c159fb2c16632f75a) --- examples/data_sysctlvalues.cf | 3 +++ 1 file changed, 3 insertions(+) diff --git a/examples/data_sysctlvalues.cf b/examples/data_sysctlvalues.cf index a9aad1e895..d2c56a8e71 100644 --- a/examples/data_sysctlvalues.cf +++ b/examples/data_sysctlvalues.cf @@ -1,3 +1,4 @@ +#+begin_src cfengine3 bundle agent inventory_sysctl # @brief Inventory each sysctl variable { @@ -27,6 +28,8 @@ bundle agent __main__ methods: "inventory_sysctl"; } +#+end_src +############################################################################### #+begin_src mock_example_output #@ ``` #@ R: data_sysctlvalues() returned:{ From cf511c9258cdf94d4ae19fea11fa09415d640a65 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 13 Oct 2021 13:56:08 +0200 Subject: [PATCH 260/333] Expand '$(with)' in the last eval pass if it has unresolved vars Even if the value of '$(with)' (the 'with' attribute) contains unexpanded variables in the last evaluation pass it should be expanded/replaced to/with the value. Ticket: CFE-3776 Changelog: Value of '$(with)' is now expanded even if it contains unresolved variable references (cherry picked from commit 88dce716e46cb7a9e3967df330c6d84b1814787b) --- libpromises/eval_context.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/libpromises/eval_context.c b/libpromises/eval_context.c index 1ab5a1ff10..ccbc1732f0 100644 --- a/libpromises/eval_context.c +++ b/libpromises/eval_context.c @@ -1481,7 +1481,8 @@ void EvalContextStackPushPromiseFrame(EvalContext *ctx, const Promise *owner) { Rval final = EvaluateFinalRval(ctx, PromiseGetPolicy(owner), NULL, "this", cp->rval, false, owner); - if (final.type == RVAL_TYPE_SCALAR && !IsCf3VarString(RvalScalarValue(final))) + if (final.type == RVAL_TYPE_SCALAR && + ((EvalContextGetPass(ctx) == CF_DONEPASSES - 1) || !IsCf3VarString(RvalScalarValue(final)))) { EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_THIS, "with", RvalScalarValue(final), CF_DATA_TYPE_STRING, "source=promise_iteration/with"); } From 2dafed8caf46ab862a01b00a45a60dbd5be0149d Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 13 Oct 2021 14:36:52 +0200 Subject: [PATCH 261/333] Only report unexpanded vars in 'reports' in the last eval pass Otherwise things like this can happen: R: var3: $(var3) R: var3: val3 Ticket: CFE-3776 Changelog: Reports with unexpanded variable references are now attempted to be held off until the reference expands (cherry picked from commit 8f9407e52e0eb320355bdcd69ea568222b8fe4ff) --- libpromises/verify_reports.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/libpromises/verify_reports.c b/libpromises/verify_reports.c index 8e84e148c9..71e190b7ef 100644 --- a/libpromises/verify_reports.c +++ b/libpromises/verify_reports.c @@ -49,6 +49,18 @@ static void ReportToLog(const char *message); PromiseResult VerifyReportPromise(EvalContext *ctx, const Promise *pp) { + assert(pp != NULL); + + /* This check needs to happen *before* the lock is acquired otherwise the + * promise would be skipped in the next evaluation pass and a report with an + * unresolved variable reference would never be shown. */ + if ((EvalContextGetPass(ctx) < (CF_DONEPASSES - 1)) && IsCf3VarString(pp->promiser)) + { + /* Unresolved variable reference in the string to be reported and there + * is still a chance it will get resolved later. */ + return PROMISE_RESULT_SKIPPED; + } + CfLock thislock; char unique_name[CF_EXPANDSIZE]; From f2bd21f7af73aa94ca58d3f23b4c62c95616f66e Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 13 Oct 2021 16:22:48 +0200 Subject: [PATCH 262/333] Add tests for unresolved variable references in 'reports' For unresolved variable references in general and for such strings appearing in the expansion of the '$(with)' value. Ticket: CFE-3776 Changelog: None (cherry picked from commit f731c3edbf0ffa8f8f4212299d9a41c3290da209) --- .../14_reports/00_output/unresolved_vars.cf | 56 +++++++++++++++++++ .../00_output/unresolved_vars.cf.sub | 23 ++++++++ .../14_reports/00_output/unresolved_with.cf | 48 ++++++++++++++++ .../00_output/unresolved_with.cf.sub | 7 +++ 4 files changed, 134 insertions(+) create mode 100644 tests/acceptance/14_reports/00_output/unresolved_vars.cf create mode 100644 tests/acceptance/14_reports/00_output/unresolved_vars.cf.sub create mode 100644 tests/acceptance/14_reports/00_output/unresolved_with.cf create mode 100644 tests/acceptance/14_reports/00_output/unresolved_with.cf.sub diff --git a/tests/acceptance/14_reports/00_output/unresolved_vars.cf b/tests/acceptance/14_reports/00_output/unresolved_vars.cf new file mode 100644 index 0000000000..723e26f0ce --- /dev/null +++ b/tests/acceptance/14_reports/00_output/unresolved_vars.cf @@ -0,0 +1,56 @@ +####################################################### +# +# Test that reports promises handle unresolved var refs in a sane way +# +####################################################### + +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +bundle agent init +{ + vars: + "expected_lines" slist => { + "R: var1: val1", + "R: var2: val2", + "R: var3: val3", + "R: var4: val4", + "R: var5: $(const.dollar)(var5)" + }; + + "expected_output" string => join("$(const.n)", expected_lines); + + files: + "$(G.testfile).expected" + create => "true", + edit_line => insert_lines("$(expected_output)"); +} + +bundle agent test +{ + meta: + "description" -> { "CFE-3776" } + string => "Test that reports with unresolved variables are only emitted during the last pass of evaluation"; + + commands: + "$(sys.cf_agent) -Kf $(this.promise_filename).sub > $(G.testfile).actual" + contain => shell; +} + +body contain shell +{ + useshell => "true"; +} + + +bundle agent check +{ + methods: + "" usebundle => dcs_check_diff("$(G.testfile).actual", + "$(G.testfile).expected", + "$(this.promise_filename)"); +} diff --git a/tests/acceptance/14_reports/00_output/unresolved_vars.cf.sub b/tests/acceptance/14_reports/00_output/unresolved_vars.cf.sub new file mode 100644 index 0000000000..93e4c6a3f9 --- /dev/null +++ b/tests/acceptance/14_reports/00_output/unresolved_vars.cf.sub @@ -0,0 +1,23 @@ +bundle agent __main__ { + vars: + any:: + "var1" string => "val1"; + def_var2:: + "var2" string => "val2"; + def_var3:: + "var3" string => "val3"; + def_var4:: + "var4" string => "val4"; + + classes: + "def_var2" expression => isvariable("var1"); + "def_var3" expression => isvariable("var2"); + "def_var4" expression => isvariable("var3"); + + reports: + "var1: $(var1)"; + "var2: $(var2)"; + "var3: $(var3)"; + "var4: $(var4)"; + "var5: $(var5)"; +} diff --git a/tests/acceptance/14_reports/00_output/unresolved_with.cf b/tests/acceptance/14_reports/00_output/unresolved_with.cf new file mode 100644 index 0000000000..63867acb93 --- /dev/null +++ b/tests/acceptance/14_reports/00_output/unresolved_with.cf @@ -0,0 +1,48 @@ +####################################################### +# +# Test that reports promises handle unresolved var refs in '$(with)' +# +####################################################### + +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +bundle agent init +{ + vars: + "expected_output" string => 'R: y: { "@{x}" }'; + + files: + "$(G.testfile).expected" + create => "true", + edit_line => insert_lines("$(expected_output)"); +} + +bundle agent test +{ + meta: + "description" -> { "CFE-3776" } + string => "Test that with reports its content during the last pass even when that content has unresolved variables "; + + commands: + "$(sys.cf_agent) -Kf $(this.promise_filename).sub > $(G.testfile).actual" + contain => shell; +} + +body contain shell +{ + useshell => "true"; +} + + +bundle agent check +{ + methods: + "" usebundle => dcs_check_diff("$(G.testfile).actual", + "$(G.testfile).expected", + "$(this.promise_filename)"); +} diff --git a/tests/acceptance/14_reports/00_output/unresolved_with.cf.sub b/tests/acceptance/14_reports/00_output/unresolved_with.cf.sub new file mode 100644 index 0000000000..dd58989738 --- /dev/null +++ b/tests/acceptance/14_reports/00_output/unresolved_with.cf.sub @@ -0,0 +1,7 @@ +bundle agent __main__ { + vars: + "y" slist => { @{x} }; + + reports: + "y: $(with)" with => format("%S", y); +} From 9f8b8a980d8e0bea650ba7cb54d325cb0589f479 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Fri, 15 Oct 2021 11:28:06 -0500 Subject: [PATCH 263/333] Refactored array example to double as a test Ticket: CFE-3794 Changelog: None (cherry picked from commit b42c7f67f7c6ea71e056ad75dd73623dc1bbb9e5) --- examples/arrays.cf | 63 ++++++++++++++++++++-------------------------- 1 file changed, 27 insertions(+), 36 deletions(-) diff --git a/examples/arrays.cf b/examples/arrays.cf index 5dc86a49e6..cb589c47d1 100644 --- a/examples/arrays.cf +++ b/examples/arrays.cf @@ -1,49 +1,40 @@ -# Copyright 2019 Northern.tech AS - -# This file is part of Cfengine 3 - written and maintained by Northern.tech AS. - -# This program is free software; you can redistribute it and/or modify it -# under the terms of the GNU General Public License as published by the -# Free Software Foundation; version 3. - -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. - -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA - -# To the extent this program is licensed as part of the Enterprise -# versions of Cfengine, the applicable Commercial Open Source License -# (COSL) may apply to this file if you as a licensee so wish it. See -# included file COSL.txt. - - -body common control -{ - bundlesequence => { "array" }; -} - - +#+begin_src cfengine3 bundle common g { vars: - "array[1]" string => "one"; - "array[2]" string => "two"; + "array[key1]" string => "one"; + "array[key2]" string => "two"; } -bundle agent array +bundle agent __main__ { vars: - "localarray[1]" string => "one"; - "localarray[2]" string => "two"; + "thing[1][color]" string => "red"; + "thing[1][name]" string => "one"; + "thing[2][color]" string => "blue"; + "thing[2][name]" string => "two"; + + "_thing_idx" slist => sort( getindices( thing ), lex ); reports: - "Global $(g.array[1]) and $(localarray[2])"; -} + "Keys in default:g.array = $(with)" + with => join( ", ", getindices( "default:g.array" )); + + "Keys of default:main.thing[1] = $(with)" + with => join( ", ", getindices( "default:main.thing[1]" )); + "Thing $(thing[$(_thing_idx)][name]) is $(thing[$(_thing_idx)][color])"; +} +#+end_src +############################################################################### +#+begin_src example_output +#@ ``` +#@ R: Keys in default:g.array = key1, key2 +#@ R: Keys of default:main.thing[1] = color, name +#@ R: Thing one is red +#@ R: Thing two is blue +#@ ``` +#+end_src From 6b8685fdd27d5cef04814a86b923269f8f3c37d9 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Tue, 19 Oct 2021 07:54:32 -0500 Subject: [PATCH 264/333] Fixed arrays example test output Ticket: CFE-3794 Changelog: None (cherry picked from commit f4e45fc8c085bff5775994cde21e447b9ebcd416) --- examples/arrays.cf | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/examples/arrays.cf b/examples/arrays.cf index cb589c47d1..09ce51d9c1 100644 --- a/examples/arrays.cf +++ b/examples/arrays.cf @@ -10,21 +10,21 @@ bundle common g bundle agent __main__ { vars: - "thing[1][color]" string => "red"; "thing[1][name]" string => "one"; "thing[2][color]" string => "blue"; "thing[2][name]" string => "two"; - "_thing_idx" slist => sort( getindices( thing ), lex ); + "_thing_idx" + slist => sort( getindices( thing ), lex ); reports: "Keys in default:g.array = $(with)" - with => join( ", ", getindices( "default:g.array" )); + with => join( ", ", sort( getindices( "default:g.array" ), lex)); "Keys of default:main.thing[1] = $(with)" - with => join( ", ", getindices( "default:main.thing[1]" )); + with => join( ", ", sort( getindices( "default:main.thing[1]" ), lex)); "Thing $(thing[$(_thing_idx)][name]) is $(thing[$(_thing_idx)][color])"; } From 323965eb02cf64743ab3f423caadad23c1ffcbe4 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Tue, 26 Oct 2021 11:45:51 +0200 Subject: [PATCH 265/333] Removed deploy part of travis yaml I don't think this has ever worked or been used. (cherry picked from commit b7e3507c5c7c4f313ca5513c30b44059235aedcc) --- .travis.yml | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/.travis.yml b/.travis.yml index ada06b4a24..a3817db381 100644 --- a/.travis.yml +++ b/.travis.yml @@ -60,20 +60,4 @@ after_success: ./travis-scripts/after_success.sh before_deploy: ./travis-scripts/before_deploy.sh -# We deploy when setting tags only, which basically never happens -# on master branch. Cherry-pick this part and alter it properly into -# the release branches. -deploy: - skip_cleanup: true - provider: releases - # Remember to set it as private variable in Travis-CI settings - api_key: $GITHUB_RELEASE_TOKEN - # Does not work, I get the error that "true" is not a boolean - prerelease: $IS_PRERELEASE - file: "$DIST_TARBALL" - on: - tags: true - # Avoid upload from all the multiple parallel matrix jobs - condition: "$CF_VERSION = latest" - after_script: ./travis-scripts/after_script.sh From 179cca585240577705bff86fa79cdd900c363341 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 3 Nov 2021 16:30:29 +0100 Subject: [PATCH 266/333] Label /opt/cfengine as cfengine_var_lib_t Same as /var/cfengine so that our daemons and processes can access it. Ticket: ENT-7898 Changelog: None (cherry picked from commit 41a873d2d14d6e6c2d798f20475da3b0f7d0e778) --- misc/selinux/cfengine-enterprise.fc | 1 + 1 file changed, 1 insertion(+) diff --git a/misc/selinux/cfengine-enterprise.fc b/misc/selinux/cfengine-enterprise.fc index 15a7a73cfd..90d1705b3c 100644 --- a/misc/selinux/cfengine-enterprise.fc +++ b/misc/selinux/cfengine-enterprise.fc @@ -6,3 +6,4 @@ /var/cfengine/bin/pg.* -- gen_context(system_u:object_r:cfengine_postgres_exec_t,s0) /var/cfengine/httpd/bin/.* -- gen_context(system_u:object_r:cfengine_httpd_exec_t,s0) /var/cfengine/httpd/php/bin/.* -- gen_context(system_u:object_r:cfengine_httpd_exec_t,s0) +/opt/cfengine(/.*)? -- gen_context(system_u:object_r:cfengine_var_lib_t,s0) \ No newline at end of file From 6b50e7a2309ea7735522d29763676c7edad9f04a Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Thu, 11 Nov 2021 15:11:48 +0100 Subject: [PATCH 267/333] Added changelog for 3.15.5 --- ChangeLog | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/ChangeLog b/ChangeLog index 991fd89ae3..2a8be4c506 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,22 @@ +3.15.5: + - Add Wants cf-postgres.service to cf-apache.service in systemd configuration + (ENT-5125) + - Added example illustrating data_sysctlvalues() behavior (CFE-3783) + - CFEngine processes now reuse log facility from + previous run for early logging before policy is loaded (ENT-6955) + - Fix dbm_quick.c DBPrivRead() argument type (CFE-3737) + - Fix dbm_tokyocab.c DBPrivRead() argument type (CFE-3737) + - Reports with unexpanded variable references are now + attempted to be held off until the reference expands + (CFE-3776) + - Revert "Added cf-postgres requirement to cf-apache and cf-hub systemd units" + (ENT-5125) + - Skip update policy run with --skip-bootstrap-policy-run + (ENT-7500, ENT-7511) + - Use ts_key file for measurement names in cf-check dump (ENT-7452) + - Value of '$(with)' is now expanded even if it contains unresolved variable references + (CFE-3776) + 3.15.4: - cf-runagent now exits with a code reflecting remote agent run status(es) (CFE-3594) From 6586131aaf5d585605ddfcceb3ba7b7897aabef0 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Thu, 11 Nov 2021 12:40:48 -0600 Subject: [PATCH 268/333] Set apache umask to 0177 Ticket: ENT-7948 Changelog: Title (cherry picked from commit 42358dd8953675469fde8be3ab3c5fcd849e077a) --- misc/systemd/cf-apache.service.in | 1 + 1 file changed, 1 insertion(+) diff --git a/misc/systemd/cf-apache.service.in b/misc/systemd/cf-apache.service.in index 0ca6e5405f..009fe9aa46 100644 --- a/misc/systemd/cf-apache.service.in +++ b/misc/systemd/cf-apache.service.in @@ -13,6 +13,7 @@ ExecStop=@workdir@/httpd/bin/apachectl stop PIDFile=@workdir@/httpd/logs/httpd.pid Restart=always RestartSec=10 +UMask=0177 [Install] WantedBy=multi-user.target From 6395cc078a2a8d6c7ba022f8631cfb94dea7d3e4 Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Mon, 15 Nov 2021 16:09:13 +0100 Subject: [PATCH 269/333] Cleaned up 3.15.5 changelog --- ChangeLog | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/ChangeLog b/ChangeLog index 2a8be4c506..b9d502e791 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,20 +1,15 @@ 3.15.5: - - Add Wants cf-postgres.service to cf-apache.service in systemd configuration + - Added "Wants cf-postgres.service" to cf-apache.service in systemd configuration (ENT-5125) - - Added example illustrating data_sysctlvalues() behavior (CFE-3783) - CFEngine processes now reuse log facility from previous run for early logging before policy is loaded (ENT-6955) - - Fix dbm_quick.c DBPrivRead() argument type (CFE-3737) - - Fix dbm_tokyocab.c DBPrivRead() argument type (CFE-3737) - Reports with unexpanded variable references are now - attempted to be held off until the reference expands + attempted to be held off until the reference expands (CFE-3776) - - Revert "Added cf-postgres requirement to cf-apache and cf-hub systemd units" - (ENT-5125) - - Skip update policy run with --skip-bootstrap-policy-run + - The option --skip-bootstrap-policy-run now skips the update policy run (ENT-7500, ENT-7511) - - Use ts_key file for measurement names in cf-check dump (ENT-7452) - - Value of '$(with)' is now expanded even if it contains unresolved variable references + - cf-check dump now shows measurement names (ENT-7452) + - The value of '$(with)' is now expanded even if it contains unresolved variable references (CFE-3776) 3.15.4: From 38192a56759c2ace236af867e9cb272e53f21921 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Tue, 16 Nov 2021 08:47:08 -0600 Subject: [PATCH 270/333] Moved httpd.pid to root of httpd workdir Ticket: ENT-7966 Changelog: Title (cherry picked from commit 102f56d345e45794621818cde2674b5b6f4831d2) --- misc/systemd/cf-apache.service.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/misc/systemd/cf-apache.service.in b/misc/systemd/cf-apache.service.in index 0ca6e5405f..b114fb02f2 100644 --- a/misc/systemd/cf-apache.service.in +++ b/misc/systemd/cf-apache.service.in @@ -10,7 +10,7 @@ PartOf=cfengine3.service Type=forking ExecStart=@workdir@/httpd/bin/apachectl start ExecStop=@workdir@/httpd/bin/apachectl stop -PIDFile=@workdir@/httpd/logs/httpd.pid +PIDFile=@workdir@/httpd/httpd.pid Restart=always RestartSec=10 From 7680f9603b3a9d74a3f2873ef24b9b8aeca39d4c Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 11 Nov 2021 16:52:17 +0100 Subject: [PATCH 271/333] Fix '-N/--negate' preventing persistent classes from being defined Ticket: ENT-5886 Changelog: '-N/--negate now prevents persistent classes from being defined' (cherry picked from commit 3e3a744743adc6e554013699aad0839a1cde461c) Conflicts: libpromises/eval_context.c --- ChangeLog | 1 + libpromises/eval_context.c | 36 ++++++++++--- libpromises/eval_context.h | 7 +++ libpromises/generic_agent.c | 9 ++++ .../02_classes/01_basic/persistent_negate.cf | 50 +++++++++++++++++++ .../01_basic/persistent_negate.cf.sub | 12 +++++ .../01_basic/persistent_negate.cf.sub2 | 20 ++++++++ 7 files changed, 128 insertions(+), 7 deletions(-) create mode 100644 tests/acceptance/02_classes/01_basic/persistent_negate.cf create mode 100644 tests/acceptance/02_classes/01_basic/persistent_negate.cf.sub create mode 100644 tests/acceptance/02_classes/01_basic/persistent_negate.cf.sub2 diff --git a/ChangeLog b/ChangeLog index b9d502e791..d7c356da4f 100644 --- a/ChangeLog +++ b/ChangeLog @@ -11,6 +11,7 @@ - cf-check dump now shows measurement names (ENT-7452) - The value of '$(with)' is now expanded even if it contains unresolved variable references (CFE-3776) + - '-N/--negate' now prevents persistent classes from being defined (ENT-5886) 3.15.4: - cf-runagent now exits with a code reflecting remote agent run status(es) diff --git a/libpromises/eval_context.c b/libpromises/eval_context.c index ccbc1732f0..6fd8df7ea2 100644 --- a/libpromises/eval_context.c +++ b/libpromises/eval_context.c @@ -176,6 +176,9 @@ struct EvalContext_ /* List if all classes set during policy evaluation */ StringSet *all_classes; + /* Negated classes (persistent classes that should not be defined). */ + StringSet *negated_classes; + /* These following two fields are needed for remote variable injection * detection (CFE-1915) */ /* Names of all bundles */ @@ -730,6 +733,8 @@ void EvalContextHeapPersistentRemove(const char *context) void EvalContextHeapPersistentLoadAll(EvalContext *ctx) { + assert(ctx != NULL); + time_t now = time(NULL); Log(LOG_LEVEL_VERBOSE, "Loading persistent classes"); @@ -782,17 +787,25 @@ void EvalContextHeapPersistentLoadAll(EvalContext *ctx) { Log(LOG_LEVEL_VERBOSE, "Persistent class '%s' for %jd more minutes", key, (intmax_t) ((info.expires - now) / 60)); - Log(LOG_LEVEL_DEBUG, "Adding persistent class '%s'", key); + if ((ctx->negated_classes != NULL) && StringSetContains(ctx->negated_classes, key)) + { + Log(LOG_LEVEL_VERBOSE, + "Not adding persistent class '%s' due to match in -N/--negate", key); + } + else + { + Log(LOG_LEVEL_DEBUG, "Adding persistent class '%s'", key); - ClassRef ref = ClassRefParse(key); - EvalContextClassPut(ctx, ref.ns, ref.name, true, CONTEXT_SCOPE_NAMESPACE, tags); + ClassRef ref = ClassRefParse(key); + EvalContextClassPut(ctx, ref.ns, ref.name, true, CONTEXT_SCOPE_NAMESPACE, tags); - StringSet *tag_set = EvalContextClassTags(ctx, ref.ns, ref.name); - assert(tag_set); + StringSet *tag_set = EvalContextClassTags(ctx, ref.ns, ref.name); + assert(tag_set); - StringSetAdd(tag_set, xstrdup("source=persistent")); + StringSetAdd(tag_set, xstrdup("source=persistent")); - ClassRefDestroy(ref); + ClassRefDestroy(ref); + } } } @@ -800,6 +813,13 @@ void EvalContextHeapPersistentLoadAll(EvalContext *ctx) CloseDB(dbp); } +void EvalContextSetNegatedClasses(EvalContext *ctx, StringSet *negated_classes) +{ + assert(ctx != NULL); + ctx->negated_classes = negated_classes; +} + + bool BundleAbort(EvalContext *ctx) { assert(ctx != NULL); @@ -1030,6 +1050,7 @@ EvalContext *EvalContextNew(void) ctx->package_promise_context = PackagePromiseConfigNew(); ctx->all_classes = NULL; + ctx->negated_classes = NULL; ctx->bundle_names = StringSetNew(); ctx->remote_var_promises = NULL; @@ -1074,6 +1095,7 @@ void EvalContextDestroy(EvalContext *ctx) FreePackagePromiseContext(ctx->package_promise_context); StringSetDestroy(ctx->all_classes); + StringSetDestroy(ctx->negated_classes); StringSetDestroy(ctx->bundle_names); if (ctx->remote_var_promises != NULL) { diff --git a/libpromises/eval_context.h b/libpromises/eval_context.h index 638b0a1549..b1846720d2 100644 --- a/libpromises/eval_context.h +++ b/libpromises/eval_context.h @@ -121,6 +121,13 @@ void EvalContextHeapPersistentSave(EvalContext *ctx, const char *name, unsigned void EvalContextHeapPersistentRemove(const char *context); void EvalContextHeapPersistentLoadAll(EvalContext *ctx); +/** + * Sets negated classes (persistent classes that should not be defined). + * + * @note Takes ownership of #negated_classes + */ +void EvalContextSetNegatedClasses(EvalContext *ctx, StringSet *negated_classes); + bool EvalContextClassPutSoft(EvalContext *ctx, const char *name, ContextScope scope, const char *tags); bool EvalContextClassPutHard(EvalContext *ctx, const char *name, const char *tags); Class *EvalContextClassGet(const EvalContext *ctx, const char *ns, const char *name); diff --git a/libpromises/generic_agent.c b/libpromises/generic_agent.c index 774fe04826..88554b7eee 100644 --- a/libpromises/generic_agent.c +++ b/libpromises/generic_agent.c @@ -2070,6 +2070,8 @@ void GenericAgentConfigDestroy(GenericAgentConfig *config) void GenericAgentConfigApply(EvalContext *ctx, const GenericAgentConfig *config) { + assert(config != NULL); + if (config->heap_soft) { StringSetIterator it = StringSetIteratorInit(config->heap_soft); @@ -2086,6 +2088,13 @@ void GenericAgentConfigApply(EvalContext *ctx, const GenericAgentConfig *config) } } + if (config->heap_negated != NULL) + { + /* Takes ownership of heap_negated. */ + EvalContextSetNegatedClasses(ctx, config->heap_negated); + ((GenericAgentConfig *)config)->heap_negated = NULL; + } + switch (LogGetGlobalLevel()) { case LOG_LEVEL_DEBUG: diff --git a/tests/acceptance/02_classes/01_basic/persistent_negate.cf b/tests/acceptance/02_classes/01_basic/persistent_negate.cf new file mode 100644 index 0000000000..5dc9e54a48 --- /dev/null +++ b/tests/acceptance/02_classes/01_basic/persistent_negate.cf @@ -0,0 +1,50 @@ +####################################################### +# +# ENT-5886 -- -N/--negate should prevent persistent classes from being defined +# +####################################################### + +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +bundle agent init +{ + vars: + "dflags" string => ifelse("EXTRA", "-DDEBUG,EXTRA", "-DDEBUG"); +} + +bundle agent test +{ + meta: + "description" -> {"ENT-5886"} + string => "-N/--negate should prevent persistent classes from being defined"; + + commands: + "$(sys.cf_agent) -K $(init.dflags) -f $(this.promise_filename).sub" + classes => test_always("done_persisting"); +} + +body classes test_always(x) +{ + promise_repaired => { "$(x)" }; + promise_kept => { "$(x)" }; + repair_failed => { "$(x)" }; + repair_denied => { "$(x)" }; + repair_timeout => { "$(x)" }; +} + +bundle agent check +{ + vars: + done_persisting:: + "subout" string => execresult("$(sys.cf_agent) -N test_class -K $(init.dflags) -f $(this.promise_filename).sub2", "noshell"); + + methods: + "" usebundle => dcs_check_strcmp($(subout), "R: Pass", + $(this.promise_filename), + "no"); +} diff --git a/tests/acceptance/02_classes/01_basic/persistent_negate.cf.sub b/tests/acceptance/02_classes/01_basic/persistent_negate.cf.sub new file mode 100644 index 0000000000..58606f8f40 --- /dev/null +++ b/tests/acceptance/02_classes/01_basic/persistent_negate.cf.sub @@ -0,0 +1,12 @@ +body common control +{ + bundlesequence => { run }; +} + +bundle common run +{ + classes: + "test_class" + expression => "any", + persistence => "120"; # 2 hours. +} diff --git a/tests/acceptance/02_classes/01_basic/persistent_negate.cf.sub2 b/tests/acceptance/02_classes/01_basic/persistent_negate.cf.sub2 new file mode 100644 index 0000000000..03931d5fc8 --- /dev/null +++ b/tests/acceptance/02_classes/01_basic/persistent_negate.cf.sub2 @@ -0,0 +1,20 @@ +body common control +{ + bundlesequence => { run }; +} + +bundle agent run +{ + classes: + "ok" expression => "!test_class"; + + reports: + ok:: + "Pass"; + + !ok.DEBUG:: + "test_class defined"; + + !ok:: + "FAIL"; +} From 6bf1daa3a79cde03d7a403e20a219e5590d1c5ee Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 18 Nov 2021 16:52:43 +0100 Subject: [PATCH 272/333] Make sure everything in /opt/cfengine is properly SELinux-labeled Using '--' means only regular files should get the label. (cherry picked from commit 4da74542b4e13ad90c7326e0a24740d2f84d4164) --- misc/selinux/cfengine-enterprise.fc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/misc/selinux/cfengine-enterprise.fc b/misc/selinux/cfengine-enterprise.fc index 90d1705b3c..d50bd4c247 100644 --- a/misc/selinux/cfengine-enterprise.fc +++ b/misc/selinux/cfengine-enterprise.fc @@ -6,4 +6,4 @@ /var/cfengine/bin/pg.* -- gen_context(system_u:object_r:cfengine_postgres_exec_t,s0) /var/cfengine/httpd/bin/.* -- gen_context(system_u:object_r:cfengine_httpd_exec_t,s0) /var/cfengine/httpd/php/bin/.* -- gen_context(system_u:object_r:cfengine_httpd_exec_t,s0) -/opt/cfengine(/.*)? -- gen_context(system_u:object_r:cfengine_var_lib_t,s0) \ No newline at end of file +/opt/cfengine(/.*)? gen_context(system_u:object_r:cfengine_var_lib_t,s0) \ No newline at end of file From 8a1095af39d5be87e9383c4aeba5c74021b5b758 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 19 Nov 2021 10:54:28 +0100 Subject: [PATCH 273/333] Allow our apache process to work with files in /tmp Apparently, PHP needs to create some temporary dirs and files there and httpd needs to mmap() some bits from there. Ticket: ENT-8029 Changelog: None (cherry picked from commit 32c9123eedf41dfd2035d0b3996bcfc31279b679) --- misc/selinux/cfengine-enterprise.te | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 6668a27205..1942fe74b4 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -537,6 +537,12 @@ allow cfengine_httpd_t sssd_var_lib_t:dir search; allow cfengine_httpd_t sssd_var_lib_t:sock_file write; allow cfengine_httpd_t syslogd_var_run_t:dir search; allow cfengine_httpd_t tmp_t:sock_file write; +allow cfengine_httpd_t tmp_t:file { create setattr unlink write }; +allow cfengine_httpd_t tmp_t:dir { add_name remove_name write }; +allow cfengine_httpd_t var_t:dir read; + +# apparently, httpd creates some temporary bits in /tmp that it needs to mmap() +allow cfengine_httpd_t tmp_t:file map; # Bidirectional DBus communication between httpd and systemd allow cfengine_httpd_t system_dbusd_t:dbus send_msg; From b18e03feb81fd88a12d21cd86d3a98b76d3583cd Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 19 Nov 2021 10:57:35 +0100 Subject: [PATCH 274/333] Add SELinux rules for cf-hub running scheduled reports It needs to be able to execute PHP and handle files under /var/cfengine in some more ways. Ticket: ENT-8029 Changelog: Non (cherry picked from commit 581d103572f5e529860af41295698615ed7dc431) --- misc/selinux/cfengine-enterprise.te | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 1942fe74b4..dd8690a83b 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -359,10 +359,14 @@ allow cfengine_hub_t unreserved_port_t:tcp_socket name_connect; allow cfengine_hub_t cfengine_log_t:dir getattr; allow cfengine_hub_t cfengine_var_lib_t:dir { add_name getattr open read search write remove_name }; -allow cfengine_hub_t cfengine_var_lib_t:file { create ioctl lock write unlink }; +allow cfengine_hub_t cfengine_var_lib_t:file { create ioctl lock write unlink append setattr }; allow cfengine_hub_t cfengine_var_lib_t:lnk_file { getattr read }; allow cfengine_hub_t cfengine_var_lib_t:sock_file { create unlink }; +# cf-hub executes PHP in case of scheduled reports +allow cfengine_hub_t cfengine_httpd_exec_t:file map; +allow cfengine_hub_t cfengine_httpd_exec_t:file { execute execute_no_trans getattr open read }; + allow cfengine_hub_t bin_t:file map; allow cfengine_hub_t bin_t:file { execute execute_no_trans }; allow cfengine_hub_t cert_t:dir search; From b742de1628b7b72c51407762265b876a561ed739 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 19 Nov 2021 11:05:42 +0100 Subject: [PATCH 275/333] Add SELinux rules for cf-hub to be able to send emails Ticket: ENT-8029 Changelog: Non (cherry picked from commit afb6c01a84390fc1803bc7cd661bc22de2155297) --- misc/selinux/cfengine-enterprise.te | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index dd8690a83b..5ff079d266 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -78,6 +78,12 @@ require { type tty_device_t; type user_devpts_t; type sysctl_t; + type postfix_etc_t; + type postfix_master_t; + type postfix_postdrop_exec_t; + type postfix_public_t; + type postfix_spool_t; + type sendmail_exec_t; class tcp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown name_connect accept listen name_bind node_bind }; class udp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown node_bind }; class sock_file { create write setattr unlink }; @@ -367,6 +373,19 @@ allow cfengine_hub_t cfengine_var_lib_t:sock_file { create unlink }; allow cfengine_hub_t cfengine_httpd_exec_t:file map; allow cfengine_hub_t cfengine_httpd_exec_t:file { execute execute_no_trans getattr open read }; +# sending scheduled reports via email +allow cfengine_hub_t postfix_etc_t:dir { getattr open read search }; +allow cfengine_hub_t postfix_etc_t:file { getattr open read }; +allow cfengine_hub_t postfix_master_t:unix_stream_socket connectto; +allow cfengine_hub_t postfix_postdrop_exec_t:file map; +allow cfengine_hub_t postfix_postdrop_exec_t:file { execute execute_no_trans open read }; +allow cfengine_hub_t postfix_public_t:dir search; +allow cfengine_hub_t postfix_public_t:sock_file { getattr write }; +allow cfengine_hub_t postfix_spool_t:dir { add_name remove_name search write }; +allow cfengine_hub_t postfix_spool_t:file { create getattr open read rename setattr write }; +allow cfengine_hub_t sendmail_exec_t:file map; +allow cfengine_hub_t sendmail_exec_t:file { execute execute_no_trans open read }; + allow cfengine_hub_t bin_t:file map; allow cfengine_hub_t bin_t:file { execute execute_no_trans }; allow cfengine_hub_t cert_t:dir search; @@ -391,10 +410,11 @@ allow cfengine_hub_t ping_exec_t:file getattr; allow cfengine_hub_t proc_net_t:file { getattr open read }; allow cfengine_hub_t proc_t:dir read; allow cfengine_hub_t rpm_exec_t:file getattr; -allow cfengine_hub_t self:capability dac_override; +allow cfengine_hub_t self:capability { dac_override chown dac_read_search }; +allow cfengine_hub_t self:process { execmem setrlimit }; allow cfengine_hub_t self:tcp_socket { connect create getopt setopt read write }; allow cfengine_hub_t self:udp_socket { connect create getattr ioctl setopt read write }; -allow cfengine_hub_t self:netlink_route_socket { create getopt setopt bind getattr }; +allow cfengine_hub_t self:netlink_route_socket { create getopt setopt bind getattr nlmsg_read }; allow cfengine_hub_t self:unix_dgram_socket { create connect read write }; allow cfengine_hub_t semanage_exec_t:file getattr; allow cfengine_hub_t shadow_t:file getattr; From 7bcaa59f3cf5331ba7eea14a3d81bf9530a74093 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 19 Nov 2021 11:06:08 +0100 Subject: [PATCH 276/333] Fix the path to httpd binaries in a comment in the SELinux rules Ticket: ENT-8029 Changelog: Non (cherry picked from commit 6999616c33aa01dff3bc1e3b79852aeb35d48ff2) --- misc/selinux/cfengine-enterprise.te | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 5ff079d266..40c2268b56 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -508,8 +508,8 @@ type cfengine_httpd_t; typeattribute cfengine_httpd_t domain; role system_r types cfengine_httpd_t; -# /var/cfengine/bin/cf-httpd has the 'cfengine_httpd_exec_t' context which is an -# entrypoint for the 'cfengine_httpd_t' domain +# /var/cfengine/httpd/bin/* files have the 'cfengine_httpd_exec_t' context which +# is an entrypoint for the 'cfengine_httpd_t' domain type cfengine_httpd_exec_t; typeattribute cfengine_httpd_exec_t entry_type; typeattribute cfengine_httpd_exec_t exec_type; From a65a9e6385013f537ead817d74168d33e3f4a4e0 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 19 Nov 2021 13:45:07 +0100 Subject: [PATCH 277/333] SELinux: allow PostgreSQL to check /etc/{passwd,group} It needs to be able to check if the 'cfpostgres' user/group exists when it starts. (cherry picked from commit d4209088f5aefd94e85e915087549bc53e46a800) --- misc/selinux/cfengine-enterprise.te | 3 +++ 1 file changed, 3 insertions(+) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 40c2268b56..636c9df948 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -494,6 +494,9 @@ allow cfengine_postgres_t tmpfs_t:file { create open read write map unlink }; allow cfengine_postgres_t tmpfs_t:filesystem getattr; allow cfengine_postgres_t var_log_t:file { append open }; +# so that PostgreSQL can check if cfpostgres user/group exists +allow cfengine_postgres_t passwd_file_t:file read; + # Needed for systemd to be able to check PostgreSQL's PID file allow init_t cfengine_var_lib_t:dir { read remove_name write }; allow init_t cfengine_var_lib_t:file { getattr open read unlink }; From 0459395b2846bfe664ed1124701dc82e56c71236 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 19 Nov 2021 14:01:29 +0100 Subject: [PATCH 278/333] Add SELinux rules for httpd/php to be able to send emails Ticket: ENT-8029 Changelog: None (cherry picked from commit 68a823a49a541df85d5151c30ea808c1a7931c0c) Conflicts: misc/selinux/cfengine-enterprise.te --- misc/selinux/cfengine-enterprise.te | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 636c9df948..f91863fae2 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -86,7 +86,7 @@ require { type sendmail_exec_t; class tcp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown name_connect accept listen name_bind node_bind }; class udp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown node_bind }; - class sock_file { create write setattr unlink }; + class sock_file { create write setattr unlink getattr }; class rawip_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; class netlink_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; class packet_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; @@ -147,7 +147,7 @@ require { class ib_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; class mpls_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; class process { setrlimit transition dyntransition execstack execheap execmem signull siginh }; - class file { execute execute_no_trans getattr ioctl map open read unlink write entrypoint lock link rename append setattr create relabelfrom relabelto }; + class file { execute execute_no_trans getattr ioctl map open read unlink write entrypoint lock link rename append setattr create relabelfrom relabelto watch watch_reads }; class fifo_file { create open getattr setattr read write append rename link unlink ioctl lock relabelfrom relabelto }; class dir { getattr read search open write add_name remove_name lock ioctl create }; class filesystem getattr; @@ -161,6 +161,7 @@ require { class memprotect mmap_zero; class peer recv; class chr_file { getattr }; + class lockdown { confidentiality integrity }; } @@ -540,6 +541,17 @@ allow cfengine_httpd_t cfengine_var_lib_t:file { append create execute getattr i allow cfengine_httpd_t cfengine_var_lib_t:dir { add_name getattr open read remove_name search write create }; allow cfengine_httpd_t cfengine_var_lib_t:lnk_file read; +# sending reports via email +allow cfengine_httpd_t postfix_etc_t:dir { getattr open read search }; +allow cfengine_httpd_t postfix_etc_t:file { getattr open read }; +allow cfengine_httpd_t postfix_postdrop_exec_t:file { execute execute_no_trans map open read }; +allow cfengine_httpd_t postfix_public_t:dir search; +allow cfengine_httpd_t postfix_public_t:sock_file { getattr write }; +allow cfengine_httpd_t postfix_spool_t:dir { add_name remove_name search write }; +allow cfengine_httpd_t postfix_spool_t:file { create getattr open read rename setattr write }; +allow cfengine_httpd_t self:process setrlimit; +allow cfengine_httpd_t sendmail_exec_t:file { execute execute_no_trans getattr map open read }; + allow cfengine_httpd_t devlog_t:lnk_file read; allow cfengine_httpd_t devlog_t:sock_file write; allow cfengine_httpd_t http_port_t:tcp_socket { name_bind name_connect }; From af02e3f937fa8d446ccb8ce305476fa010aca49d Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 19 Nov 2021 15:12:16 +0100 Subject: [PATCH 279/333] Add a special SELinux type and domain for Mission Portal action scripts The notification scripts that can be hooked up to alerts are supposed to be able to do basically anything on the system. Thus we need to allow that, but we shouldn't enable our httpd and PHP do anything on the system. Ticket: ENT-8029 Changelog: None (cherry picked from commit ef652c0d21c04fc5792e34075a52849d76497010) Conflicts: misc/selinux/cfengine-enterprise.te --- misc/selinux/cfengine-enterprise.fc | 3 ++- misc/selinux/cfengine-enterprise.te | 38 ++++++++++++++++++++++++++++- 2 files changed, 39 insertions(+), 2 deletions(-) diff --git a/misc/selinux/cfengine-enterprise.fc b/misc/selinux/cfengine-enterprise.fc index d50bd4c247..6108af4d8a 100644 --- a/misc/selinux/cfengine-enterprise.fc +++ b/misc/selinux/cfengine-enterprise.fc @@ -6,4 +6,5 @@ /var/cfengine/bin/pg.* -- gen_context(system_u:object_r:cfengine_postgres_exec_t,s0) /var/cfengine/httpd/bin/.* -- gen_context(system_u:object_r:cfengine_httpd_exec_t,s0) /var/cfengine/httpd/php/bin/.* -- gen_context(system_u:object_r:cfengine_httpd_exec_t,s0) -/opt/cfengine(/.*)? gen_context(system_u:object_r:cfengine_var_lib_t,s0) \ No newline at end of file +/opt/cfengine(/.*)? gen_context(system_u:object_r:cfengine_var_lib_t,s0) +/opt/cfengine/notification_scripts(/.*)? gen_context(system_u:object_r:cfengine_action_script_exec_t,s0) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index f91863fae2..25fb224ad8 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -541,6 +541,10 @@ allow cfengine_httpd_t cfengine_var_lib_t:file { append create execute getattr i allow cfengine_httpd_t cfengine_var_lib_t:dir { add_name getattr open read remove_name search write create }; allow cfengine_httpd_t cfengine_var_lib_t:lnk_file read; +# allow httpd/php to upload notification/alert scripts +allow cfengine_httpd_t cfengine_action_script_exec_t:dir { add_name getattr search write remove_name }; +allow cfengine_httpd_t cfengine_action_script_exec_t:file { create write setattr unlink }; + # sending reports via email allow cfengine_httpd_t postfix_etc_t:dir { getattr open read search }; allow cfengine_httpd_t postfix_etc_t:file { getattr open read }; @@ -576,7 +580,7 @@ allow cfengine_httpd_t sssd_var_lib_t:dir search; allow cfengine_httpd_t sssd_var_lib_t:sock_file write; allow cfengine_httpd_t syslogd_var_run_t:dir search; allow cfengine_httpd_t tmp_t:sock_file write; -allow cfengine_httpd_t tmp_t:file { create setattr unlink write }; +allow cfengine_httpd_t tmp_t:file { create setattr unlink write rename }; allow cfengine_httpd_t tmp_t:dir { add_name remove_name write }; allow cfengine_httpd_t var_t:dir read; @@ -594,3 +598,35 @@ allow init_t cfengine_httpd_t:dbus send_msg; allow cfengine_httpd_t passwd_file_t:file { getattr open read }; allow cfengine_httpd_t shell_exec_t:file map; allow cfengine_httpd_t shell_exec_t:file { execute execute_no_trans }; + + +#============= cfengine_action_script_t ============== +# A special type and domain for action (notification/alert) scripts executed by +# Mission Portal. They can do anything, so they need to run in an unconstrained +# domain. At the same time we don't want our Apache and PHP to do anything so +# these scripts cannot just run in the http_t domain. + +type cfengine_action_script_t; +typeattribute cfengine_action_script_t domain; +role system_r types cfengine_action_script_t; + +# this is a macro invocation, the file has to be processed with +# make -f /usr/share/selinux/devel/Makefile +unconfined_domain(cfengine_action_script_t) + +# /opt/cfengine/notification_scripts/* files have the +# 'cfengine_action_script_exec_t' context which is an entrypoint for the +# 'cfengine_action_script_t' domain +type cfengine_action_script_exec_t; +typeattribute cfengine_action_script_exec_t entry_type; +typeattribute cfengine_action_script_exec_t exec_type; +typeattribute cfengine_action_script_exec_t file_type, non_security_file_type, non_auth_file_type; +role object_r types cfengine_action_script_exec_t; + +type_transition init_t cfengine_action_script_exec_t:process cfengine_action_script_t; +allow cfengine_httpd_t cfengine_action_script_t:process transition; +allow cfengine_httpd_t cfengine_action_script_exec_t:file { execute execute_no_trans getattr open read }; +allow cfengine_httpd_t cfengine_action_script_t:process siginh; + +allow cfengine_action_script_t cfengine_action_script_exec_t:file entrypoint; +allow cfengine_action_script_t cfengine_action_script_exec_t:file { ioctl read getattr lock map execute open }; From f9ca79c20a163b0987dc059be5096e7d867c62fe Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Tue, 30 Nov 2021 01:35:08 +0100 Subject: [PATCH 280/333] Bumped .CFVERSION number to 3.15.6 Signed-off-by: Ole Herman Schumacher Elgesem --- .CFVERSION | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.CFVERSION b/.CFVERSION index 9658bd085c..9bf718c0a8 100644 --- a/.CFVERSION +++ b/.CFVERSION @@ -1 +1 @@ -3.15.5 +3.15.6 From 4a8984512f6276073c4cf92a3e413eeef5c9a624 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 1 Dec 2021 09:34:18 +0100 Subject: [PATCH 281/333] SELinux: Allow access to /opt/cfengine for commands over SSH Needed for Federated Reporting. Ticket: ENT-7493 Changelog: Federated Reporting is now supported on RHEL 8 hubs (cherry picked from commit c3b22990a05357a3b03fad6af2172f534d7a5ca1) --- misc/selinux/cfengine-enterprise.te | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 25fb224ad8..bfd3823e36 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -84,6 +84,7 @@ require { type postfix_public_t; type postfix_spool_t; type sendmail_exec_t; + type sshd_t; class tcp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown name_connect accept listen name_bind node_bind }; class udp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown node_bind }; class sock_file { create write setattr unlink getattr }; @@ -630,3 +631,7 @@ allow cfengine_httpd_t cfengine_action_script_t:process siginh; allow cfengine_action_script_t cfengine_action_script_exec_t:file entrypoint; allow cfengine_action_script_t cfengine_action_script_exec_t:file { ioctl read getattr lock map execute open }; + +#============= special rules for Federated Reporting ============= +# sshd needs access to files in /opt/cfengine +allow sshd_t cfengine_var_lib_t:file { getattr open read }; From cab6f559df4abc67f723ba82f1c33ce23e3c6b55 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 1 Dec 2021 14:39:28 +0100 Subject: [PATCH 282/333] Allow cf-agent to transition into the SELinux domain of RPM scriptlets So that they can be executed when cf-agent executes yum (by means of the yum package module). Ticket: ENT-7493 Changelog: cf-agent can now properly install packages on RHEL 8 (cherry picked from commit 472bbce8f43e8bce2ba5661c369f674eac701684) --- misc/selinux/cfengine-enterprise.te | 3 +++ 1 file changed, 3 insertions(+) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index bfd3823e36..905c8aaf20 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -85,6 +85,7 @@ require { type postfix_spool_t; type sendmail_exec_t; type sshd_t; + type rpm_script_t; class tcp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown name_connect accept listen name_bind node_bind }; class udp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown node_bind }; class sock_file { create write setattr unlink getattr }; @@ -187,6 +188,8 @@ role object_r types cfengine_agent_exec_t; allow cfengine_agent_t cfengine_agent_exec_t:file entrypoint; allow cfengine_agent_t cfengine_agent_exec_t:file { ioctl read getattr lock map execute open }; +# cf-agent needs to be able to transition into the domain of RPM scriplets +allow cfengine_agent_t rpm_script_t:process transition; #============= cfengine_execd_t ============== # allow cf-execd to run cf-agent and make sure the forked process run in the From b232648ca1e4de278a9d68f55028dc0410561abb Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Fri, 10 Dec 2021 11:24:50 -0600 Subject: [PATCH 283/333] Remove inaccurate build badges Travis badges are for any PRs on the branches so not what we want. Jenkins badges aren't accurate either due to lack of known issue exceptions. Ticket: ENT-7870 Changelog: none (cherry picked from commit cdfb560ffafe3211c685a2b6ec99c2812b69ef52) --- README.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/README.md b/README.md index 2a0207bde8..cef8aec416 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,4 @@ [![Gitter chat](https://badges.gitter.im/cfengine/core.png)](https://gitter.im/cfengine/core) - -| Version | [Core](https://github.com/cfengine/core) | [MPF](https://github.com/cfengine/masterfiles) | -|------------|--------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------| -| master | [![Core Build Status](https://travis-ci.org/cfengine/core.svg?branch=master)](https://travis-ci.org/cfengine/core) | [![MPF Build Status](https://travis-ci.org/cfengine/masterfiles.svg?branch=master)](https://travis-ci.org/cfengine/masterfiles) | -| 3.15.x LTS | [![Core Build Status](https://travis-ci.org/cfengine/core.svg?branch=3.15.x)](https://travis-ci.org/cfengine/core) | [![MPF Build Status](https://travis-ci.org/cfengine/masterfiles.svg?branch=3.15.x)](https://travis-ci.org/cfengine/masterfiles) | -| 3.12.x LTS | [![Core Build Status](https://travis-ci.org/cfengine/core.svg?branch=3.12.x)](https://travis-ci.org/cfengine/core) | [![MPF Build Status](https://travis-ci.org/cfengine/masterfiles.svg?branch=3.12.x)](https://travis-ci.org/cfengine/masterfiles) | -| 3.10.x LTS | [![Core Build Status](https://travis-ci.org/cfengine/core.svg?branch=3.10.x)](https://travis-ci.org/cfengine/core) | [![MPF Build Status](https://travis-ci.org/cfengine/masterfiles.svg?branch=3.10.x)](https://travis-ci.org/cfengine/masterfiles) | - [![Language grade: C](https://img.shields.io/lgtm/grade/cpp/g/cfengine/core.svg?logo=lgtm&logoWidth=18&label=code%20quality)](https://lgtm.com/projects/g/cfengine/core/) # CFEngine 3 From f24ddbcb2cd9bb2b5cce94c93e772e4d3e7e8520 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Fri, 7 Jan 2022 15:40:48 +0100 Subject: [PATCH 284/333] Only replace the dot after the scope in MangleScopedVarNameIntoSpecialScopeName() In very rare cases where a scoped variable which has the same string as "scope." inside square brackets (classic array index) needs to be put into a special scope, MangleScopedVarNameIntoSpecialScopeName() has to be more careful and only replace the dot after the scope, not in the same string inside the square brackets. Ticket: ENT-8289 Changelog: None (cherry picked from commit d4e7a40e2006714251b16648c2aba0e095a59057) Conflicts: libntech libpromises/eval_context.c --- libntech | 2 +- libpromises/eval_context.c | 6 ++++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/libntech b/libntech index 1ddc193972..e8411e0251 160000 --- a/libntech +++ b/libntech @@ -1 +1 @@ -Subproject commit 1ddc193972340d5c0dd2e49f613ded00aece9497 +Subproject commit e8411e02519874db298e2fce394c04a8b9e65262 diff --git a/libpromises/eval_context.c b/libpromises/eval_context.c index 6fd8df7ea2..84afea4df0 100644 --- a/libpromises/eval_context.c +++ b/libpromises/eval_context.c @@ -1993,8 +1993,10 @@ static inline char *MangleScopedVarNameIntoSpecialScopeName(const char *scope, c char *scope_with_dot = StringConcatenate(2, scope, "."); char *scope_with_underscores = StringConcatenate(2, scope, NESTED_SCOPE_SEP); - ssize_t ret = StringReplace(new_var_name, var_name_len + sizeof(NESTED_SCOPE_SEP), - scope_with_dot, scope_with_underscores); + /* Only replace the first "scope." occurrence (there might be "scope." + * inside square brackets). */ + NDEBUG_UNUSED ssize_t ret = StringReplaceN(new_var_name, var_name_len + sizeof(NESTED_SCOPE_SEP), + scope_with_dot, scope_with_underscores, 1); assert(ret == (var_name_len + sizeof(NESTED_SCOPE_SEP) - 2)); free(scope_with_dot); From ef99ae14a9c11496b4f129f332cbeb440dfe5128 Mon Sep 17 00:00:00 2001 From: larsewi Date: Thu, 6 Jan 2022 16:20:29 +0100 Subject: [PATCH 285/333] No longer possible to undefine reserved hard classes Fixed a bug allowing you to undefine a hard classes. Ticket: ENT-7718 Changelog: Title Signed-off-by: larsewi (cherry picked from commit 945c8483b5fb1baf38110bbccbe8e8b1519beda4) --- libpromises/eval_context.c | 1 + .../02_classes/01_basic/cancel_hardclass.cf | 46 +++++++++++++++++++ 2 files changed, 47 insertions(+) create mode 100644 tests/acceptance/02_classes/01_basic/cancel_hardclass.cf diff --git a/libpromises/eval_context.c b/libpromises/eval_context.c index 84afea4df0..138b2bc591 100644 --- a/libpromises/eval_context.c +++ b/libpromises/eval_context.c @@ -2804,6 +2804,7 @@ static void DeleteAllClasses(EvalContext *ctx, const Rlist *list) { Log(LOG_LEVEL_ERR, "You cannot cancel a reserved hard class '%s' in post-condition classes", RlistScalarValue(rp)); + return; } const char *string = RlistScalarValue(rp); diff --git a/tests/acceptance/02_classes/01_basic/cancel_hardclass.cf b/tests/acceptance/02_classes/01_basic/cancel_hardclass.cf new file mode 100644 index 0000000000..39b8d95ef3 --- /dev/null +++ b/tests/acceptance/02_classes/01_basic/cancel_hardclass.cf @@ -0,0 +1,46 @@ +############################################################################# +# +# Test that undefining hardclasses is not be permitted. +# +############################################################################# + +body common control +{ + bundlesequence => { "init", "test", "check" }; +} + +bundle agent init +{ + +} + +body classes undefine(class) +{ + cancel_kept => { "$(class)" }; + cancel_repaired => { "$(class)" }; + cancel_notkept => { "$(class)" }; +} + +bundle agent test +{ + meta: + "description" -> { "ENT-7718" } + string => "Test that undefining hardclasses is not be permitted."; + + commands: + "/bin/true" + classes => undefine("cfengine"); +} + +bundle agent check +{ + classes: + "passed" + expression => "cfengine"; + + reports: + passed:: + "$(this.promise_filename) Pass"; + !passed:: + "$(this.promise_filename) FAIL"; +} From b33ceed64ad269b88ef3e396584d50470ce609fa Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Tue, 4 Jan 2022 13:29:29 +0100 Subject: [PATCH 286/333] Fixed memory leak in cf-secret Ticket: None Changelog: None Signed-off-by: Lars Erik Wik (cherry picked from commit bad09ea505074bce97f618e92e7fcbce0d8c4f17) --- cf-secret/cf-secret.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/cf-secret/cf-secret.c b/cf-secret/cf-secret.c index 2a63466606..98121586a0 100644 --- a/cf-secret/cf-secret.c +++ b/cf-secret/cf-secret.c @@ -178,6 +178,7 @@ static char *GetHostRSAKey(const char *host, HostRSAKeyType type) Log(LOG_LEVEL_VERBOSE, "Using localhost%s key", key_ext); found = true; snprintf(buffer, BUFSIZE, "%s/ppkeys/localhost%s", WORKDIR, key_ext); + freeaddrinfo(result); return buffer; } found = Address2Hostkey(hash, sizeof(hash), ipaddress); @@ -206,6 +207,7 @@ static char *GetHostRSAKey(const char *host, HostRSAKeyType type) } } } + freeaddrinfo(result); return NULL; } From 13604c7943ab53135f92a84eede82bc9c50d498f Mon Sep 17 00:00:00 2001 From: larsewi Date: Fri, 7 Jan 2022 14:45:30 +0100 Subject: [PATCH 287/333] Fixed workdir path in cf-secret being hardcoded Ticket: None Changelog: None Signed-off-by: larsewi (cherry picked from commit 686d34100ab864710dab7a7ed5ecd52e45bff8ef) --- cf-secret/cf-secret.c | 32 ++++++++++++++++++++++++++++---- 1 file changed, 28 insertions(+), 4 deletions(-) diff --git a/cf-secret/cf-secret.c b/cf-secret/cf-secret.c index 98121586a0..60c7690929 100644 --- a/cf-secret/cf-secret.c +++ b/cf-secret/cf-secret.c @@ -164,7 +164,7 @@ static char *GetHostRSAKey(const char *host, HostRSAKeyType type) return NULL; } - char *buffer = malloc(BUFSIZE); + char *buffer = malloc(PATH_MAX); char hash[CF_HOSTKEY_STRING_SIZE]; char ipaddress[64]; bool found = false; @@ -177,7 +177,15 @@ static char *GetHostRSAKey(const char *host, HostRSAKeyType type) { Log(LOG_LEVEL_VERBOSE, "Using localhost%s key", key_ext); found = true; - snprintf(buffer, BUFSIZE, "%s/ppkeys/localhost%s", WORKDIR, key_ext); + int ret = snprintf(buffer, PATH_MAX, "%s/ppkeys/localhost%s", + GetWorkDir(), key_ext); + if (ret < 0 || ret >= PATH_MAX) + { + Log(LOG_LEVEL_ERR, "Path to RSA key is too long (%d > %d)", + ret, PATH_MAX - 1); + freeaddrinfo(result); + return NULL; + } freeaddrinfo(result); return buffer; } @@ -186,7 +194,15 @@ static char *GetHostRSAKey(const char *host, HostRSAKeyType type) if (found) { Log(LOG_LEVEL_DEBUG, "Found host '%s' for address '%s'", hash, ipaddress); - snprintf(buffer, BUFSIZE, "%s/ppkeys/root-%s%s", WORKDIR, hash, key_ext); + int ret = snprintf(buffer, PATH_MAX, "%s/ppkeys/root-%s%s", + GetWorkDir(), hash, key_ext); + if (ret < 0 || ret >= PATH_MAX) + { + Log(LOG_LEVEL_ERR, "Path to RSA key is too long (%d > %d)", ret, + PATH_MAX - 1); + freeaddrinfo(result); + return NULL; + } freeaddrinfo(result); return buffer; } @@ -198,7 +214,15 @@ static char *GetHostRSAKey(const char *host, HostRSAKeyType type) inet_ntop(res->ai_family, GetIPAddress((struct sockaddr *) res->ai_addr), ipaddress, sizeof(ipaddress)); - snprintf(buffer, BUFSIZE, "%s/ppkeys/root-%s%s", WORKDIR, ipaddress, key_ext); + int ret = snprintf(buffer, BUFSIZE, "%s/ppkeys/root-%s%s", + GetWorkDir(), ipaddress, key_ext); + if (ret < 0 || ret >= PATH_MAX) + { + Log(LOG_LEVEL_ERR, "Path to RSA key is too long (%d > %d)", + ret, PATH_MAX - 1); + freeaddrinfo(result); + return NULL; + } if (access(buffer, F_OK) == 0) { Log(LOG_LEVEL_DEBUG, "Found matching key: '%s'", buffer); From 789da54cf32965f10810af247c454ff270d723eb Mon Sep 17 00:00:00 2001 From: larsewi Date: Fri, 7 Jan 2022 14:47:11 +0100 Subject: [PATCH 288/333] Fix assertion error in cf-secret encrypt cf-secret encrypt now encrypts for localhost if no key or host is specified. Ticket: CFE-3874 Changelog: Body Signed-off-by: larsewi (cherry picked from commit 6da2ce8f0940043450a95c5c456e78a26b22390c) --- cf-secret/cf-secret.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/cf-secret/cf-secret.c b/cf-secret/cf-secret.c index 60c7690929..c58673d23a 100644 --- a/cf-secret/cf-secret.c +++ b/cf-secret/cf-secret.c @@ -1006,6 +1006,13 @@ int main(int argc, char *argv[]) key_paths = SeqNew(16, free); } + // Default to localhost on encryption + char *localhost = "127.0.0.1"; + if (encrypt && key_path_arg == NULL && host_arg == NULL) + { + host_arg = localhost; + } + if (host_arg != NULL) { Log(LOG_LEVEL_DEBUG, "-H/--host given: '%s'", host_arg); From ec840f94055c9ac262113079f939c7b6ff731495 Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Tue, 4 Jan 2022 13:30:16 +0100 Subject: [PATCH 289/333] Acceptance test cf-secret encrypt defaults to localhost Acceptance test testing that encrypting with no host-/key argument defaults to encrypting for local host. Ticket: CFE-3874 Changelog: None Signed-off-by: Lars Erik Wik (cherry picked from commit ad2dd7051d93422294a99f176e40966fde36f240) --- .../acceptance/27_cf-secret/encrypt-no-key.cf | 69 +++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 tests/acceptance/27_cf-secret/encrypt-no-key.cf diff --git a/tests/acceptance/27_cf-secret/encrypt-no-key.cf b/tests/acceptance/27_cf-secret/encrypt-no-key.cf new file mode 100644 index 0000000000..ddc1ba9f93 --- /dev/null +++ b/tests/acceptance/27_cf-secret/encrypt-no-key.cf @@ -0,0 +1,69 @@ +############################################################################## +# +# Test that encrypting with no host- / key argument defaults to encrypting for +# localhost. +# +############################################################################## + +body common control +{ + inputs => { "../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +body delete init_delete +{ + dirlinks => "delete"; + rmdirs => "true"; +} + +bundle agent init +{ + methods: + "any" + usebundle => generate_key; + "any" + usebundle => trust_key; + + files: + # Create plain text file + "$(G.testfile).plaintext" + create => "true", + content => "Hello World!"; + + # Delete encrypted file + "$(G.testfile).encrypted" + delete => init_delete; + + # Delete decrypted file + "$(G.testfile).decrypted" + delete => init_delete; +} + +bundle agent test +{ + meta: + "description" -> { "CFE-3874" } + string => "Test that encrypting with no host- / key argument defaults to encrypting for localhost."; + + commands: + # Encrypt test file + "$(sys.cf_secret)" + args => "encrypt -o $(G.testfile).encrypted $(G.testfile).plaintext", + handle => "file-is-encrypted"; + + # Decrypt test file + "$(sys.cf_secret)" + args => "decrypt -o $(G.testfile).decrypted $(G.testfile).encrypted", + depends_on => { "file-is-encrypted" }; +} + +bundle agent check +{ + methods: + "any" + usebundle => dcs_check_diff("$(G.testfile).plaintext", + "$(G.testfile).decrypted", + "$(this.promise_filename)"); +} From 73969325495fd76ef347cba7d58255329ab230f6 Mon Sep 17 00:00:00 2001 From: larsewi Date: Thu, 13 Jan 2022 11:38:07 +0100 Subject: [PATCH 290/333] Make acceptance test compatible with CFE 3.15 Make the acceptance test cherry-picked from commit bad09ea compatible with 3.15. Ticket: CFE-3874 Changelog: None Signed-off-by: larsewi --- tests/acceptance/27_cf-secret/encrypt-no-key.cf | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/tests/acceptance/27_cf-secret/encrypt-no-key.cf b/tests/acceptance/27_cf-secret/encrypt-no-key.cf index ddc1ba9f93..4f3105972c 100644 --- a/tests/acceptance/27_cf-secret/encrypt-no-key.cf +++ b/tests/acceptance/27_cf-secret/encrypt-no-key.cf @@ -18,6 +18,12 @@ body delete init_delete rmdirs => "true"; } +bundle edit_line hello_world +{ + insert_lines: + "Hello World!"; +} + bundle agent init { methods: @@ -30,7 +36,7 @@ bundle agent init # Create plain text file "$(G.testfile).plaintext" create => "true", - content => "Hello World!"; + edit_line => hello_world; # Delete encrypted file "$(G.testfile).encrypted" From 493ffeedd28b172aa83505ad74b26a2e892e79c0 Mon Sep 17 00:00:00 2001 From: larsewi Date: Fri, 7 Jan 2022 15:45:46 +0100 Subject: [PATCH 291/333] Now using StringIsLocalHostIP instead of manual checking Ticket: None Changelog: None Signed-off-by: larsewi (cherry picked from commit 1dd006b0bd84e092766735ebebd9e07b623428e1) --- cf-secret/cf-secret.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/cf-secret/cf-secret.c b/cf-secret/cf-secret.c index c58673d23a..e52fefc2fe 100644 --- a/cf-secret/cf-secret.c +++ b/cf-secret/cf-secret.c @@ -60,6 +60,7 @@ #include #include #include /* DoCleanupAndExit(), CallCleanupFunctions() */ +#include #define BUFSIZE 1024 @@ -173,7 +174,7 @@ static char *GetHostRSAKey(const char *host, HostRSAKeyType type) inet_ntop(res->ai_family, GetIPAddress((struct sockaddr *) res->ai_addr), ipaddress, sizeof(ipaddress)); - if (StringStartsWith(ipaddress, "127.") || StringEqual(ipaddress, "::1")) + if (StringIsLocalHostIP(ipaddress)) { Log(LOG_LEVEL_VERBOSE, "Using localhost%s key", key_ext); found = true; From 157fae474566e17716cb7472b330306aeb0b1ce1 Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Mon, 27 Dec 2021 12:36:43 +0100 Subject: [PATCH 292/333] Replace backticks with single quotes in changelogs Issue was that backticks in changelogs break docs builds, and Nick had to fix it manually. Quote from https://github.com/cfengine/core/pull/4863: > The documentation pulls in changelogs and having markdown inside these files > breaks when the doc build is unable to automatically resolve links. Ticket: ENT-7979 Changelog: None (cherry picked from commit 69d2537c55393e2e4771cb1af33f0a162aa233ba) --- misc/changelog-generator/changelog-generator | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/misc/changelog-generator/changelog-generator b/misc/changelog-generator/changelog-generator index 4fdc7c85c5..148dab26d2 100755 --- a/misc/changelog-generator/changelog-generator +++ b/misc/changelog-generator/changelog-generator @@ -127,7 +127,8 @@ for repo in repos: log_entry_local = False log_entry = "" for line in blob.stdout: - line = line.decode().rstrip('\r\n') + # ENT-7979: we don't want to see backticks ` in our changelogs + line = line.decode().rstrip('\r\n').replace('`',"'") if line == "" and log_entry: add_entry(sha, log_entry) From d464d254dab45211a134f02974e8d849885e2b0e Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Wed, 9 Feb 2022 15:35:06 -0600 Subject: [PATCH 293/333] Moved errors from data_sysctlvalues from inform to verbose Inform level logging is intended for messages about changes that CFEngine is making. These messages are about errors reading files in proc. Prior to this change you would see errors like this while running as a privileged user: info: Error while reading file '/proc/sys/net/ipv6/conf/lo/stable_secret' (Input/output error) info: Error while reading file '/proc/sys/fs/binfmt_misc/register' (Invalid argument) info: Error while reading file '/proc/sys/net/ipv6/route/flush' (Permission denied) info: Error while reading file '/proc/sys/net/ipv6/conf/default/stable_secret' (Input/output error) info: Error while reading file '/proc/sys/net/ipv6/conf/enp0s25/stable_secret' (Input/output error) info: Error while reading file '/proc/sys/net/ipv6/conf/all/stable_secret' (Input/output error) info: Error while reading file '/proc/sys/vm/compact_memory' (Permission denied) info: Error while reading file '/proc/sys/net/ipv6/conf/wlp3s0/stable_secret' (Input/output error) info: Error while reading file '/proc/sys/net/ipv4/route/flush' (Permission denied) info: Error while reading file '/proc/sys/vm/drop_caches' (Permission denied) When running as an unprivledged user you will see many more. This change moves these messages from inform to verbose log level so as not to dirty up output logs. Ticket: CFE-3818 Changelog: Title (cherry picked from commit 031ef4aa0b84bcf5518a56db0865dff71b3f484a) --- libpromises/evalfunction.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libpromises/evalfunction.c b/libpromises/evalfunction.c index 21010bf11e..6e490fc35c 100644 --- a/libpromises/evalfunction.c +++ b/libpromises/evalfunction.c @@ -949,7 +949,7 @@ static FnCallResult FnCallSysctlValue(ARG_UNUSED EvalContext *ctx, if (w == NULL) { - Log(LOG_LEVEL_INFO, "Error while reading file '%s' (%s)", + Log(LOG_LEVEL_VERBOSE, "Error while reading file '%s' (%s)", BufferData(filenamebuf), GetErrorStr()); BufferDestroy(filenamebuf); return FnFailure(); From a6f23938653d4d6972f3fbc580883d1f4b3f3579 Mon Sep 17 00:00:00 2001 From: Lars Erik Wik Date: Fri, 18 Feb 2022 14:27:05 +0100 Subject: [PATCH 294/333] Fix segfault when failing to render inline_mustache Fix segfault when failing to render inline_mustache edit_template_string. Ticket: None Changelog: None Signed-off-by: Lars Erik Wik (cherry picked from commit bded291404e134ec5ecc2f249a9ab6629def2d30) --- cf-agent/verify_files.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/cf-agent/verify_files.c b/cf-agent/verify_files.c index 69b6b2c809..745ce3681d 100644 --- a/cf-agent/verify_files.c +++ b/cf-agent/verify_files.c @@ -675,7 +675,13 @@ static PromiseResult RenderTemplateMustache(EvalContext *ctx, const Promise *pp, } else { - cfPS(ctx, LOG_LEVEL_ERR, PROMISE_RESULT_FAIL, pp, &a, "Error rendering mustache template '%s'", a.edit_template); + /* Use `edit_template` attribute when template method is "mustache". + * Use `edit_template_string` attribute when template method is + * "inline_mustache". + */ + char *tmpl = StringEqual(attr->template_method, "mustache") + ? attr->edit_template : attr->edit_template_string; + cfPS(ctx, LOG_LEVEL_ERR, PROMISE_RESULT_FAIL, pp, &a, "Error rendering mustache template '%s'", tmpl); result = PromiseResultUpdate(result, PROMISE_RESULT_FAIL); BufferDestroy(output_buffer); JsonDestroy(destroy_this); From 9287673cc73d5697c7ef7e4ad662788c7b14dd59 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Thu, 21 Apr 2022 12:40:52 -0500 Subject: [PATCH 295/333] Added protocol 3 (cookie) to syntax description This change adds protocol 3 (cookie) to the syntax-description output which is needed in order to keep automatically generated docs that use the CFEngine_promise_attribute macro up to date. Ticket: ENT-8560 Changelog: Title (cherry picked from commit 909c219ed9c904c64d73bc7d2bad34d562b88067) --- libcfnet/protocol_version.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/libcfnet/protocol_version.c b/libcfnet/protocol_version.c index 59a3cf4f37..5fe1b7aeaf 100644 --- a/libcfnet/protocol_version.c +++ b/libcfnet/protocol_version.c @@ -29,6 +29,10 @@ ProtocolVersion ParseProtocolVersionPolicy(const char *const s) { return CF_PROTOCOL_TLS; } + else if (StringEqual(s, "3") || StringEqual(s, "cookie")) + { + return CF_PROTOCOL_COOKIE; + } else if (StringEqual(s, "latest")) { return CF_PROTOCOL_LATEST; From e99ff8587829018c3b582ca804789c34a440a399 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Fri, 22 Apr 2022 08:32:58 -0500 Subject: [PATCH 296/333] Fixed brace indentation per lars review (cherry picked from commit bfff11b225305afb40bd6e97aa92fe44843fd637) --- libcfnet/protocol_version.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libcfnet/protocol_version.c b/libcfnet/protocol_version.c index 5fe1b7aeaf..31925e9a96 100644 --- a/libcfnet/protocol_version.c +++ b/libcfnet/protocol_version.c @@ -30,7 +30,7 @@ ProtocolVersion ParseProtocolVersionPolicy(const char *const s) return CF_PROTOCOL_TLS; } else if (StringEqual(s, "3") || StringEqual(s, "cookie")) - { + { return CF_PROTOCOL_COOKIE; } else if (StringEqual(s, "latest")) From 99942c357321320e9f8e6c2b707ddf32fe3562e3 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Fri, 22 Apr 2022 08:53:33 -0500 Subject: [PATCH 297/333] Fixed syntax-description output, added cookie, removed undefined (cherry picked from commit f13eefe1cf6d3610f89f1539111a7b5bb1b3df0c) --- libpromises/mod_common.c | 2 +- libpromises/mod_files.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/libpromises/mod_common.c b/libpromises/mod_common.c index e4237ae24b..d8e71a4cde 100644 --- a/libpromises/mod_common.c +++ b/libpromises/mod_common.c @@ -261,7 +261,7 @@ const ConstraintSyntax CFG_CONTROLBODY[COMMON_CONTROL_MAX + 1] = ConstraintSyntaxNewBool("fips_mode", "Activate full FIPS mode restrictions. Default value: false", SYNTAX_STATUS_NORMAL), ConstraintSyntaxNewReal("bwlimit", CF_VALRANGE, "Limit outgoing protocol bandwidth in Bytes per second", SYNTAX_STATUS_NORMAL), ConstraintSyntaxNewBool("cache_system_functions", "Cache the result of system functions. Default value: true", SYNTAX_STATUS_NORMAL), - ConstraintSyntaxNewOption("protocol_version", "0,undefined,1,classic,2,latest", "CFEngine protocol version to use when connecting to the server. Default: \"latest\"", SYNTAX_STATUS_NORMAL), + ConstraintSyntaxNewOption("protocol_version", "1,classic,2,tls,3,cookie,latest", "CFEngine protocol version to use when connecting to the server. Default: \"latest\"", SYNTAX_STATUS_NORMAL), ConstraintSyntaxNewString("tls_ciphers", "", "List of acceptable ciphers in outgoing TLS connections, defaults to OpenSSL's default. For syntax help see man page for \"openssl ciphers\"", SYNTAX_STATUS_NORMAL), ConstraintSyntaxNewString("tls_min_version", "", "Minimum acceptable TLS version for outgoing connections, defaults to OpenSSL's default", SYNTAX_STATUS_NORMAL), ConstraintSyntaxNewStringList("package_inventory", ".*", "Name of the package manager used for software inventory management", SYNTAX_STATUS_NORMAL), diff --git a/libpromises/mod_files.c b/libpromises/mod_files.c index 2c4ff30a6a..e4ca23a2fa 100644 --- a/libpromises/mod_files.c +++ b/libpromises/mod_files.c @@ -325,7 +325,7 @@ static const ConstraintSyntax copy_from_constraints[] = ConstraintSyntaxNewBool("trustkey", "true/false trust public keys from remote server if previously unknown. Default value: false", SYNTAX_STATUS_NORMAL), ConstraintSyntaxNewBool("type_check", "true/false compare file types before copying and require match", SYNTAX_STATUS_NORMAL), ConstraintSyntaxNewBool("verify", "true/false verify transferred file by hashing after copy (resource penalty). Default value: false", SYNTAX_STATUS_NORMAL), - ConstraintSyntaxNewOption("protocol_version", "0,undefined,1,classic,2,latest", "CFEngine protocol version to use when connecting to the server. Default: undefined", SYNTAX_STATUS_NORMAL), + ConstraintSyntaxNewOption("protocol_version", "1,classic,2,tls,3,cookie,latest", "CFEngine protocol version to use when connecting to the server. Default: undefined", SYNTAX_STATUS_NORMAL), ConstraintSyntaxNewBool("missing_ok", "true/false Do not treat missing file as an error. Default value: false", SYNTAX_STATUS_NORMAL), ConstraintSyntaxNewNull() }; From bea58531d43b1c78ce40e216a3a28522444b1592 Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Mon, 9 May 2022 16:41:24 +0200 Subject: [PATCH 298/333] Revert "ENT-6939 disable package_lock.cf on all platforms" This reverts commit 5a67674601f2b253deb6556d6fd5115c23ed1074. --- .../00_basics/ifelapsed_and_expireafter/timed/package_lock.cf | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/package_lock.cf b/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/package_lock.cf index 120a8fca0f..334b1de0ac 100644 --- a/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/package_lock.cf +++ b/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/package_lock.cf @@ -13,8 +13,7 @@ bundle agent init # "no_fds" won't work correctly on Windows. # Also Solaris 9 and 10 don't seem to handle the backgrounding used in # this test. But it seems unrelated to the actual fix. - # Moreover, this test was disabled on all platforms in ENT-6939 - "test_skip_needs_work" string => "any"; + "test_skip_needs_work" string => "windows|sunos_5_9|sunos_5_10"; # The backgrounding doesn't work well under fakeroot. "test_skip_unsupported" string => "using_fakeroot"; From 093cff5782ea03c8fd81941751bea4c15db74f06 Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Fri, 22 Apr 2022 11:53:28 +0200 Subject: [PATCH 299/333] Increase waiting time for the package_lock.cf To give spawned cf-agent more time to finish Ticket: ENT-6967 Changelog: none (cherry picked from commit 8d352d1c249a55eaa16c970b96567f7558f68f06) --- .../00_basics/ifelapsed_and_expireafter/timed/package_lock.cf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/package_lock.cf b/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/package_lock.cf index 334b1de0ac..76ab8a2908 100644 --- a/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/package_lock.cf +++ b/tests/acceptance/00_basics/ifelapsed_and_expireafter/timed/package_lock.cf @@ -46,8 +46,8 @@ bundle agent check test_pass_1:: # We wait 61 seconds in the sub invocation, but in practice test platforms # experience all sorts of timing delays, let's give it plenty of time to - # make sure it finishes. - "any" usebundle => dcs_wait($(this.promise_filename), 120); + # make sure it finishes. 300 seconds = 5 minutes. + "any" usebundle => dcs_wait($(this.promise_filename), 300); test_pass_2:: "any" usebundle => dcs_check_diff($(G.testfile), From 7f67651cb0e5ae1be720e1f8ef8344d21795d4eb Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Tue, 14 Jun 2022 16:30:05 +0200 Subject: [PATCH 300/333] Added changelog for 3.15.6 --- ChangeLog | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/ChangeLog b/ChangeLog index d7c356da4f..2bb49ce845 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,11 @@ +3.15.6: + - Added protocol 3 (cookie) to syntax description (ENT-8560) + - cf-secret encrypt now encrypts for localhost if no key or host is specified (CFE-3874) + - Federated Reporting is now supported on RHEL 8 hubs (ENT-7493) + - Moved errors from data_sysctlvalues from inform to verbose (CFE-3818) + - No longer possible to undefine reserved hard classes (ENT-7718) + - cf-agent can now properly install packages on RHEL 8 (ENT-7493) + 3.15.5: - Added "Wants cf-postgres.service" to cf-apache.service in systemd configuration (ENT-5125) From e9b804bd708f485fe85e084f26850b01acedbae4 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 8 Jun 2022 11:13:01 +0200 Subject: [PATCH 301/333] Define a new type for our patched apachectl It needs 'bin_t:file execute;' and granting this to the cfengine_httpd_t doman that it has without this change would mean giving that access to our httpd and php processes too. (cherry picked from commit dec0d70c4b1326ffcaa49a62799eadd8beffb0d9) Conflicts: misc/selinux/cfengine-enterprise.te --- misc/selinux/cfengine-enterprise.fc | 1 + misc/selinux/cfengine-enterprise.te | 52 +++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) diff --git a/misc/selinux/cfengine-enterprise.fc b/misc/selinux/cfengine-enterprise.fc index 6108af4d8a..7af4fb8ff7 100644 --- a/misc/selinux/cfengine-enterprise.fc +++ b/misc/selinux/cfengine-enterprise.fc @@ -4,6 +4,7 @@ /var/cfengine/bin/cf-agent -- gen_context(system_u:object_r:cfengine_agent_exec_t,s0) /var/cfengine/bin/cf-hub -- gen_context(system_u:object_r:cfengine_hub_exec_t,s0) /var/cfengine/bin/pg.* -- gen_context(system_u:object_r:cfengine_postgres_exec_t,s0) +/var/cfengine/httpd/bin/apachectl -- gen_context(system_u:object_r:cfengine_apachectl_exec_t,s0) /var/cfengine/httpd/bin/.* -- gen_context(system_u:object_r:cfengine_httpd_exec_t,s0) /var/cfengine/httpd/php/bin/.* -- gen_context(system_u:object_r:cfengine_httpd_exec_t,s0) /opt/cfengine(/.*)? gen_context(system_u:object_r:cfengine_var_lib_t,s0) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 905c8aaf20..12994aea73 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -148,8 +148,13 @@ require { class dccp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; class ib_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; class mpls_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; +<<<<<<< HEAD class process { setrlimit transition dyntransition execstack execheap execmem signull siginh }; class file { execute execute_no_trans getattr ioctl map open read unlink write entrypoint lock link rename append setattr create relabelfrom relabelto watch watch_reads }; +======= + class process { setrlimit transition dyntransition execstack execheap execmem signull siginh getattr }; + class file { execute execute_no_trans getattr ioctl map open read unlink write entrypoint lock link rename append setattr create relabelfrom relabelto }; +>>>>>>> dec0d70c4 (Define a new type for our patched apachectl) class fifo_file { create open getattr setattr read write append rename link unlink ioctl lock relabelfrom relabelto }; class dir { getattr read search open write add_name remove_name lock ioctl create }; class filesystem getattr; @@ -604,6 +609,53 @@ allow cfengine_httpd_t shell_exec_t:file map; allow cfengine_httpd_t shell_exec_t:file { execute execute_no_trans }; +#============= cfengine_apachectl_t ============== +type cfengine_apachectl_t; +typeattribute cfengine_apachectl_t domain; +role system_r types cfengine_apachectl_t; + +# /var/cfengine/httpd/bin/apachectl has the 'cfengine_apachectl_exec_t' context which +# is an entrypoint for the 'cfengine_apachectl_t' domain +type cfengine_apachectl_exec_t; +typeattribute cfengine_apachectl_exec_t entry_type; +typeattribute cfengine_apachectl_exec_t exec_type; +typeattribute cfengine_apachectl_exec_t file_type, non_security_file_type, non_auth_file_type; +role object_r types cfengine_apachectl_exec_t; + +# allow transitions from init_t (systemd) to cfengine_apachectl_t to cfengine_httpd_t +type_transition init_t cfengine_apachectl_exec_t:process cfengine_apachectl_t; +allow init_t cfengine_apachectl_t:process transition; +allow init_t cfengine_apachectl_exec_t:file { execute getattr open read }; +allow init_t cfengine_apachectl_t:process siginh; + +type_transition cfengine_apachectl_t cfengine_httpd_exec_t:process cfengine_httpd_t; +allow cfengine_apachectl_t cfengine_httpd_t:process transition; +allow cfengine_apachectl_t cfengine_httpd_exec_t:file { execute getattr open read }; +allow cfengine_apachectl_t cfengine_httpd_t:process siginh; + +allow cfengine_apachectl_t cfengine_apachectl_exec_t:file entrypoint; +allow cfengine_apachectl_t cfengine_apachectl_exec_t:file { ioctl read getattr lock map execute open }; + +allow cfengine_apachectl_t cfengine_var_lib_t:dir search; +allow cfengine_apachectl_t cfengine_var_lib_t:file { getattr open read }; +allow cfengine_apachectl_t init_t:unix_stream_socket ioctl; +allow cfengine_apachectl_t passwd_file_t:file { getattr open read }; +allow cfengine_apachectl_t shell_exec_t:file { map execute }; +allow cfengine_apachectl_t sssd_public_t:dir search; +allow cfengine_apachectl_t sssd_public_t:file { getattr open read map }; +allow cfengine_apachectl_t sssd_t:unix_stream_socket connectto; +allow cfengine_apachectl_t sssd_var_lib_t:dir search; +allow cfengine_apachectl_t sssd_var_lib_t:sock_file write; +allow cfengine_apachectl_t bin_t:file { execute execute_no_trans map }; +allow cfengine_apachectl_t proc_t:dir read; +allow cfengine_apachectl_t proc_t:file { open read }; + +# allow apachectl to run 'ps' and thus gather information about all running processes on the system +# this is a macro invocation, the file has to be processed with +# make -f /usr/share/selinux/devel/Makefile +ps_process_pattern(cfengine_apachectl_t, domain) + + #============= cfengine_action_script_t ============== # A special type and domain for action (notification/alert) scripts executed by # Mission Portal. They can do anything, so they need to run in an unconstrained From e37969d858069bba9a45f2871edd026f9103f4c6 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 8 Jun 2022 13:15:22 +0200 Subject: [PATCH 302/333] Allow cfengine_httpd_t to 'setopt' on a UDP socket 'connect', 'create' and 'getattr' are already allowed, 'setopt' should be fine too. (cherry picked from commit 7974ce69489aa7ece6692e571580da168995026f) --- misc/selinux/cfengine-enterprise.te | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 12994aea73..cd91cb34bb 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -579,7 +579,7 @@ allow cfengine_httpd_t self:netlink_route_socket { bind create getattr nlmsg_rea allow cfengine_httpd_t self:process execmem; allow cfengine_httpd_t unconfined_t:process signull; allow cfengine_httpd_t self:tcp_socket { accept bind connect create getattr getopt listen setopt shutdown read write }; -allow cfengine_httpd_t self:udp_socket { connect create getattr }; +allow cfengine_httpd_t self:udp_socket { connect create getattr setopt }; allow cfengine_httpd_t self:unix_dgram_socket { connect create }; allow cfengine_httpd_t sssd_public_t:dir search; allow cfengine_httpd_t sssd_public_t:file map; From 9a8cb16eb2ece98dd7bfbd7ddf7c106e9c6f9501 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 9 Jun 2022 21:48:06 +0200 Subject: [PATCH 303/333] Reduce noise by not reporting known SELinux issues while still blocking access (cherry picked from commit 11f73716ca0e30c08f7bbf9e2b4714ac90f99f77) --- misc/selinux/cfengine-enterprise.te | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index cd91cb34bb..00c7a0aa41 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -690,3 +690,17 @@ allow cfengine_action_script_t cfengine_action_script_exec_t:file { ioctl read g #============= special rules for Federated Reporting ============= # sshd needs access to files in /opt/cfengine allow sshd_t cfengine_var_lib_t:file { getattr open read }; + + +#============= TO REMOVE ============== +# Daemons should have proper access to files based on DAC rules (file +# permissions), not just because they run under root. +dontaudit cfengine_execd_t self:capability dac_override; +dontaudit cfengine_serverd_t self:capability { dac_override dac_read_search }; + +# cf-promises run by the daemons shouldn't check if a function is available in +# PostgreSQL, the respective returnszero() is in an agent bundle +dontaudit cfengine_execd_t cfengine_postgres_t:unix_stream_socket connectto; +dontaudit cfengine_execd_t tmp_t:sock_file write; +dontaudit cfengine_serverd_t cfengine_postgres_t:unix_stream_socket connectto; +dontaudit cfengine_serverd_t tmp_t:sock_file write; From 43cce6780fcab3de7407de6bebfe9be45f3cde50 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Thu, 16 Jun 2022 08:52:21 +0200 Subject: [PATCH 304/333] Remove an unresolved conflict from backporting SELinux rules Accidentally introduced in e9b804bd708 when backporting things from master. --- misc/selinux/cfengine-enterprise.te | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 00c7a0aa41..e42fd70f8d 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -148,13 +148,8 @@ require { class dccp_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; class ib_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; class mpls_socket { create ioctl read getattr lock write setattr append bind connect getopt setopt shutdown }; -<<<<<<< HEAD - class process { setrlimit transition dyntransition execstack execheap execmem signull siginh }; - class file { execute execute_no_trans getattr ioctl map open read unlink write entrypoint lock link rename append setattr create relabelfrom relabelto watch watch_reads }; -======= class process { setrlimit transition dyntransition execstack execheap execmem signull siginh getattr }; - class file { execute execute_no_trans getattr ioctl map open read unlink write entrypoint lock link rename append setattr create relabelfrom relabelto }; ->>>>>>> dec0d70c4 (Define a new type for our patched apachectl) + class file { execute execute_no_trans getattr ioctl map open read unlink write entrypoint lock link rename append setattr create relabelfrom relabelto watch watch_reads }; class fifo_file { create open getattr setattr read write append rename link unlink ioctl lock relabelfrom relabelto }; class dir { getattr read search open write add_name remove_name lock ioctl create }; class filesystem getattr; From 508af104328740ddc01912ab083f775027f8138f Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Fri, 10 Jun 2022 12:25:07 +0200 Subject: [PATCH 305/333] ENT-8819 Do not unshallow git repo in Travis script New version of determitne-version.sh doesn't need it Also, stop fetching submodules recursevely IDK how it worked before, but recently Travis started failing with an error like this: Fetching submodule libntech fatal: remote error: upload-pack: not our ref 54e12b54a0be037b8fab70433c29194debcb1c91 Errors during submodule fetch: libntech (cherry picked from commit 29567b8ecb8e6014fd1a80c9304ac551a86c0327) --- travis-scripts/script.sh | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/travis-scripts/script.sh b/travis-scripts/script.sh index 7b452f4935..4f1c0d7132 100755 --- a/travis-scripts/script.sh +++ b/travis-scripts/script.sh @@ -18,11 +18,10 @@ INSTDIR=$HOME/cf_install # exit 0 # fi -# Unshallow the clone. Fetch the tags from upstream even if we are on a +# Fetch the tags from upstream even if we are on a # foreign clone. Needed for determine-version.sh to work. -git fetch --unshallow git remote add upstream https://github.com/cfengine/core.git \ - && git fetch upstream 'refs/tags/*:refs/tags/*' + && git fetch --no-recurse-submodules upstream 'refs/tags/*:refs/tags/*' if [ "$TRAVIS_OS_NAME" = osx ] then From 1e95e32c23d338f29b0d0ea882339750b9384de8 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Fri, 17 Jun 2022 12:52:12 -0500 Subject: [PATCH 306/333] Added example illustrating how class guards are sticky Ticket: CFE-3999 Changelog: None (cherry picked from commit be5cf9861612ae2c295423f2e4ea0d781678aef4) --- ...asses_context_applies_multiple_promises.cf | 36 +++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 examples/classes_context_applies_multiple_promises.cf diff --git a/examples/classes_context_applies_multiple_promises.cf b/examples/classes_context_applies_multiple_promises.cf new file mode 100644 index 0000000000..6f1cbdcad6 --- /dev/null +++ b/examples/classes_context_applies_multiple_promises.cf @@ -0,0 +1,36 @@ +#+begin_src cfengine3 +bundle agent __main__ +{ + reports: + "This promise is not restricted."; + + any:: + "Neither is this promise restricted, 'any' is always defined."; + + linux:: + "This promise is restricted to hosts that have the class 'linux' defined."; + + "This promise is also restricted to hosts that have the class 'linux' defined."; + + linux.cfengine_4:: + "This promise is restricted to hosts that have both the 'linux' class AND the 'cfengine_4' class."; + + !any:: + "This promise will never be actuated."; + + vars: + "Message" string => "Hello World!"; + + reports: + "And this promise is again unrestricted"; +} +#+end_src +#+begin_src example_output +#@ ``` +#@ R: This promise is not restricted. +#@ R: Neither is this promise restricted, 'any' is always defined. +#@ R: This promise is restricted to hosts that have the class 'linux' defined. +#@ R: This promise is also restricted to hosts that have the class 'linux' defined. +#@ R: And this promise is again unrestricted +#@ ``` +#+end_src From 16c606cd318223d8cbbe1db9fbb270828aa551eb Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 22 Jun 2022 13:34:17 +0200 Subject: [PATCH 307/333] Fixes for the cfengine_action_script_t domain It should be entered by a transition from the cfengine_httpd_t domain not from init_t. (cherry picked from commit 2646f98e1948dede617e2f15e51e9fdd1f194dc9) --- misc/selinux/cfengine-enterprise.te | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index e42fd70f8d..10f2060d6d 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -674,7 +674,7 @@ typeattribute cfengine_action_script_exec_t exec_type; typeattribute cfengine_action_script_exec_t file_type, non_security_file_type, non_auth_file_type; role object_r types cfengine_action_script_exec_t; -type_transition init_t cfengine_action_script_exec_t:process cfengine_action_script_t; +type_transition cfengine_httpd_t cfengine_action_script_exec_t:process cfengine_action_script_t; allow cfengine_httpd_t cfengine_action_script_t:process transition; allow cfengine_httpd_t cfengine_action_script_exec_t:file { execute execute_no_trans getattr open read }; allow cfengine_httpd_t cfengine_action_script_t:process siginh; From 8e51ad56e5a328fb69dc3de2715c72367db8e21f Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Wed, 29 Jun 2022 22:03:51 +0200 Subject: [PATCH 308/333] Bumped .CFVERSION number to 3.15.7 Signed-off-by: Ole Herman Schumacher Elgesem --- .CFVERSION | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.CFVERSION b/.CFVERSION index 9bf718c0a8..99458f7c25 100644 --- a/.CFVERSION +++ b/.CFVERSION @@ -1 +1 @@ -3.15.6 +3.15.7 From 9a6c3459dc9d390023a22b41b1e9c80f953521f2 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Fri, 27 May 2022 10:47:56 -0500 Subject: [PATCH 309/333] Added test for ENT-8788 This change contains 2 tests. One (tests/acceptance/31_tickets/ENT-8788/1/main.cf) illustrates the issue as described in the ticket. It is marked as a soft-fail since it is not working. The other (tests/acceptance/31_tickets/ENT-8788/2/main.cf) illustrates the workaround described in the ticket. It is currently passing and is not marked to soft fail. Ticket: ENT-8788 Changelog: None (cherry picked from commit 465f41d02f453bcdb4e213827d3fef4a3bf52736) --- .../ENT-8788/1/desired-result.xml.txt | 189 ++++++++++++++++++ .../acceptance/31_tickets/ENT-8788/1/main.cf | 70 +++++++ .../31_tickets/ENT-8788/1/newcontent.xml.txt | 14 ++ .../31_tickets/ENT-8788/1/start.xml.txt | 175 ++++++++++++++++ .../ENT-8788/2/desired-result.xml.txt | 187 +++++++++++++++++ .../acceptance/31_tickets/ENT-8788/2/main.cf | 66 ++++++ .../31_tickets/ENT-8788/2/newcontent.xml.txt | 12 ++ .../31_tickets/ENT-8788/2/start.xml.txt | 175 ++++++++++++++++ 8 files changed, 888 insertions(+) create mode 100644 tests/acceptance/31_tickets/ENT-8788/1/desired-result.xml.txt create mode 100644 tests/acceptance/31_tickets/ENT-8788/1/main.cf create mode 100644 tests/acceptance/31_tickets/ENT-8788/1/newcontent.xml.txt create mode 100644 tests/acceptance/31_tickets/ENT-8788/1/start.xml.txt create mode 100644 tests/acceptance/31_tickets/ENT-8788/2/desired-result.xml.txt create mode 100644 tests/acceptance/31_tickets/ENT-8788/2/main.cf create mode 100644 tests/acceptance/31_tickets/ENT-8788/2/newcontent.xml.txt create mode 100644 tests/acceptance/31_tickets/ENT-8788/2/start.xml.txt diff --git a/tests/acceptance/31_tickets/ENT-8788/1/desired-result.xml.txt b/tests/acceptance/31_tickets/ENT-8788/1/desired-result.xml.txt new file mode 100644 index 0000000000..ae78051030 --- /dev/null +++ b/tests/acceptance/31_tickets/ENT-8788/1/desired-result.xml.txt @@ -0,0 +1,189 @@ + + + + Atlassian JIRA Web Application + The Atlassian JIRA web application - see http://www.atlassian.com/software/jira for more information + + + + + + + + + + JiraImportProgressFilter + com.atlassian.jira.web.filters.JiraImportProgressFilter + + + + SetupProgressFilter + com.atlassian.jira.web.filters.SetupProgressFilter + + + + StartupProgressFilter + com.atlassian.jira.web.filters.StartupProgressFilter + + + + + JiraFirstFilter + com.atlassian.jira.web.filters.JiraFirstFilter + + + + MetricsCollectorFilter + com.atlassian.jira.servermetrics.MetricsCollectorFilter + + + + MultipartBoundaryCheckFilter + com.atlassian.jira.web.filters.MultipartBoundaryCheckFilter + + + + filter-plugin-dispatcher-before-dispatch-include + com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter + + location + before-dispatch + + + dispatcher + INCLUDE + + + + + filter-plugin-dispatcher-before-dispatch-error + com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter + + location + before-dispatch + + + dispatcher + ERROR + + + + + + + MyLoginFilter + com.atlassian.jira.authenticator.my.MyLoginFilter + + + + + MyLoginFilter + /* + REQUEST + FORWARD + + + + JiraLastFilter + com.atlassian.jira.web.filters.JiraLastFilter + + + + + + JiraImportProgressFilter + /importprogress + + + + + + + + + com.atlassian.jira.web.ServletContextProviderListener + + + + + appstatus + com.atlassian.jira.servlet.ApplicationStatusServlet + + + + jsp.func.service.service_002dexecutor_jsp + /func/service/service-executor.jsp + + + + + 600 + + + + + + wsdl + text/xml + + + + + default.jsp + index.html + + + + + 401 + /display-error + + + + + + webwork + /WEB-INF/tld/webwork.tld + + + + sitemesh-page + /WEB-INF/tld/sitemesh-page.tld + + + sitemesh-decorator + /WEB-INF/tld/sitemesh-decorator.tld + + + jiratags + /WEB-INF/tld/atlassian-jira-tags.tld + + + + + + diff --git a/tests/acceptance/31_tickets/ENT-8788/1/main.cf b/tests/acceptance/31_tickets/ENT-8788/1/main.cf new file mode 100644 index 0000000000..6f5975e3bc --- /dev/null +++ b/tests/acceptance/31_tickets/ENT-8788/1/main.cf @@ -0,0 +1,70 @@ +body file control +{ + inputs => { + "../../../default.cf.sub", + }; +} +bundle agent __main__ +# If this is the policy entry (cf-agent --file) then this bundle will be run by default. +{ + methods: + "bundlesequence" usebundle => default("$(this.promise_filename)"); +} + +bundle agent test +{ + meta: + "description" + string => "Inserting content that contains blank lines with file_preserve_block before select_line_matching does not split inserted content before and after select_line_matching", + meta => { "ENT-8788" }; + + "test_soft_fail" + string => "any", + meta => { "ENT-8788" }; + + vars: + "test_file" string => "$(this.promise_dirname)/start.xml.txt"; + "new_content_file" string => "$(this.promise_dirname)/newcontent.xml.txt"; + + files: + "$(test_file)" + create => "false", + edit_defaults => size("500k"), + edit_line => insert_file_as_block_relative_to_first_or_last_line( + "$(new_content_file)", # File with content to insert if not found + "before", # Relative position to insert content (before|after) + "last", # which line match will we insert based on? + "\s+" # Regex to match the line we want to insert relative to + ); +} + +bundle agent check +{ + methods: + + "Pass/Fail" usebundle => dcs_check_diff($(test.test_file), + "$(this.promise_dirname)/desired-result.xml.txt", + $(this.promise_filename)); +} + +bundle edit_line insert_file_as_block_relative_to_first_or_last_line(templatefile, relative_location, first_or_last, location_line_regex) +{ + insert_lines: + + "$(templatefile)" + comment => "Insert the template file into the file being edited", + insert_type => "file_preserve_block", + location => location_before_after_first_or_last_line( $(relative_location), $(first_or_last), $(location_line_regex) ); +} +body location location_before_after_first_or_last_line(relative_position, first_or_last, regex_matching_line) +{ + before_after => "$(relative_position)"; + first_last => "$(first_or_last)"; + select_line_matching => "$(regex_matching_line)"; +} + +body edit_defaults size(bigenough) +{ + max_file_size => "$(bigenough)"; +} + diff --git a/tests/acceptance/31_tickets/ENT-8788/1/newcontent.xml.txt b/tests/acceptance/31_tickets/ENT-8788/1/newcontent.xml.txt new file mode 100644 index 0000000000..b77d8f8770 --- /dev/null +++ b/tests/acceptance/31_tickets/ENT-8788/1/newcontent.xml.txt @@ -0,0 +1,14 @@ + + + MyLoginFilter + com.atlassian.jira.authenticator.my.MyLoginFilter + + + + + MyLoginFilter + /* + REQUEST + FORWARD + + diff --git a/tests/acceptance/31_tickets/ENT-8788/1/start.xml.txt b/tests/acceptance/31_tickets/ENT-8788/1/start.xml.txt new file mode 100644 index 0000000000..f779044aeb --- /dev/null +++ b/tests/acceptance/31_tickets/ENT-8788/1/start.xml.txt @@ -0,0 +1,175 @@ + + + + Atlassian JIRA Web Application + The Atlassian JIRA web application - see http://www.atlassian.com/software/jira for more information + + + + + + + + + + JiraImportProgressFilter + com.atlassian.jira.web.filters.JiraImportProgressFilter + + + + SetupProgressFilter + com.atlassian.jira.web.filters.SetupProgressFilter + + + + StartupProgressFilter + com.atlassian.jira.web.filters.StartupProgressFilter + + + + + JiraFirstFilter + com.atlassian.jira.web.filters.JiraFirstFilter + + + + MetricsCollectorFilter + com.atlassian.jira.servermetrics.MetricsCollectorFilter + + + + MultipartBoundaryCheckFilter + com.atlassian.jira.web.filters.MultipartBoundaryCheckFilter + + + + filter-plugin-dispatcher-before-dispatch-include + com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter + + location + before-dispatch + + + dispatcher + INCLUDE + + + + + filter-plugin-dispatcher-before-dispatch-error + com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter + + location + before-dispatch + + + dispatcher + ERROR + + + + + + JiraLastFilter + com.atlassian.jira.web.filters.JiraLastFilter + + + + + + JiraImportProgressFilter + /importprogress + + + + + + + + + com.atlassian.jira.web.ServletContextProviderListener + + + + + appstatus + com.atlassian.jira.servlet.ApplicationStatusServlet + + + + jsp.func.service.service_002dexecutor_jsp + /func/service/service-executor.jsp + + + + + 600 + + + + + + wsdl + text/xml + + + + + default.jsp + index.html + + + + + 401 + /display-error + + + + + + webwork + /WEB-INF/tld/webwork.tld + + + + sitemesh-page + /WEB-INF/tld/sitemesh-page.tld + + + sitemesh-decorator + /WEB-INF/tld/sitemesh-decorator.tld + + + jiratags + /WEB-INF/tld/atlassian-jira-tags.tld + + + + + + diff --git a/tests/acceptance/31_tickets/ENT-8788/2/desired-result.xml.txt b/tests/acceptance/31_tickets/ENT-8788/2/desired-result.xml.txt new file mode 100644 index 0000000000..df38b2e941 --- /dev/null +++ b/tests/acceptance/31_tickets/ENT-8788/2/desired-result.xml.txt @@ -0,0 +1,187 @@ + + + + Atlassian JIRA Web Application + The Atlassian JIRA web application - see http://www.atlassian.com/software/jira for more information + + + + + + + + + + JiraImportProgressFilter + com.atlassian.jira.web.filters.JiraImportProgressFilter + + + + SetupProgressFilter + com.atlassian.jira.web.filters.SetupProgressFilter + + + + StartupProgressFilter + com.atlassian.jira.web.filters.StartupProgressFilter + + + + + JiraFirstFilter + com.atlassian.jira.web.filters.JiraFirstFilter + + + + MetricsCollectorFilter + com.atlassian.jira.servermetrics.MetricsCollectorFilter + + + + MultipartBoundaryCheckFilter + com.atlassian.jira.web.filters.MultipartBoundaryCheckFilter + + + + filter-plugin-dispatcher-before-dispatch-include + com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter + + location + before-dispatch + + + dispatcher + INCLUDE + + + + + filter-plugin-dispatcher-before-dispatch-error + com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter + + location + before-dispatch + + + dispatcher + ERROR + + + + + + + MyLoginFilter + com.atlassian.jira.authenticator.my.MyLoginFilter + + + MyLoginFilter + /* + REQUEST + FORWARD + + + + JiraLastFilter + com.atlassian.jira.web.filters.JiraLastFilter + + + + + + JiraImportProgressFilter + /importprogress + + + + + + + + + com.atlassian.jira.web.ServletContextProviderListener + + + + + appstatus + com.atlassian.jira.servlet.ApplicationStatusServlet + + + + jsp.func.service.service_002dexecutor_jsp + /func/service/service-executor.jsp + + + + + 600 + + + + + + wsdl + text/xml + + + + + default.jsp + index.html + + + + + 401 + /display-error + + + + + + webwork + /WEB-INF/tld/webwork.tld + + + + sitemesh-page + /WEB-INF/tld/sitemesh-page.tld + + + sitemesh-decorator + /WEB-INF/tld/sitemesh-decorator.tld + + + jiratags + /WEB-INF/tld/atlassian-jira-tags.tld + + + + + + diff --git a/tests/acceptance/31_tickets/ENT-8788/2/main.cf b/tests/acceptance/31_tickets/ENT-8788/2/main.cf new file mode 100644 index 0000000000..0f5226643e --- /dev/null +++ b/tests/acceptance/31_tickets/ENT-8788/2/main.cf @@ -0,0 +1,66 @@ +body file control +{ + inputs => { + "../../../default.cf.sub", + }; +} +bundle agent __main__ +# If this is the policy entry (cf-agent --file) then this bundle will be run by default. +{ + methods: + "bundlesequence" usebundle => default("$(this.promise_filename)"); +} + +bundle agent test +{ + meta: + "description" + string => "Inserting content that does not contain blank lines with file_preserve_block before select_line_matching works as expected", + meta => { "ENT-8788" }; + + vars: + "test_file" string => "$(this.promise_dirname)/start.xml.txt"; + "new_content_file" string => "$(this.promise_dirname)/newcontent.xml.txt"; + + files: + "$(test_file)" + create => "false", + edit_defaults => size("500k"), + edit_line => insert_file_as_block_relative_to_first_or_last_line( + "$(new_content_file)", # File with content to insert if not found + "before", # Relative position to insert content (before|after) + "last", # which line match will we insert based on? + "\s+" # Regex to match the line we want to insert relative to + ); +} + +bundle agent check +{ + methods: + + "Pass/Fail" usebundle => dcs_check_diff($(test.test_file), + "$(this.promise_dirname)/desired-result.xml.txt", + $(this.promise_filename)); +} + +bundle edit_line insert_file_as_block_relative_to_first_or_last_line(templatefile, relative_location, first_or_last, location_line_regex) +{ + insert_lines: + + "$(templatefile)" + comment => "Insert the template file into the file being edited", + insert_type => "file_preserve_block", + location => location_before_after_first_or_last_line( $(relative_location), $(first_or_last), $(location_line_regex) ); +} +body location location_before_after_first_or_last_line(relative_position, first_or_last, regex_matching_line) +{ + before_after => "$(relative_position)"; + first_last => "$(first_or_last)"; + select_line_matching => "$(regex_matching_line)"; +} + +body edit_defaults size(bigenough) +{ + max_file_size => "$(bigenough)"; +} + diff --git a/tests/acceptance/31_tickets/ENT-8788/2/newcontent.xml.txt b/tests/acceptance/31_tickets/ENT-8788/2/newcontent.xml.txt new file mode 100644 index 0000000000..1b0224d437 --- /dev/null +++ b/tests/acceptance/31_tickets/ENT-8788/2/newcontent.xml.txt @@ -0,0 +1,12 @@ + + + MyLoginFilter + com.atlassian.jira.authenticator.my.MyLoginFilter + + + MyLoginFilter + /* + REQUEST + FORWARD + + diff --git a/tests/acceptance/31_tickets/ENT-8788/2/start.xml.txt b/tests/acceptance/31_tickets/ENT-8788/2/start.xml.txt new file mode 100644 index 0000000000..f779044aeb --- /dev/null +++ b/tests/acceptance/31_tickets/ENT-8788/2/start.xml.txt @@ -0,0 +1,175 @@ + + + + Atlassian JIRA Web Application + The Atlassian JIRA web application - see http://www.atlassian.com/software/jira for more information + + + + + + + + + + JiraImportProgressFilter + com.atlassian.jira.web.filters.JiraImportProgressFilter + + + + SetupProgressFilter + com.atlassian.jira.web.filters.SetupProgressFilter + + + + StartupProgressFilter + com.atlassian.jira.web.filters.StartupProgressFilter + + + + + JiraFirstFilter + com.atlassian.jira.web.filters.JiraFirstFilter + + + + MetricsCollectorFilter + com.atlassian.jira.servermetrics.MetricsCollectorFilter + + + + MultipartBoundaryCheckFilter + com.atlassian.jira.web.filters.MultipartBoundaryCheckFilter + + + + filter-plugin-dispatcher-before-dispatch-include + com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter + + location + before-dispatch + + + dispatcher + INCLUDE + + + + + filter-plugin-dispatcher-before-dispatch-error + com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter + + location + before-dispatch + + + dispatcher + ERROR + + + + + + JiraLastFilter + com.atlassian.jira.web.filters.JiraLastFilter + + + + + + JiraImportProgressFilter + /importprogress + + + + + + + + + com.atlassian.jira.web.ServletContextProviderListener + + + + + appstatus + com.atlassian.jira.servlet.ApplicationStatusServlet + + + + jsp.func.service.service_002dexecutor_jsp + /func/service/service-executor.jsp + + + + + 600 + + + + + + wsdl + text/xml + + + + + default.jsp + index.html + + + + + 401 + /display-error + + + + + + webwork + /WEB-INF/tld/webwork.tld + + + + sitemesh-page + /WEB-INF/tld/sitemesh-page.tld + + + sitemesh-decorator + /WEB-INF/tld/sitemesh-decorator.tld + + + jiratags + /WEB-INF/tld/atlassian-jira-tags.tld + + + + + + From ed3d9b610f09f453ca376cbd6f545dc102fa6390 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 27 Jul 2022 15:49:13 +0200 Subject: [PATCH 310/333] Respect 'file_preserve_block' and 'preserve_all_lines' in InsertLineAtLocation() Ticket: ENT-8788 Changelog: Insertion of contents of a file with blank lines into another file with blank lines no longer results in mixed content (cherry picked from commit 2037084dc1ce20f628904366ce1d3d9afab9aee4) --- cf-agent/files_editline.c | 9 ++++++++- tests/acceptance/31_tickets/ENT-8788/1/main.cf | 4 ---- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/cf-agent/files_editline.c b/cf-agent/files_editline.c index 6bce616d63..9314897b6b 100644 --- a/cf-agent/files_editline.c +++ b/cf-agent/files_editline.c @@ -1847,7 +1847,14 @@ static bool InsertLineAtLocation(EvalContext *ctx, char *newline, Item **start, /* Check line neighbourhood in whole file to avoid edge effects, iff we are not preseving block structure */ -{ int preserve_block = StringEqual(a->sourcetype, "preserve_block"); +{ + assert(start != NULL); + assert(a != NULL); + assert(edcontext != NULL); + + bool preserve_block = (StringEqual(a->sourcetype, "preserve_block") || + StringEqual(a->sourcetype, "file_preserve_block") || + StringEqual(a->sourcetype, "preserve_all_lines")); if (!prev) /* Insert at first line */ { diff --git a/tests/acceptance/31_tickets/ENT-8788/1/main.cf b/tests/acceptance/31_tickets/ENT-8788/1/main.cf index 6f5975e3bc..aebde408fc 100644 --- a/tests/acceptance/31_tickets/ENT-8788/1/main.cf +++ b/tests/acceptance/31_tickets/ENT-8788/1/main.cf @@ -18,10 +18,6 @@ bundle agent test string => "Inserting content that contains blank lines with file_preserve_block before select_line_matching does not split inserted content before and after select_line_matching", meta => { "ENT-8788" }; - "test_soft_fail" - string => "any", - meta => { "ENT-8788" }; - vars: "test_file" string => "$(this.promise_dirname)/start.xml.txt"; "new_content_file" string => "$(this.promise_dirname)/newcontent.xml.txt"; From d70497c4e82e5e02baed272ea6c859da13713bad Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Mon, 12 Apr 2021 12:08:44 -0500 Subject: [PATCH 311/333] Update old test for this.promiser in ifvarclass (cherry picked from commit a9a4a2c8cb708d797bd0d6cde8ee3f8ebc86cce2) --- .../01_basic/this_promiser_ifvarclass.cf | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/tests/acceptance/01_vars/01_basic/this_promiser_ifvarclass.cf b/tests/acceptance/01_vars/01_basic/this_promiser_ifvarclass.cf index 5845319128..e39d0d2c84 100644 --- a/tests/acceptance/01_vars/01_basic/this_promiser_ifvarclass.cf +++ b/tests/acceptance/01_vars/01_basic/this_promiser_ifvarclass.cf @@ -1,5 +1,5 @@ # Test that this.promiser works in ifvarclass -# Redmine:7880 (https://cfengine.com/dev/issues/7880) +# CFE-2262 (https://tracker.mender.io/browse/CFE-2262) body common control { @@ -18,26 +18,26 @@ bundle agent init bundle agent test { meta: - "description" - string => "Test that it is possible to use this.promiser in ifvarclass.", - meta => { "redmine#7880" }; + "description" -> { "redmine#7880", "CFE-2262" } + string => "Test that it is possible to use this.promiser in ifvarclass.", + meta => { "redmine#7880", "CFE-2262" }; - "test_soft_fail" - string => "any", - meta => { "redmine#7880" }; + "test_soft_fail" + string => "any", + meta => { "CFE-2262" }; files: # I should be able to use this.promiser to check if the file is a plain # file "$(G.testfile)" delete => tidy, - ifvarclass => isplain( $(this.promiser) ); + ifvarclass => isplain( "$(this.promiser)" ); } bundle agent check { classes: - "ok" not => fileexists("$(G.testfile)"); + "ok" not => fileexists( "$(G.testfile)" ); reports: ok:: From 18cfab43cadf82cd029f397eb6dee43df73c0166 Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Tue, 26 Jul 2022 15:44:03 +0200 Subject: [PATCH 312/333] Define '$(this.promiser)' for checking if/unless in 'files' promises Just like for other promise types, use the real promiser from the policy file (with variables expanded) for if/ifvarclass/unless checking and then undefine it again to let the special behavior of '$(this.promiser)' in 'files' promises take part. Ticket: ENT-7008 Ticket: CFE-2262 Changelog: '$(this.promiser)' can now be used in 'files' promise attributes 'if', 'ifvarclass' and 'unless' (cherry picked from commit 433c193b7d4b4fefc623a505251848b344d6af58) Conflicts: libpromises/promises.c --- libpromises/promises.c | 15 ++++++++++++--- .../01_vars/01_basic/this_promiser_ifvarclass.cf | 4 ---- 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/libpromises/promises.c b/libpromises/promises.c index 079352862f..103d0ace76 100644 --- a/libpromises/promises.c +++ b/libpromises/promises.c @@ -585,9 +585,7 @@ Promise *ExpandDeRefPromise(EvalContext *ctx, const Promise *pp, bool *excluded) Promise *pcopy = xcalloc(1, sizeof(Promise)); pcopy->promiser = RvalScalarValue(returnval); - /* TODO remove the conditions here for fixing redmine#7880. */ - if ((strcmp("files", pp->parent_promise_type->name) != 0) && - (strcmp("storage", pp->parent_promise_type->name) != 0)) + if (!StringEqual("storage", pp->parent_promise_type->name)) { EvalContextVariablePutSpecial(ctx, SPECIAL_SCOPE_THIS, "promiser", pcopy->promiser, CF_DATA_TYPE_STRING, "source=promise"); @@ -726,6 +724,17 @@ Promise *ExpandDeRefPromise(EvalContext *ctx, const Promise *pp, bool *excluded) } } + /* NOTE: We have to undefine the '$(this.promiser)' variable for a 'files' + * promise now because it is later defined for the individual + * expansions of the promise and so it has to be left unexpanded in + * the constraints/attributes. It is, however, defined above just like + * for any other promise so that it can be used in the common + * if/ifvarclass/unless checking. */ + if (StringEqual(pp->parent_promise_type->name, "files")) + { + EvalContextVariableRemoveSpecial(ctx, SPECIAL_SCOPE_THIS, "promiser"); + } + /* Evaluate all constraints. */ for (size_t i = 0; i < SeqLength(pp->conlist); i++) { diff --git a/tests/acceptance/01_vars/01_basic/this_promiser_ifvarclass.cf b/tests/acceptance/01_vars/01_basic/this_promiser_ifvarclass.cf index e39d0d2c84..c623b601fa 100644 --- a/tests/acceptance/01_vars/01_basic/this_promiser_ifvarclass.cf +++ b/tests/acceptance/01_vars/01_basic/this_promiser_ifvarclass.cf @@ -22,10 +22,6 @@ bundle agent test string => "Test that it is possible to use this.promiser in ifvarclass.", meta => { "redmine#7880", "CFE-2262" }; - "test_soft_fail" - string => "any", - meta => { "CFE-2262" }; - files: # I should be able to use this.promiser to check if the file is a plain # file From 3fc1cc95622b89eeabfab6a852b546410414abfc Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Wed, 3 Aug 2022 22:53:00 -0500 Subject: [PATCH 313/333] Adjust cf-check and package module code for empty updates list In some cases the response from the module results in empty data and sometimes no data at all. Not sure why. Regardless it makes sense to refactor the package module handling code to save the updates key with a zero length value in this case instead of sometimes a zero length and sometimes a `\n` (newline) character. Adjusted cf-check to allow zero length values in both dump and validate portions of the code. Ticket: ENT-9050 Changelog: title (cherry picked from commit aebabc46e7adc63d000fa1b0eb4f673fe9273235) --- cf-agent/package_module.c | 4 ++-- cf-check/dump.c | 1 - cf-check/validate.c | 17 ++++++++++------- 3 files changed, 12 insertions(+), 10 deletions(-) diff --git a/cf-agent/package_module.c b/cf-agent/package_module.c index c3b07d3d78..88ce9a0f77 100644 --- a/cf-agent/package_module.c +++ b/cf-agent/package_module.c @@ -741,9 +741,9 @@ int UpdatePackagesDB(Rlist *data, const char *pm_name, UpdateType type) char *inventory_list = BufferClose(inventory_data); /* We can have empty list of installed software or available updates. */ - if (!inventory_list) + if (inventory_list == NULL) { - WriteDB(db_cached, inventory_key, "\n", 1); + WriteDB(db_cached, inventory_key, "", 0); } else { diff --git a/cf-check/dump.c b/cf-check/dump.c index a9ea155ba7..2a5dc0a73d 100644 --- a/cf-check/dump.c +++ b/cf-check/dump.c @@ -47,7 +47,6 @@ static void print_json_string( const char *const data, size_t size, const bool strip_strings) { assert(data != NULL); - assert(size != 0); // Don't know of anything we store which can be size 0 printf("\""); if (size == 0) diff --git a/cf-check/validate.c b/cf-check/validate.c index 9ecd8f13ec..0b1b70309d 100644 --- a/cf-check/validate.c +++ b/cf-check/validate.c @@ -56,11 +56,19 @@ typedef struct ValidatorState static Slice *NewLMDBSlice(void *data, size_t size) { assert(data != NULL); - assert(size > 0); Slice *r = xcalloc(1, sizeof(Slice)); r->size = size; - r->data = xmemdup(data, size); + if (size == 0) + { + // don't call xmemdup(malloc inside) which is indeterminate whether + // it returns NULL or a valid pointer given size is 0. + r->data = NULL; + } + else + { + r->data = xmemdup(data, size); + } return r; } @@ -383,11 +391,6 @@ static void UpdateValidatorLock( static bool ValidateMDBValue( ValidatorState *state, MDB_val value, const char *name) { - if (value.mv_size <= 0) - { - ValidationError(state, "0 size %s", name); - return false; - } if (value.mv_data == NULL) { ValidationError(state, "NULL %s", name); From cf039cf9b2ee499b84856297280a0eaba97d2d8d Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Tue, 2 Aug 2022 14:55:43 +0200 Subject: [PATCH 314/333] Teach acceptance tests about OPENsuse Our acceptance tests know about "suse" and "sles", but not always about "opensuse". This commit fixes it. Ticket: ENT-9064 changelog: none (cherry picked from commit 767a11fb9f6cc3825e8145b807ec1f6542118134) --- tests/acceptance/02_classes/01_basic/expected_os_classes.cf | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/acceptance/02_classes/01_basic/expected_os_classes.cf b/tests/acceptance/02_classes/01_basic/expected_os_classes.cf index 36c65f35f3..4d56f4e742 100644 --- a/tests/acceptance/02_classes/01_basic/expected_os_classes.cf +++ b/tests/acceptance/02_classes/01_basic/expected_os_classes.cf @@ -25,6 +25,7 @@ bundle agent check "aix", "hpux", "suse", + "opensuse", "windows", "freebsd", "macos", From 1ef844d77d8d814ef5f061d0bba98e98c76728ad Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Fri, 5 Aug 2022 14:41:29 +0200 Subject: [PATCH 315/333] Disable tests that fails on opensuse These tests fail on opensuse. Reason is unclear: it might be same as on Ubuntu (ENT-2480), or maybe something else. Pending investigation. Ticket: ENT-9055 changelog: none (cherry picked from commit 8d22f5ea3a4c41ac3730fc934b858135c92643f3) --- .../16_cf-serverd/serial/simple_copy_from_admit_localhost.cf | 3 +++ .../16_cf-serverd/serial/simple_copy_from_deny_localhost.cf | 3 +++ 2 files changed, 6 insertions(+) diff --git a/tests/acceptance/16_cf-serverd/serial/simple_copy_from_admit_localhost.cf b/tests/acceptance/16_cf-serverd/serial/simple_copy_from_admit_localhost.cf index 1bbb800bb4..3cdab09d6f 100644 --- a/tests/acceptance/16_cf-serverd/serial/simple_copy_from_admit_localhost.cf +++ b/tests/acceptance/16_cf-serverd/serial/simple_copy_from_admit_localhost.cf @@ -11,6 +11,9 @@ bundle agent test meta: "test_suppress_fail" string => "windows", meta => { "redmine6405" }; + # Test fails on OpenSUSE 15, pending investigation. + "test_skip_unsupported" string => "opensuse_leap_15", + meta => { "ENT-9055" }; methods: # source file diff --git a/tests/acceptance/16_cf-serverd/serial/simple_copy_from_deny_localhost.cf b/tests/acceptance/16_cf-serverd/serial/simple_copy_from_deny_localhost.cf index fd5ae62a08..2b2d60d80b 100644 --- a/tests/acceptance/16_cf-serverd/serial/simple_copy_from_deny_localhost.cf +++ b/tests/acceptance/16_cf-serverd/serial/simple_copy_from_deny_localhost.cf @@ -11,6 +11,9 @@ bundle agent test meta: "test_suppress_fail" string => "windows", meta => { "redmine6405" }; + # Test fails on OpenSUSE 15, pending investigation. + "test_skip_unsupported" string => "opensuse_leap_15", + meta => { "ENT-9055" }; methods: # source file From 3e6269da1a612b715c3288c91c026ac80250e731 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Tue, 9 Aug 2022 16:42:28 -0500 Subject: [PATCH 316/333] Redacted useless ifelse If this, do that, else, do that anyway. Instead, just do that. Ticket: CFE-4022 Changelog: None (cherry picked from commit e9bf9a246ec50a4afa08e2acabc907153788af5f) --- cf-monitord/env_monitor.c | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/cf-monitord/env_monitor.c b/cf-monitord/env_monitor.c index 318bab7cf1..9f9bedfc1f 100644 --- a/cf-monitord/env_monitor.c +++ b/cf-monitord/env_monitor.c @@ -640,15 +640,7 @@ static void ArmClasses(EvalContext *ctx, const Averages *const av) strcat(ldt_buff, buff); } - if (CF_THIS[i] > av->Q[i].expect) - { - snprintf(buff, CF_BUFSIZE, "%s_high_ldt", name); - } - else - { - snprintf(buff, CF_BUFSIZE, "%s_high_ldt", name); - } - + snprintf(buff, CF_BUFSIZE, "%s_high_ldt", name); AppendItem(&mon_data, buff, "2"); EvalContextHeapPersistentSave(ctx, buff, CF_PERSISTENCE, CONTEXT_STATE_POLICY_PRESERVE, ""); EvalContextClassPutSoft(ctx, buff, CONTEXT_SCOPE_NAMESPACE, ""); From 391f16111e06a5a8bd864b2399959323bb1422aa Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 10 Aug 2022 16:21:22 +0200 Subject: [PATCH 317/333] Fix the default cipher list in acceptance tests 'TLS_AES_256_GCM_SHA384' is part of the list since we introduced TLS 1.3 support. Ticket: ENT-9018 Changelog: None (cherry picked from commit 3e37a1c045b0a526aca3d96c56e2058766b78d4d) --- .../16_cf-serverd/serial/default_ciphers_tlsversion.srv | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/acceptance/16_cf-serverd/serial/default_ciphers_tlsversion.srv b/tests/acceptance/16_cf-serverd/serial/default_ciphers_tlsversion.srv index d8ab374eb2..58e2107236 100644 --- a/tests/acceptance/16_cf-serverd/serial/default_ciphers_tlsversion.srv +++ b/tests/acceptance/16_cf-serverd/serial/default_ciphers_tlsversion.srv @@ -16,8 +16,8 @@ body server control # Use the defaults, i.e. the following: # -# allowciphers => "AES256-GCM-SHA384:AES256-SHA"; -# allowtlsversion => "1.0"; +# allowciphers => "AES256-GCM-SHA384:AES256-SHA:TLS_AES_256_GCM_SHA384"; +# allowtlsversion => "1.1"; allowconnects => { "127.0.0.1" , "::1" }; allowallconnects => { "127.0.0.1" , "::1" }; From 520e4b0927581c7aace7864393dcab10bf607acc Mon Sep 17 00:00:00 2001 From: Vratislav Podzimek Date: Wed, 10 Aug 2022 16:22:58 +0200 Subject: [PATCH 318/333] Make sure cipher list strings don't have trailing colons When splitting the colon-separated list of ciphers between cipher suites (TLS 1.3+) and ciphers (TLS <= 1.2) care must be taken to make sure neither of the new lists ends up with a trailing colon. The items from the original list can be copied over together with their colons, but then we must make sure the last one in each list doesn't have a colon. Also add a verbose log message making it easier to see which ciphers and cipher suits are being enabled. Ticket: ENT-9018 Changelog: Allowed ciphers are now properly split into TLS 1.3 cipher suites and ciphers used for TLS 1.2 and older (cherry picked from commit 15553daa91e302f46ff9d35363e13e6fab3edc4d) --- libcfnet/tls_generic.c | 15 ++++++ .../copy_from_reordered_ciphers_success.cf | 23 +++++++++ ...copy_from_reordered_ciphers_success.cf.sub | 47 +++++++++++++++++++ .../reordered_default_ciphers_tlsversion.srv | 34 ++++++++++++++ 4 files changed, 119 insertions(+) create mode 100644 tests/acceptance/16_cf-serverd/serial/copy_from_reordered_ciphers_success.cf create mode 100644 tests/acceptance/16_cf-serverd/serial/copy_from_reordered_ciphers_success.cf.sub create mode 100644 tests/acceptance/16_cf-serverd/serial/reordered_default_ciphers_tlsversion.srv diff --git a/libcfnet/tls_generic.c b/libcfnet/tls_generic.c index 1a2459f3cb..f4260985a6 100644 --- a/libcfnet/tls_generic.c +++ b/libcfnet/tls_generic.c @@ -1021,8 +1021,22 @@ bool TLSSetCipherList(SSL_CTX *ssl_ctx, const char *cipher_list) } } + /* The above code can leave a trailing colon in the lists because it copies + * the items over *with* the colons splitting them. */ + if ((ciphers_len > 0) && (ciphers[ciphers_len - 1] == ':')) + { + ciphers[ciphers_len - 1] = '\0'; + ciphers_len--; + } + if ((cipher_suites_len > 0) && (cipher_suites[cipher_suites_len - 1] == ':')) + { + cipher_suites[cipher_suites_len - 1] = '\0'; + cipher_suites_len--; + } + if (ciphers_len != 0) /* TLS <= 1.2 ciphers */ { + Log(LOG_LEVEL_VERBOSE, "Enabling ciphers '%s' for TLS 1.2 and older", ciphers); int ret = SSL_CTX_set_cipher_list(ssl_ctx, ciphers); if (ret != 1) { @@ -1034,6 +1048,7 @@ bool TLSSetCipherList(SSL_CTX *ssl_ctx, const char *cipher_list) #ifdef HAVE_TLS_1_3 if (cipher_suites_len != 0) /* TLS >= 1.3 ciphers */ { + Log(LOG_LEVEL_VERBOSE, "Enabling cipher suites '%s' for TLS 1.3 and newer", cipher_suites); int ret = SSL_CTX_set_ciphersuites(ssl_ctx, cipher_suites); if (ret != 1) { diff --git a/tests/acceptance/16_cf-serverd/serial/copy_from_reordered_ciphers_success.cf b/tests/acceptance/16_cf-serverd/serial/copy_from_reordered_ciphers_success.cf new file mode 100644 index 0000000000..d809fc6d31 --- /dev/null +++ b/tests/acceptance/16_cf-serverd/serial/copy_from_reordered_ciphers_success.cf @@ -0,0 +1,23 @@ +body common control +{ + inputs => { "../../default.cf.sub", "../../run_with_server.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + version => "1.0"; +} + +bundle agent test +{ + methods: + # source file + "any" usebundle => file_make("$(G.testdir)/source_file", + "Source file to copy, always fresh, $(sys.date)"); + + # ensure destination files are not there + "any" usebundle => dcs_fini("$(G.testdir)/destfile1"); + + "any" usebundle => generate_key; + + "any" usebundle => start_server("$(this.promise_dirname)/reordered_default_ciphers_tlsversion.srv"); + "any" usebundle => run_test("$(this.promise_filename).sub"); + "any" usebundle => stop_server("$(this.promise_dirname)/reordered_default_ciphers_tlsversion.srv"); +} diff --git a/tests/acceptance/16_cf-serverd/serial/copy_from_reordered_ciphers_success.cf.sub b/tests/acceptance/16_cf-serverd/serial/copy_from_reordered_ciphers_success.cf.sub new file mode 100644 index 0000000000..df151f01e6 --- /dev/null +++ b/tests/acceptance/16_cf-serverd/serial/copy_from_reordered_ciphers_success.cf.sub @@ -0,0 +1,47 @@ +####################################################### +# +# Tries to copy using TLS (which is default now), from two servers: one +# with the default TLS ciphers list and another with a non-default very +# restricted one. +# +# It should succeed in both cases since default client-side cipherlist +# is OpenSSL's default, which is very broad. +# +####################################################### + +body common control +{ + inputs => { "../../default.cf.sub" }; + bundlesequence => { default("$(this.promise_filename)") }; + + # Do not set this, just use the very broad OpenSSL default. + # + # tls_ciphers => ""; +} + +bundle agent init +{ +} + +bundle agent test +{ + files: + # Server with reordered default "allowciphers" + "$(G.testdir)/destfile1" + copy_from => dcs_remote_cp("simple_source", "127.0.0.1", "9893"), + classes => classes_generic("copy1"); +} + +####################################################### + +bundle agent check +{ + classes: + "dummy" expression => regextract("(.*)\.sub", $(this.promise_filename), "fn"); + "exists1" expression => fileexists("$(G.testdir)/destfile1"); + + reports: + + copy1_repaired.exists1:: + "$(fn[1]) Pass"; +} diff --git a/tests/acceptance/16_cf-serverd/serial/reordered_default_ciphers_tlsversion.srv b/tests/acceptance/16_cf-serverd/serial/reordered_default_ciphers_tlsversion.srv new file mode 100644 index 0000000000..d72e0de62a --- /dev/null +++ b/tests/acceptance/16_cf-serverd/serial/reordered_default_ciphers_tlsversion.srv @@ -0,0 +1,34 @@ +body common control +{ + bundlesequence => { "access_rules" }; + inputs => { "../../default.cf.sub" }; + +} + +######################################################### +# Server config +######################################################### + +body server control + +{ + port => "9893"; + + # Use the defaults, but reordered: + allowciphers => "TLS_AES_256_GCM_SHA384:AES256-GCM-SHA384:AES256-SHA"; + + allowconnects => { "127.0.0.1" , "::1" }; + allowallconnects => { "127.0.0.1" , "::1" }; + trustkeysfrom => { "127.0.0.1" , "::1" }; +} + +######################################################### + +bundle server access_rules() + +{ + access: + "$(G.testdir)/source_file" + admit => { "127.0.0.1", "::1" }, + shortcut => "simple_source"; +} From 67fdc4bd46f38619b34ff6cbc0dfc6c88526bcaa Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Mon, 15 Aug 2022 15:14:23 -0500 Subject: [PATCH 319/333] Revert "Redacted useless ifelse" This reverts commit e9bf9a246ec50a4afa08e2acabc907153788af5f. (cherry picked from commit 37c7bfa31fe742162832404ec2eb566f624c2c3f) --- cf-monitord/env_monitor.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/cf-monitord/env_monitor.c b/cf-monitord/env_monitor.c index 9f9bedfc1f..318bab7cf1 100644 --- a/cf-monitord/env_monitor.c +++ b/cf-monitord/env_monitor.c @@ -640,7 +640,15 @@ static void ArmClasses(EvalContext *ctx, const Averages *const av) strcat(ldt_buff, buff); } - snprintf(buff, CF_BUFSIZE, "%s_high_ldt", name); + if (CF_THIS[i] > av->Q[i].expect) + { + snprintf(buff, CF_BUFSIZE, "%s_high_ldt", name); + } + else + { + snprintf(buff, CF_BUFSIZE, "%s_high_ldt", name); + } + AppendItem(&mon_data, buff, "2"); EvalContextHeapPersistentSave(ctx, buff, CF_PERSISTENCE, CONTEXT_STATE_POLICY_PRESERVE, ""); EvalContextClassPutSoft(ctx, buff, CONTEXT_SCOPE_NAMESPACE, ""); From 01d2eb5adf286e9aeff2142968564ef574f8ea44 Mon Sep 17 00:00:00 2001 From: Nick Anderson Date: Mon, 15 Aug 2022 15:14:40 -0500 Subject: [PATCH 320/333] Fixed definition of _low_ldt class from cf-monitord Ticket: CFE-4022 Changelog: Title (cherry picked from commit ad2b8e078847483275998bd4e3b7e1a5dfa37f6d) --- cf-monitord/env_monitor.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cf-monitord/env_monitor.c b/cf-monitord/env_monitor.c index 318bab7cf1..ec953e3215 100644 --- a/cf-monitord/env_monitor.c +++ b/cf-monitord/env_monitor.c @@ -646,7 +646,7 @@ static void ArmClasses(EvalContext *ctx, const Averages *const av) } else { - snprintf(buff, CF_BUFSIZE, "%s_high_ldt", name); + snprintf(buff, CF_BUFSIZE, "%s_low_ldt", name); } AppendItem(&mon_data, buff, "2"); From 5eb943b3102e9a165e97b653734191f3adc9e324 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Mon, 12 Sep 2022 12:45:54 -0500 Subject: [PATCH 321/333] Added github action to compile core and run unit tests Ticket: ENT-9155 Changelog: none (cherry picked from commit 739cc23eab6a05e0d299b3b849472e5eb8885ffc) --- .github/workflows/unit_tests.yml | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 .github/workflows/unit_tests.yml diff --git a/.github/workflows/unit_tests.yml b/.github/workflows/unit_tests.yml new file mode 100644 index 0000000000..20320bd119 --- /dev/null +++ b/.github/workflows/unit_tests.yml @@ -0,0 +1,29 @@ +name: Unit Tests + +on: + # run this workflow on pull_request activity + # this includes opening and pushing more commits + pull_request: + branches: [ master, 3.18.x, 3.15.x ] + + # run this workflow on push/merge activity + # pull_request activity won't detect changes + # in the upstream branch before we merge + push: + branches: [ master, 3.18.x, 3.15.x ] + +jobs: + unit_tests: + runs-on: ubuntu-20.04 + steps: + - uses: actions/checkout@v2 + with: + submodules: recursive + - name: Install dependencies + run: sudo apt-get update -y && sudo apt-get install -y libssl-dev libpam0g-dev liblmdb-dev byacc curl + - name: Run autotools / configure + run: ./autogen.sh --enable-debug + - name: Compile and link (make) + run: make -j8 CFLAGS="-Werror -Wall" + - name: Run unit tests + run: make -C tests/unit check From 8465d09f610255a881da3c361a1a924d66adab68 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Tue, 13 Sep 2022 11:11:49 -0500 Subject: [PATCH 322/333] Removed -Werror in unit tests github action for 3.15.x CFE-3409 addressed these issues around 3.17.x time frame. We don't intend to backport these fixes as 3.15 is nearly EOL at this time. Ticket: ENT-9155 Changelog: none --- .github/workflows/unit_tests.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/unit_tests.yml b/.github/workflows/unit_tests.yml index 20320bd119..d0b7bb46f1 100644 --- a/.github/workflows/unit_tests.yml +++ b/.github/workflows/unit_tests.yml @@ -24,6 +24,6 @@ jobs: - name: Run autotools / configure run: ./autogen.sh --enable-debug - name: Compile and link (make) - run: make -j8 CFLAGS="-Werror -Wall" + run: make -j8 CFLAGS="-Wall" - name: Run unit tests run: make -C tests/unit check From 7e5bb31fd2211f1ee795e9c549099e5f668ec5eb Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Tue, 13 Sep 2022 10:34:27 -0500 Subject: [PATCH 323/333] Added github action to run acceptance tests Ticket: ENT-9157 Changelog: none (cherry picked from commit b95aa015b8b331147b730239af7342ed4099127a) --- .github/workflows/acceptance_tests.yml | 32 ++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 .github/workflows/acceptance_tests.yml diff --git a/.github/workflows/acceptance_tests.yml b/.github/workflows/acceptance_tests.yml new file mode 100644 index 0000000000..eedb160899 --- /dev/null +++ b/.github/workflows/acceptance_tests.yml @@ -0,0 +1,32 @@ +name: Acceptance Tests + +on: + # run this workflow on pull_request activity + # this includes opening and pushing more commits + pull_request: + branches: [ master, 3.18.x, 3.15.x ] + + # run this workflow on push/merge activity + # pull_request activity won't detect changes + # in the upstream branch before we merge + push: + branches: [ master, 3.18.x, 3.15.x ] + +jobs: + acceptance_tests: + runs-on: ubuntu-20.04 + steps: + - uses: actions/checkout@v2 + with: + submodules: recursive + - name: Install dependencies + run: sudo apt-get update -y && sudo apt-get install -y libssl-dev libpam0g-dev liblmdb-dev byacc curl libyaml-dev + - name: Run autotools / configure + run: ./autogen.sh --enable-debug + - name: Compile and link (make) + run: make -j8 CFLAGS="-Werror -Wall" + - name: Run acceptance tests + run: | + cd tests/acceptance + chmod -R go-w . + ./testall --printlog --tests=common,errorlog From 4d68f5a78ed24717c3b8ed6fdeb582956edf4c6d Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Tue, 13 Sep 2022 15:58:37 -0500 Subject: [PATCH 324/333] Removed -Werror in acceptance test compile step 3.15.x doesn't have fixes from CFE-3409 and won't get them Ticket: ENT-9157 Changelog: none --- .github/workflows/acceptance_tests.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/acceptance_tests.yml b/.github/workflows/acceptance_tests.yml index eedb160899..d20b13d85e 100644 --- a/.github/workflows/acceptance_tests.yml +++ b/.github/workflows/acceptance_tests.yml @@ -24,7 +24,7 @@ jobs: - name: Run autotools / configure run: ./autogen.sh --enable-debug - name: Compile and link (make) - run: make -j8 CFLAGS="-Werror -Wall" + run: make -j8 CFLAGS="-Wall" - name: Run acceptance tests run: | cd tests/acceptance From 6179445ee024dacde8fad5af2a5afb8773a8f929 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Wed, 14 Sep 2022 10:03:44 -0500 Subject: [PATCH 325/333] Update libntech to current master To include man page fixes. Ticket: ENT-9115 Changelog: none (cherry picked from commit f3ee94f22991230172ebc23275fe3612ee388b49) --- libntech | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libntech b/libntech index e8411e0251..c6cc919aad 160000 --- a/libntech +++ b/libntech @@ -1 +1 @@ -Subproject commit e8411e02519874db298e2fce394c04a8b9e65262 +Subproject commit c6cc919aad723ef2899edadfc8be64db2ee14e40 From 89997b1b4fc27522c4603736e54900698dffab2c Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Tue, 20 Sep 2022 11:48:09 -0500 Subject: [PATCH 326/333] Refined testall check for rpmcmpver binary On systems without rpm don't bother checking. Ticket: ENT-9162 Changelog: none (cherry picked from commit 42bf0e74cbfa3c36235ca3c041c0a6e557f86016) Conflicts: tests/acceptance/testall 3.15.x didn't get diffutils so had to exclude that from commit. --- tests/acceptance/testall | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/acceptance/testall b/tests/acceptance/testall index b43b9bd5fe..96ac9ca887 100755 --- a/tests/acceptance/testall +++ b/tests/acceptance/testall @@ -927,7 +927,8 @@ do esac done -if [ -n "$AGENT" -o -n "$CF_PROMISES" -o -n "$CF_SERVERD" -o -n "$CF_EXECD" -o -n "$CF_KEY" -o -n "$CF_SECRET" -o -n "$CF_NET" -o -n "$CF_CHECK" -o -n "$CF_RUNAGENT" -o -n "$RPMVERCMP" ] +# check specified binaries, but only check RPMVERCMP on systems with rpm command +if [ -n "$AGENT" -o -n "$CF_PROMISES" -o -n "$CF_SERVERD" -o -n "$CF_EXECD" -o -n "$CF_KEY" -o -n "$CF_SECRET" -o -n "$CF_NET" -o -n "$CF_CHECK" -o -n "$CF_RUNAGENT" -o \( -n "$(command -v rpm)" -a -n "$RPMVERCMP" \) ] then if [ -n "$BINDIR" ] then From 14d995bca64a3a0204140bcb887ed3cdc791c5a2 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Tue, 20 Sep 2022 15:37:51 -0500 Subject: [PATCH 327/333] Refined testall check for rpmcmpver binary (follow up) Needed to add the check for `rpm` to another section where we check that all set binaries exist and are executable. Ticket: ENT-9162 Changelog: none (cherry picked from commit 0c7ca2bb833fe65908ead1737c2825ddf708c458) --- tests/acceptance/testall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/acceptance/testall b/tests/acceptance/testall index 96ac9ca887..b8c1bfb018 100755 --- a/tests/acceptance/testall +++ b/tests/acceptance/testall @@ -978,7 +978,7 @@ CF_RUNAGENT=${CF_RUNAGENT:-${DEFCF_RUNAGENT}} RPMVERCMP=${RPMVERCMP:-${DEFRPMVERCMP}} LIBTOOL=${LIBTOOL:-${DEFLIBTOOL}} -if [ ! -x "$AGENT" -o ! -x "$CF_PROMISES" -o ! -x "$CF_SERVERD" -o ! -x "$CF_EXECD" -o ! -x "$CF_KEY" -o ! -x "$CF_SECRET" -o ! -x "$CF_NET" -o ! -x "$CF_CHECK" -o ! -x "$CF_RUNAGENT" -o ! -x "$RPMVERCMP" ] +if [ ! -x "$AGENT" -o ! -x "$CF_PROMISES" -o ! -x "$CF_SERVERD" -o ! -x "$CF_EXECD" -o ! -x "$CF_KEY" -o ! -x "$CF_SECRET" -o ! -x "$CF_NET" -o ! -x "$CF_CHECK" -o ! -x "$CF_RUNAGENT" -o \( -n "$(command -v rpm)" -a ! -x "$RPMVERCMP" \) ] then echo "ERROR: can't find cf-agent or other binary;" \ " Are you sure you're running this from OBJDIR" \ From 42dfff8f3686c5e70006368fa5e054bfb356e212 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Wed, 21 Sep 2022 15:25:59 -0500 Subject: [PATCH 328/333] Refined check of rpmvercmp in testall script On a test system running Ubuntu, the rpm binary was installed but almost certainly will have no packages installed that way. So make that test instead: rpm -qa | wc -l != 0 means we need rpmvercmp. Ticket: ENT-9162 Changelog: none (cherry picked from commit 6ba3c26591027c77d698b79f3de9aa722f56c408) Conflicts: tests/acceptance/testall 3.15.x didn't get diffutils so had to remove that from commit. --- tests/acceptance/testall | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/tests/acceptance/testall b/tests/acceptance/testall index b8c1bfb018..1f2db16214 100755 --- a/tests/acceptance/testall +++ b/tests/acceptance/testall @@ -927,8 +927,15 @@ do esac done -# check specified binaries, but only check RPMVERCMP on systems with rpm command -if [ -n "$AGENT" -o -n "$CF_PROMISES" -o -n "$CF_SERVERD" -o -n "$CF_EXECD" -o -n "$CF_KEY" -o -n "$CF_SECRET" -o -n "$CF_NET" -o -n "$CF_CHECK" -o -n "$CF_RUNAGENT" -o \( -n "$(command -v rpm)" -a -n "$RPMVERCMP" \) ] +# check if rpm is used on this host, sometimes it can be installed but not used say on Ubuntu +if [ -n "$(command -v rpm)" -a "0" != "$(rpm -qa | wc -l)" ] +then + NEED_RPMVERCMP="yes" +else + NEED_RPMVERCMP="no" +fi + +if [ -n "$AGENT" -o -n "$CF_PROMISES" -o -n "$CF_SERVERD" -o -n "$CF_EXECD" -o -n "$CF_KEY" -o -n "$CF_SECRET" -o -n "$CF_NET" -o -n "$CF_CHECK" -o -n "$CF_RUNAGENT" -o \( "$NEED_RPMVERCMP" = "yes" -a -n "$RPMVERCMP" \) ] then if [ -n "$BINDIR" ] then @@ -961,7 +968,10 @@ find_default_binary DEFCF_SECRET cf-secret find_default_binary DEFCF_NET cf-net find_default_binary DEFCF_CHECK cf-check find_default_binary DEFCF_RUNAGENT cf-runagent -find_default_binary DEFRPMVERCMP rpmvercmp +if [ "$NEED_RPMVERCMP" = "yes" ] +then + find_default_binary DEFRPMVERCMP rpmvercmp +fi [ -x "`pwd`/libtool" ] && DEFLIBTOOL="`pwd`/libtool" [ -x "`pwd`/../../libtool" ] && DEFLIBTOOL="`pwd`/../../libtool" @@ -978,7 +988,7 @@ CF_RUNAGENT=${CF_RUNAGENT:-${DEFCF_RUNAGENT}} RPMVERCMP=${RPMVERCMP:-${DEFRPMVERCMP}} LIBTOOL=${LIBTOOL:-${DEFLIBTOOL}} -if [ ! -x "$AGENT" -o ! -x "$CF_PROMISES" -o ! -x "$CF_SERVERD" -o ! -x "$CF_EXECD" -o ! -x "$CF_KEY" -o ! -x "$CF_SECRET" -o ! -x "$CF_NET" -o ! -x "$CF_CHECK" -o ! -x "$CF_RUNAGENT" -o \( -n "$(command -v rpm)" -a ! -x "$RPMVERCMP" \) ] +if [ ! -x "$AGENT" -o ! -x "$CF_PROMISES" -o ! -x "$CF_SERVERD" -o ! -x "$CF_EXECD" -o ! -x "$CF_KEY" -o ! -x "$CF_SECRET" -o ! -x "$CF_NET" -o ! -x "$CF_CHECK" -o ! -x "$CF_RUNAGENT" -o \( "$NEED_RPMVERCMP" = "yes" -a ! -x "$RPMVERCMP" \) ] then echo "ERROR: can't find cf-agent or other binary;" \ " Are you sure you're running this from OBJDIR" \ From 84dab9029887a38ce4e8e768907a03ae9861c8ac Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Thu, 22 Sep 2022 10:15:16 -0500 Subject: [PATCH 329/333] Refactored github actions to add dependencies No need to run acceptance tests which take a lot of minutes if unit tests don't pass. Ticket: ENT-9203 Changelog: none (cherry picked from commit bfbcfc4f0307c1efa4e0f00a8e0078ca98d29e3c) Conflicts: .github/workflows/cifuzz.yml .github/workflows/ci.yml Removed cifuzz, asan_unit_tests and macos_unit_tests as not present in 3.15.x (cherry picked from commit f02db53f2273098e5ad4690f45dc646849a33332) --- .github/workflows/acceptance_tests.yml | 13 ++----------- .github/workflows/ci.yml | 21 +++++++++++++++++++++ .github/workflows/unit_tests.yml | 16 +++------------- 3 files changed, 26 insertions(+), 24 deletions(-) create mode 100644 .github/workflows/ci.yml diff --git a/.github/workflows/acceptance_tests.yml b/.github/workflows/acceptance_tests.yml index d20b13d85e..d2fa7dfdcc 100644 --- a/.github/workflows/acceptance_tests.yml +++ b/.github/workflows/acceptance_tests.yml @@ -1,22 +1,13 @@ name: Acceptance Tests on: - # run this workflow on pull_request activity - # this includes opening and pushing more commits - pull_request: - branches: [ master, 3.18.x, 3.15.x ] - - # run this workflow on push/merge activity - # pull_request activity won't detect changes - # in the upstream branch before we merge - push: - branches: [ master, 3.18.x, 3.15.x ] + workflow_call jobs: acceptance_tests: runs-on: ubuntu-20.04 steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v3 with: submodules: recursive - name: Install dependencies diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000000..791192837f --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,21 @@ +name: Continuous Integration + +on: + # run this workflow on pull_request activity + # this includes opening and pushing more commits + pull_request: + branches: [ master, 3.18.x, 3.15.x ] + + # run this workflow on push/merge activity + # pull_request activity won't detect changes + # in the upstream branch before we merge + push: + branches: [ master, 3.18.x, 3.15.x ] + + +jobs: + unit_tests: + uses: ./.github/workflows/unit_tests.yml + acceptance_tests: + needs: unit_tests + uses: ./.github/workflows/acceptance_tests.yml diff --git a/.github/workflows/unit_tests.yml b/.github/workflows/unit_tests.yml index d0b7bb46f1..9ddb6c5256 100644 --- a/.github/workflows/unit_tests.yml +++ b/.github/workflows/unit_tests.yml @@ -1,22 +1,12 @@ -name: Unit Tests - on: - # run this workflow on pull_request activity - # this includes opening and pushing more commits - pull_request: - branches: [ master, 3.18.x, 3.15.x ] - - # run this workflow on push/merge activity - # pull_request activity won't detect changes - # in the upstream branch before we merge - push: - branches: [ master, 3.18.x, 3.15.x ] + workflow_call jobs: unit_tests: + name: Run Unit Tests runs-on: ubuntu-20.04 steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v3 with: submodules: recursive - name: Install dependencies From 6457563a8ad5be4201be7b6999140bceee0177bc Mon Sep 17 00:00:00 2001 From: Aleksei Shpakovskii Date: Thu, 20 Oct 2022 13:11:16 +0200 Subject: [PATCH 330/333] Added changelog for 3.15.7 --- ChangeLog | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/ChangeLog b/ChangeLog index 2bb49ce845..69aa46b3ab 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,14 @@ +3.15.7: + - '$(this.promiser)' can now be used in 'files' promise attributes + 'if', 'ifvarclass' and 'unless' (CFE-2262, ENT-7008) + - Adjusted cf-check and package module code for empty updates list + (ENT-9050) + - Allowed ciphers are now properly split into TLS 1.3 cipher suites + and ciphers used for TLS 1.2 and older (ENT-9018) + - Fixed definition of _low_ldt class from cf-monitord (CFE-4022) + - Insertion of contents of a file with blank lines into another file + with blank lines no longer results in mixed content (ENT-8788) + 3.15.6: - Added protocol 3 (cookie) to syntax description (ENT-8560) - cf-secret encrypt now encrypts for localhost if no key or host is specified (CFE-3874) From d8c059da4cc899708e2d5bbc9c4fbc86febdbc87 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Tue, 1 Nov 2022 13:18:05 -0500 Subject: [PATCH 331/333] Allow selinux hub context to send email Found when testing scheduled reports emailing. Ticket: ENT-9557 Changelog: title (cherry picked from commit a304cf100825886af8c929456128194be64b7680) --- misc/selinux/cfengine-enterprise.te | 1 + 1 file changed, 1 insertion(+) diff --git a/misc/selinux/cfengine-enterprise.te b/misc/selinux/cfengine-enterprise.te index 10f2060d6d..19f26cd32c 100644 --- a/misc/selinux/cfengine-enterprise.te +++ b/misc/selinux/cfengine-enterprise.te @@ -423,6 +423,7 @@ allow cfengine_hub_t self:netlink_route_socket { create getopt setopt bind getat allow cfengine_hub_t self:unix_dgram_socket { create connect read write }; allow cfengine_hub_t semanage_exec_t:file getattr; allow cfengine_hub_t shadow_t:file getattr; +allow cfengine_hub_t smtp_port_t:tcp_socket name_connect; allow cfengine_hub_t sssd_public_t:dir search; allow cfengine_hub_t sssd_public_t:file map; allow cfengine_hub_t sssd_public_t:file { getattr open read }; From 613e5f607e789bbd1e376d6cd77bb21f3fd0eb63 Mon Sep 17 00:00:00 2001 From: Craig Comstock Date: Tue, 1 Nov 2022 13:55:56 -0500 Subject: [PATCH 332/333] Fix storage promise for nfs on MacOS Previously the format of the line to check in /etc/fstab would be empty, now it will conform properly to the BSD style format as mentioned in man fstab on MacOS. Ticket: CFE-4093 Changelog: title (cherry picked from commit 50ca18585b3581b7b4876e712b16e57f03023732) --- cf-agent/nfs.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/cf-agent/nfs.c b/cf-agent/nfs.c index be153d33ea..78b493e5f5 100644 --- a/cf-agent/nfs.c +++ b/cf-agent/nfs.c @@ -486,12 +486,14 @@ int VerifyInFstab(EvalContext *ctx, char *name, const Attributes *a, const Promi mountpt, rmountpt, fstype, fstype, host, opts); #elif defined(__linux__) snprintf(fstab, CF_BUFSIZE, "%s:%s \t %s \t %s \t %s", host, rmountpt, mountpt, fstype, opts); -#elif defined(__NetBSD__) || defined(__OpenBSD__) || defined(__DragonFly__) || defined(__FreeBSD__) +#elif defined(__NetBSD__) || defined(__OpenBSD__) || defined(__DragonFly__) || defined(__FreeBSD__) || defined(__APPLE__) snprintf(fstab, CF_BUFSIZE, "%s:%s \t %s \t %s \t %s 0 0", host, rmountpt, mountpt, fstype, opts); #elif defined(__sun) || defined(sco) || defined(__SCO_DS) snprintf(fstab, CF_BUFSIZE, "%s:%s - %s %s - yes %s", host, rmountpt, mountpt, fstype, opts); #elif defined(__CYGWIN__) snprintf(fstab, CF_BUFSIZE, "/bin/mount %s:%s %s", host, rmountpt, mountpt); +#else + #error "Could not determine format of fstab entry on this platform." #endif Log(LOG_LEVEL_VERBOSE, "Verifying '%s' in '%s'", mountpt, VFSTAB[VSYSTEMHARDCLASS]); From 49c8092984d12c2be2475e49af30e97fff37695e Mon Sep 17 00:00:00 2001 From: Ole Herman Schumacher Elgesem Date: Tue, 15 Nov 2022 18:06:27 +0100 Subject: [PATCH 333/333] Bumped .CFVERSION number to 3.15.8 Signed-off-by: Ole Herman Schumacher Elgesem --- .CFVERSION | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.CFVERSION b/.CFVERSION index 99458f7c25..7416f455db 100644 --- a/.CFVERSION +++ b/.CFVERSION @@ -1 +1 @@ -3.15.7 +3.15.8