VAULT-32657 deprecate duplicate attributes in HCL configs and policies #30386
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This upgrades the hcl dependency for the API pkg, and adapts its usage so users of our API pkg are not affected. There's no good way of communicating a warning via a library call so we don't. The tokenHelper which is used by all Vault CLI commands in order to create the Vault client, as well as directly used by the login and server commands, is implemented on the api pkg, so this upgrade also affects all of those commands. Seems like this was only moved to the api pkg because the Terraform provider uses it, and I thought creating a full copy of all those files back under command would be too much spaghetti. Also leaving some TODOs to make next deprecation steps easier.
- vault agent (unit test on CMD warning) - vault proxy (unit test on CMD warning) - vault server (no test for the warning) - vault operator diagnose (no tests at all, uses the same function as vault server
Following operations can trigger this warning when they run into a policy with duplicate attributes: * replication filtered path namespaces invalidation * policy read API * building an ACL (for many different purposes like most authZ operations) * looking up DR token policies * creating a token with named policies * when caching the policies for all namespaces during unseal
No unit tests on these as new test infra would have to be built on all. Operations affected, which will now print a log warning when the retrieved token has an inline policy with duplicate attributes: * capabilities endpoints in sys mount * handing events under a subscription with a token with duplicate attrs in inline policies * token used to create another token has duplicate attrs in inline policies (sudo check) * all uses of fetchACLTokenEntryAndEntity when the request uses a token with inline policies with duplicate attrs. Almost all reqs are subject to this * when tokens are created with inline policies (unclear exactly how that can happen)
|
CI Results: |
|
Build Results: |
good thing it was covered by unit tests
This commit in the hcl library was not in the v1.0.1-vault-5 version we're using but is included in v1.0.1-vault-7: hashicorp/hcl@e80118a This thing of reusing when parsing means that our approach of manually re-parsing fields on top of fields that have already been parsed by the hcl annotation causes strings (maybe more?) to concatenate. Fix that by removing annotation. There's actually more occurrences of this thing of automatically parsing something that is also manually parsing. In some places we could just remove the boilerplate manual parsing, in others we better remove the auto parsing, but I don't wanna pull at that thread right now. I just checked that all places at least fully overwrite the automatically parsed field instead of reusing it as the target of the decode call. The only exception is the AOP field on ent but that doesn't have maps or slices, so I think it's fine. An alternative approach would be to ensure that the auto-parsed value is discarded, like the current parseCache function does note how it's template not templates
mpalmi
previously approved these changes
May 2, 2025
schavis
requested changes
May 2, 2025
website/content/partials/deprecation/duplicate-hcl-attributes.mdx
Outdated
Show resolved
Hide resolved
website/content/partials/deprecation/duplicate-hcl-attributes.mdx
Outdated
Show resolved
Hide resolved
website/content/partials/deprecation/duplicate-hcl-attributes.mdx
Outdated
Show resolved
Hide resolved
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
Co-authored-by: Sarah Chavis <62406755+schavis@users.noreply.github.com>
6 tasks
mpalmi
previously approved these changes
May 22, 2025
schavis
previously approved these changes
May 23, 2025
mpalmi
approved these changes
May 23, 2025
stevendpclark
approved these changes
May 23, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is
30B4
invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR deprecates the usage of duplicate attributes in HCL configuration files and policy definitions. An example of a duplicate attribute is the following, which until support is fully removed will continue parsing to the last occurence of the attribute:
Following our deprecation process, these changes correspond to the deprecation phase to be announced in v1.20 (and cherry picked to v1.19, likely out in v1.19.4), implementing the following:
WARNING: Duplicate keys found. A comprehensive list of those commands is:vault server,vault operator migrate,vault agent,vault proxyandvault operator diagnose;$HOME/.vault) with duplicate attributes will print a similarWARNING: Duplicate keys foundmessage;policy contains duplicate attributes. These include initial caching of policies after unseal, reading policies via the API, using a token that references such a policy among others;Warningsfield in the server response.This was accomplished by upgrading the
hcldependency to reject duplicate attributes, and using the new functions exposed in hashicorp/hcl#707 to create a helper functionParseAndCheckForDuplicateHclAttributesthat parses the hcl and reports if there are duplicates.Unfortunately, the library also had a behavior change that I traced back to hashicorp/hcl@e80118a not being included in the
v1.0.1-vault-5of the library we're using (as it was already included inv1.0.1-vault-4). This means that an approach we've been using of double-parsing some config fields now doesn't work the same anymore, which I'm trying to fix by removing the automatic parsing from the fields that are manually parsed. More details on the specific commit message.Throughout the code I've left these
TODO (HCL_DUP_KEYS_DEPRECATION)comments to more easily find all the places that will need to be changed/deleted after each of the following steps on the deprecation process.For 1.20 we will only have these warnings, but on v1.21 all the operations described above involving HCL definitions with duplicate attributes will simply fail. We will add an environment variable though to allow the behavior to be rolled back to a log-only mode, at least until the final removal is completed on a future version.
Jira: VAULT-32657
ADR: VLT-006: Deprecate and remove duplicate attributes in HCL files in Vault
TODO only if you're a HashiCorp employee
backport/label that matches the desired release branch. Note that in the CE repo, the latest release branch will look likebackport/x.x.x, but older release branches will bebackport/ent/x.x.x+ent.of a public function, even if that change is in a CE file, double check that
applying the patch for this PR to the ENT repo and running tests doesn't
break any tests. Sometimes ENT only tests rely on public functions in CE
files.
in the PR description, commit message, or branch name.
description. Also, make sure the changelog is in this PR, not in your ENT PR.