KV Engine: Recursively list keys #5275

vfauth · 2018-09-05T15:03:43Z

Is your feature request related to a problem? Please describe.
With a KV engine, if I want to list all keys in the directory /foo/, it only returns keys directly under /foo/
For example, if I have the following keys:

/foo/some_key
/foo/bar/some_other_key

A LIST operation on /foo/ returns some_key and bar/, while I would like to have some_key and bar/some_other_key

Describe the solution you'd like
Add a parameter to either recursively return ALL keys in the provided path.

Describe alternatives you've considered
Another way to do it would be to add a parameter specifying the depth up to which look recursively for keys.

The text was updated successfully, but these errors were encountered:

extrail · 2018-11-27T16:20:01Z

This is a very useful feature, especially for the HTTP API

Adds `-recursive', `-depth', `-filter' and `-concurrent' flags to the vault CLI's `kv list' subcommand. Description of the flags: * recursive Recursively list data for a given path. * depth Specifies the depth for recursive listing. * filter Specifies a regular expression for filtering paths. * concurrent Specifies the number of concurrent recursions to run. Reference: hashicorp#5275.

agaudreault · 2019-07-05T19:07:48Z

If anyone stumble upon this I made a little script (not really efficient) while we wait for a native call. Not battle tested but good enough!

./vault-list will list everything you have access in a KV engine
./vault-list secrets/example will list everything under secrets/example/ KV engine

#!/usr/bin/env bash

# Recursive function that will
# - List all the secrets in the given $path
# - Call itself for all path values in the given $path
function traverse {
    local -r path="$1"

    result=$(vault kv list -format=json $path 2>&1)

    status=$?
    if [ ! $status -eq 0 ];
    then
        if [[ $result =~ "permission denied" ]]; then
            return
        fi
        >&2 echo "$result"
    fi

    for secret in $(echo "$result" | jq -r '.[]'); do
        if [[ "$secret" == */ ]]; then
            traverse "$path$secret"
        else
            echo "$path$secret"
        fi
    done
}

# Iterate on all kv engines or start from the path provided by the user
if [[ "$1" ]]; then
    # Make sure the path always end with '/'
    vaults=("${1%"/"}/")
else
    vaults=$(vault secrets list -format=json | jq -r 'to_entries[] | select(.value.type =="kv") | .key')
fi

for vault in $vaults; do
    traverse $vault
done

QWERTY92009 · 2019-09-26T21:29:28Z

Also interested in this kind of feature.

kir4h · 2020-07-22T21:02:12Z

For anyone ending up here, I created a small cli to perform recursive kv read/list operations while we wait for the native solution.
Not very tested yet, I will be fixing bugs as they show up.

antonu17 · 2021-05-28T12:50:32Z

@agaudreault-jive, thanks!
Btw, local readonly path="$1" doesn't work as you might expect. use local -r path="$1" https://stackoverflow.com/a/45409823

vishalnayak · 2021-06-24T20:48:17Z

Issues that are not reproducible and/or have not had any interaction for a long time are stale issues. Sometimes even the valid issues remain stale lacking traction either by the maintainers or the community. In order to provide faster responses and better engagement with the community, we strive to keep the issue tracker clean and the issue count low. In this regard, our current policy is to close stale issues after 30 days. If a feature request is being closed, it means that it is not on the product roadmap. Closed issues will still be indexed and available for future viewers. If users feel that the issue is still relevant but is wrongly closed, we encourage reopening them.

Please refer to our contributing guidelines for details on issue lifecycle.

dondrich · 2021-08-24T14:04:08Z

Would love this feature

rnsc · 2021-08-25T11:09:09Z

I would love to have this natively, this is how I implemented it in Go for a project, it's probably not optimal or anything, but it worked for me.

var secretListPath []string

func isNil(v interface{}) bool {
	return v == nil || (reflect.ValueOf(v).Kind() == reflect.Ptr && reflect.ValueOf(v).IsNil())
}

// ListSecret returns a list of secrets from Vault
func ListSecret(vaultCli *vault.Client, path string) (*vault.Secret, error) {
	secret, err := vaultCli.Logical().List(path)
	if err != nil {
		log.Println("Couldn't list from the Vault.")
	}

	if isNil(secret) {
		log.Printf("Couldn't list %s from the Vault.", path)
	}
	return secret, err
}

// RecursiveListSecret returns a list of secrets paths from Vault
func RecursiveListSecret(vaultCli *vault.Client, path string) []string {
	secretList, err := ListSecret(vaultCli, path)
	if err == nil && secretList != nil {
		for _, secret := range secretList.Data["keys"].([]interface{}) {
			if strings.HasSuffix(secret.(string), "/") {
				RecursiveListSecret(vaultCli, path+secret.(string))
			} else {
				secretListPath = append([]string{strings.Replace(path, "metadata", "data", -1) + secret.(string)}, secretListPath...)
			}
		}
	}
	return secretListPath
}

kir4h · 2021-08-25T11:23:24Z

I would love to have this natively, this is how I implemented it in Go for a project, it's probably not optimal or anything, but it worked for me.

@rnsc Just in case you didn't notice my comment above and it can help, I wrote a small go based cli for this (#5275 (comment))

colinjcahill · 2021-11-05T20:23:12Z

Adding my vote for this feature please.

mechaHarry · 2022-02-18T01:21:29Z

Ditto!
Essentially looking for some kind of vault-grep to find out any secret/engine/etc. that has some key, value, or pathname that matches a desired string.

VWRalf · 2022-05-04T11:02:27Z

One more vote for search

jsalatiel · 2022-08-16T22:58:02Z

please implement this. It has been 4 years!

tguvdamm · 2022-08-23T13:58:37Z

This is a must have, listing all secrets now is a major issue.

maxadamo · 2022-08-26T22:18:14Z

if 103 people upvoted this feature request, maybe it's about time to start adding this feature 😸
For instance, Consul lists all the keys recursively, and Redis does it by default (using: KEYS *).
And whilst the solution from @kir4h works pretty well, it would be nice to expose this feature through the API, and libraries for any language can make use of this functionality.

Sharkzeng · 2022-09-23T08:50:07Z

If anyone stumble upon this I made a little script (not really efficient) while we wait for a native call. Not battle tested but good enough!

./vault-list will list everything you have access in a KV engine ./vault-list secrets/example will list everything under secrets/example/ KV engine

#!/usr/bin/env bash

# Recursive function that will
# - List all the secrets in the given $path
# - Call itself for all path values in the given $path
function traverse {
    local -r path="$1"

    result=$(vault kv list -format=json $path 2>&1)

    status=$?
    if [ ! $status -eq 0 ];
    then
        if [[ $result =~ "permission denied" ]]; then
            return
        fi
        >&2 echo "$result"
    fi

    for secret in $(echo "$result" | jq -r '.[]'); do
        if [[ "$secret" == */ ]]; then
            traverse "$path$secret"
        else
            echo "$path$secret"
        fi
    done
}

# Itera
8000
te on all kv engines or start from the path provided by the user
if [[ "$1" ]]; then
    # Make sure the path always end with '/'
    vaults=("${1%"/"}/")
else
    vaults=$(vault secrets list -format=json | jq -r 'to_entries[] | select(.value.type =="kv") | .key')
fi

for vault in $vaults; do
    traverse $vault
done

if have not install jq command then

ndlanier · 2022-11-14T18:13:55Z

If anyone stumble upon this I made a little script (not really efficient) while we wait for a native call. Not battle tested but good enough!
./vault-list will list everything you have access in a KV engine ./vault-list secrets/example will list everything under secrets/example/ KV engine

#!/usr/bin/env bash

# Recursive function that will
# - List all the secrets in the given $path
# - Call itself for all path values in the given $path
function traverse {
    local -r path="$1"

    result=$(vault kv list -format=json $path 2>&1)

    status=$?
    if [ ! $status -eq 0 ];
    then
        if [[ $result =~ "permission denied" ]]; then
            return
        fi
        >&2 echo "$result"
    fi

    for secret in $(echo "$result" | jq -r '.[]'); do
        if [[ "$secret" == */ ]]; then
            traverse "$path$secret"
        else
            echo "$path$secret"
        fi
    done
}

# Iterate on all kv engines or start from the path provided by the user
if [[ "$1" ]]; then
    # Make sure the path always end with '/'
    vaults=("${1%"/"}/")
else
    vaults=$(vault secrets list -format=json | jq -r 'to_entries[] | select(.value.type =="kv") | .key')
fi

for vault in $vaults; do
    traverse $vault
done

if have not install jq command then

This script doesn't work if you have spaces in the path name. Even with passing the argument with double quotes.

tysonkamp · 2023-04-14T21:14:07Z

@finnstech This is great news. Is there any low/high confidence guess about when such functionality woudl be released? This quarter, this year?

staehler · 2023-07-08T18:30:27Z

Is there any progress on this issue? I'm keen on using this feature.

bitroniq · 2023-07-13T18:03:30Z

Also interested in this feature

bastian-schlueter · 2023-07-14T09:11:23Z

Same here.
We had to super awkwardly implement width-first-search with limited depth in Terraform to be able to retrieve all the paths of secrets of a sub-tree.

mitraed · 2023-08-09T10:58:06Z

very interested in this feature

ramizpolic · 2023-08-16T10:22:26Z

Any updates on this? This would be a highly beneficial feature for our bank-vaults project.

oyvhvi · 2023-08-17T07:02:50Z

How can you not have search yet?

bitroniq · 2023-08-17T15:47:28Z

How can you not have search yet?

@oyvhvi exactly!

At the moment we are implementing our own vault-cli with all workarounds due to this missing feature.

xraystyle · 2023-11-01T20:26:38Z

Just adding another vote for this. I know search in Vault has been a highly contentious topic for literally years at this point, but even some basic search functionality in the KV engine would be incredibly beneficial. As a mitigation for all the performance concerns that are always brought up, how about making it an option one could enable or disable on a given KV engine?

alalkamys · 2023-11-03T13:09:15Z

If anyone stumble upon this I made a little script (not really efficient) while we wait for a native call. Not battle tested but good enough!

./vault-list will list everything you have access in a KV engine ./vault-list secrets/example will list everything under secrets/example/ KV engine

#!/usr/bin/env bash

# Recursive function that will
# - List all the secrets in the given $path
# - Call itself for all path values in the given $path
function traverse {
    local -r path="$1"

    result=$(vault kv list -format=json $path 2>&1)

    status=$?
    if [ ! $status -eq 0 ];
    then
        if [[ $result =~ "permission denied" ]]; then
            return
        fi
        >&2 echo "$result"
    fi

    for secret in $(echo "$result" | jq -r '.[]'); do
        if [[ "$secret" == */ ]]; then
            traverse "$path$secret"
        else
            echo "$path$secret"
        fi
    done
}

# Iterate on all kv engines or start from the path provided by the user
if [[ "$1" ]]; then
    # Make sure the path always end with '/'
    vaults=("${1%"/"}/")
else
    vaults=$(vault secrets list -format=json | jq -r 'to_entries[] | select(.value.type =="kv") | .key')
fi

for vault in $vaults; do
    traverse $vault
done

I built on your code, I had a use case to export all the KV secrets for a given path and save it locally in a secrets.json file. I refactored it to export the secrets rather than list them. My Vault was sitting behind Cloudflare Zero trust, I adjusted the script to optionally pass CF_TOKEN to by pass Cloudflare as well, thought people might find this helpful for their use case:

#!/usr/bin/env bash

# vault-kv-export.sh

set -eo pipefail

readonly ARB_TEMP_SECRETS_FILE="arbitrary_temp_secrets.json"
readonly TEMP_SECRETS_FILE="temp_secrets.json"
readonly SECRETS_FILE="secrets.json"

log() {
    local log_type="$1"
    local message="$2"
    local timestamp
    timestamp=$(date +"%Y-%m-%d %H:%M:%S")
    echo "[$log_type] [$timestamp] $message"
}

log_info() {
    log "INFO" "$1"
}

log_error() {
    log "ERROR" "$1"
    exit 1
}

traverse() {
    local path="$1"
    local result

    local headers=()
    if [[ -n "${CF_TOKEN}" ]]; then
        headers+=("-header" "cf-access-token=${CF_TOKEN}")
    fi

    result=$(vault kv list -format=json "${headers[@]}" "${path}" 2>&1) || log_error "Failed to list secrets: ${result}"

    while IFS= read -r secret; do
        if [[ "${secret}" == */ ]]; then
            traverse "${path}${secret}"
        else
            local secret_data
            secret_data=$(vault kv get -format=json "${headers[@]}" "${path}${secret}" | jq -r '.data') || log_error "Failed to get secret data: ${secret_data}"

            if [[ "${secret_data}" != "null" ]]; then
                echo "{\"path\":\"${path}${secret}\",\"value\":{\"data\":${secret_data}}}," >>"${ARB_TEMP_SECRETS_FILE}"
            fi
        fi
    done < <(echo "${result}" | jq -r '.[]')
}

main() {
    log_info "Starting secrets retrieval process."

    [[ -f "${ARB_TEMP_SECRETS_FILE}" ]] && rm -f "${ARB_TEMP_SECRETS_FILE}"
    [[ -f "${TEMP_SECRETS_FILE}" ]] && rm -f "${TEMP_SECRETS_FILE}"
    [[ -f "${SECRETS_FILE}" ]] && rm -f "${SECRETS_FILE}"

    if [[ -n "${CF_TOKEN}" ]]; then
        log_info "CF_TOKEN detected."
    else
        log_info "CF_TOKEN not provided. Headers will not be attached to Vault requests."
    fi

    local vaults
    if [[ "$1" ]]; then
        vaults=("${1%"/"}/")
        log_info "Retrieving all secrets under ${vaults[*]}.."
    else
        local headers=()
        if [[ -n "${CF_TOKEN}" ]]; then
            headers+=("-header" "cf-access-token=${CF_TOKEN}")
        fi
        log_info "No secret engine provided. Retrieving all secrets.."
        result=$(vault secrets list -format=json "${headers[@]}" 2>&1) || log_error "Failed to list secrets engines: ${result}"
        mapfile -t vaults < <(echo "${result}" | jq -r 'to_entries[] | select(.value.type=="kv") | .key')
    fi

    for vault in "${vaults[@]}"; do
        traverse "${vault}"
    done

    echo "[" >"${TEMP_SECRETS_FILE}"
    sed '$s/,$//' "${ARB_TEMP_SECRETS_FILE}" >>"${TEMP_SECRETS_FILE}"
    echo "]" >>"${TEMP_SECRETS_FILE}"

    jq . "${TEMP_SECRETS_FILE}" >"${SECRETS_FILE}"
    rm "${ARB_TEMP_SECRETS_FILE}" "${TEMP_SECRETS_FILE}"

    log_info "Secrets retrieval completed and saved to ${SECRETS_FILE}"
}

[[ "$0" == "${BASH_SOURCE[0]}" ]] && main "$@"

a sample output:

$ ./vault-kv-export.sh secret2/confluent_cloud

[INFO] [2023-11-03 15:06:15] Starting secrets retrieval process.
[INFO] [2023-11-03 15:06:15] CF_TOKEN detected.
[INFO] [2023-11-03 15:06:15] Retrieving all secrets under secret2/confluent_cloud/..
[INFO] [2023-11-03 15:06:27] Secrets retrieval completed and saved to secrets.json

reading secrets.json

$ cat secrets.json

[
  {
    "path": "secret2/confluent_cloud/global-creds",
    "value": {
      "data": {
        "data": {
          "confluent_cloud_token": "REDACTED"
        },
        <
F438
span class="pl-pds">"metadata": {
          "created_time": "2023-11-02T13:14:09.616943342Z",
          "custom_metadata": null,
          "deletion_time": "",
          "destroyed": false,
          "version": 1
        }
      }
    }
  },
  {
    "path": "secret2/confluent_cloud/service/credentials",
    "value": {
      "data": {
        "data": {
          "api_key": "REDACTED",
          "api_secret": "REDACTED"
        },
        "metadata": {
          "created_time": "2023-11-02T13:14:10.336792836Z",
          "custom_metadata": null,
          "deletion_time": "",
          "destroyed": false,
          "version": 1
        }
      }
    }
  },
...

Hope it helps

biloocabba · 2023-11-15T09:54:43Z

I am adding my vote, is there any news about when we can have it ? or at least what are the best alternatives for the moment?

fvogl · 2023-11-16T15:14:33Z

would also like to see this implemented

JohnDuncanScott · 2023-12-06T10:20:57Z

It would be great to get an update on this. The last official comment on this was March. Secret cycling without this is unbelievably frustrating, especially for complex systems you weren't around for at the beginning (e.g. is this value still used somewhere?).

vickicello · 2023-12-08T20:07:21Z

Adding my vote - this would be super helpful for our use case!

MochaCaffe · 2023-12-09T19:42:29Z

Hi everyone,

I managed to implement a server-side recursive secret search feature by adding a new API endpoint to the KV engine.
This was done by modifying the existing KV secret plugin source code, which means that Vault needs to be recompiled locally.

You can check my work here if you're interested for carrying further testing: https://github.com/kosmos-education/vault-plugin-secrets-kv

As I mentioned in the repository, please note that this is a highly experimental feature which could potentially lead to stability issues or vulnerabilities.
In my case, the plugin was tested against a Vault instance with 6000+ secrets stored

adiii717 · 2023-12-15T09:17:07Z

Any progress or developments regarding this? This feature would provide significant benefits.

voiprodrigo · 2023-12-30T17:23:50Z

I know this feature request is focused on CLI/API, but are there any plans to provide a tree ui for KV mounts on top of this upcoming work? I think that would make the UI much more usable for ops that need to deal on a daily basis with a lot of static secrets.

Also interested in knowing what the community thinks about this. Cheers.

adrianlzt · 2024-01-04T08:51:24Z

Using vault-kv-search and fzf creates a nice way to browse through kv secrets

vault-kv-search --search=path kv -r . --json | jq -r .path | uniq | fzf --preview 'vault kv get --format=yaml ${} | faq -f yaml .data

huyz · 2024-02-18T16:24:00Z

@adrianlzt What's the faq command? (And you're missing a closing quote)

adrianlzt · 2024-02-18T18:07:50Z

faq is like jq but with form input/output formats (yaml between them).

I'm finally using (as the first get of all keys in my environment is quite slow):

vault read -format json sys/internal/ui/mounts/ | faq -r '.data.secret | keys | .[]' | grep -vw -e sys -e cubbyhole -e identity | xargs -n 1 vaku folder list -p > $temp_vault_path_file
cat $temp_vault_path_file | fzf --preview 'vault kv get -format=yaml ${} | faq -f yaml .data.data' -q "$1 $2 $3"

dirsigler · 2024-09-23T11:32:49Z

6 years and no basic implementation of this feature?

heatherezell · 2024-11-13T20:43:24Z

Hello folks! We appreciate the feedback here. Our product managers are looking for folks willing to chat in deeper detail (ie via Zoom). Please email me (heather dot ezell, at hashicorp dot com) if you're willing to set up a deeper-dive conversation around your needs.

heatherezell · 2024-11-13T22:01:27Z

(Corrected my email address.)

pPanda-beta · 2024-12-24T12:39:18Z

End of 2024 and we still did not get this.

Scripts shared above works fine, but have certain limitations like:

How about continue on errors like 403? In real production systems not all team members have access to all keys. So while doing traversal it is safe to ignore 403s
Huge amount of keys: Yes, not all systems are localhost:8200, production systems may contain 10,000+ keys. So need performance.
Not all companies are interested to pay letsencrypt, so they use self signed certificates. VAULT_SKIP_VERIFY does the job.

I made a hacky script (only performance no readability) to list down 10K+ keys in just few seconds.

List Recursive Parallel

list_recur_par() {
  local root="${1:-secret/}" conc="${2:-32}" out_file="${3:-all_paths}"

  echo "$root" > "$out_file"
  dq='"'
  tail -n +1 -f "$out_file" \
    | (grep --line-buffered '/$' || grep '/$') \
    | xargs -P "$conc" -I _KEY bash -E -c "
        export data=\$( vault kv list  '_KEY' | awk -v pref='_KEY' ' NR>=3 { print pref\$0 }' || true );
        flock $out_file -c  ' echo $dq\$data$dq >> $out_file'
      "
}

export VAULT_ADDR=https://......:8200
export VAULT_TOKEN='hvs....'

list_recur_par secret/ 128 /tmp/all_keys
# List all subkeys under "secret/" in 128 parallel processes and save to  /tmp/all_keys, then hang indefinitely (See TODO).

Notes For OSX users

Install flock, brew install flock (I am not skilled developer hence I could not use mkfifo)
add -S 2048 to xargs, yes not every implementation is as good as original linux binaries. Large vault keys are failed in OSX xargs. It also does not respect ARGS_MAX env variable
Use latest version of bash of zsh, OSX's old bash is unusable in 2024.

Works on my❌ every ✅ machine

Use docker

docker run -v $PWD:/work  --rm -it --dns ...(dns only if your company's vault is in VPN)  -e VAULT_SKIP_VERIFY=true -e VAULT_ADDR=https://...:8200 -e VAULT_TOKEN='hvs....'   --entrypoint bash bitnami/vault:1.18.3-debian-12-r0

or the original alpine image (real devs never use alpine or busybox which has pathetic implementation of grep and awk sub command).

docker run -v $PWD:/work  --rm -it --dns ...(dns only if your company's vault is in VPN)  -e VAULT_SKIP_VERIFY=true -e VAULT_ADDR=https://...:8200 -e VAULT_TOKEN='hvs....'  hashicorp/vault:1.18  sh

apk add bash
# ... The script goes here

# (grep --line-buffered '/$' || grep '/$') is used to fallback to busybox grep and you may see error for unknown argument --line-buffered, but that is ok. busybox grep does not have --line-buffered, and awk does not have -W interactive .

TODO

This script is incomplete, it is an infinite snake which eats it's own tail. When there are no child process with vault binary in a steady state, like 5s, you can kill the script considering its complete. In future it will be updated to proper multi process queue management in shell script.

heatherezell · 2024-12-25T07:33:17Z

Hello @pPanda-beta - as you can see from my recent interactions on this issue, I do feel it is somewhat unfair to begin your comment with "End of 2024 and we still did not get this". Please understand that this is an engineering team with constraints on their time, and that there's a strong desire to do what's right for the product and its users instead of a drive to implement anything half-baked just to check a box.

My previous comments also still apply - if you'd like to speak to me or our engineering teams I hope you will email me at heather dot ezell at hashicorp dot com and once the New Year comes, I will be happy to set some time up with you.

In the meantime, please remember that we are all human beings trying our best, and I do apologize for those instances where we fail to meet your expectations as a result. Thanks for your patience.

icy · 2025-01-06T11:17:57Z

My past team desperately wanted this feature because they couldn't afford to buy the enterprise version of Hashicorp Vault. They had to rely on this feature to copy data between vault clusters. There were some adhoc tools from the community, but I think it's a lot easier if this was/is supported officially.

lucabotti · 2025-03-02T17:28:33Z

Hi all,
I created a small python utility to search recursively for values in vault. You can find it here:
vault search replace
A previous (and mentioned) go utility did not work for my use case.

kBite · 2025-05-07T14:09:48Z

@heatherezell Thanks for your feedback on this request. I appreciate your offer. And I'm sure others do, too.

Offering a call with your engineering team is very generous. Have people reached out to you already?

I understand that to an engineer this feature request is not as well-defined as it seems to users.

Can you please share what exactly you would want talk about?

I imagine very few people being confident enough to take up on your offer blindly.

clickyotomy mentioned this issue Mar 23, 2019

Add flags to recursively list secrets for KV Engine. #6463

Closed

catsby added feature-request secret/kv labels Nov 8, 2019

michelvocks mentioned this issue Dec 11, 2019

Recursive list feature hashicorp/vault-plugin-secrets-kv#38

Closed

michelvocks self-assigned this Dec 11, 2019

michelvocks removed their assignment Feb 3, 2020

vishalnayak closed this as completed Jun 24, 2021

kalafut added tmp/rt and removed tmp/rt labels Jun 25, 2021

kalafut reopened this Jun 25, 2021

heatherezell added the community-sentiment Tracking high-profile issues from the community label Oct 12, 2021

heatherezell added the enhancement label May 4, 2022

huyz mentioned this issue Apr 4, 2024

fix: search in fullpath xbglowx/vault-kv-search#69

Merged

cipherboy mentioned this issue Sep 23, 2024

KV: Recursively list keys (Vault #5275) openbao/openbao#549

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KV Engine: Recursively list keys #5275

KV Engine: Recursively list keys #5275

KV Engine: Recursively list keys #5275

KV Engine: Recursively list keys #5275

Comments

List Recursive Parallel

Notes For OSX users

Works on my❌ every ✅ machine

TODO