Description
What is the issue?
I have set log level to debug for my linkerd proxy configuration. But we observed that, this debug log level is resulting in linkerd-proxy containers logging whole incoming request headers. This is a serious security issue as headers contains bearer token, secrets etc and ideally proxy pod should not have logged any http request payload/header. The documentation for log levels says nothing about logging customer data: https://linkerd.io/2.15/reference/proxy-log-level/
Why is linkerd proxy logging contents of the incoming http request as part of application logs??
Do not log the contents/payload/headers of the incoming and outgoing requests ever even for debug or trace log levels.
Only log your linkerd application flow traces/strings in trace logs with no content from requests.
Linkerd-version: 2.13.1
Example Console log:
[305438.538290s] DEBUG ThreadId(01) inbound:accept{client.addr=172.18.37.97:46520}:server{port=80}:http:http{name=data-plane--weather-data-provider-service:80}:profile:http1: linkerd_proxy_http::client: headers={"host": "data-plane--weather-data-provider-service", "user-agent": "",........ and all the headers including all secrets and "traceparent": "00-fc9fcfda6d04569e26aa0f002a24d6ed-ccf325f9bce1e444-00", "content-type": "application/json; charset=utf-8", "content-length": "3009", "l5d-client-id": "default.default.serviceaccount.identity.linkerd.cluster.local"}
The request's source is an internal pod which also has linkerd-proxy side-car container injected.
How can it be reproduced?
Set linkerd-proxy log level to debug.
Logs, error output, etc
Added above
output of linkerd check -o short
linkerd check -o short
Linkerd core checks
===================
linkerd-version
---------------
‼ cli is up-to-date
unsupported version channel: stable-2.11.0
see https://linkerd.io/2.11/checks/#l5d-version-cli for hints
control-plane-version
---------------------
‼ control plane is up-to-date
failed to parse actual version: unsupported version format: 2.13.1
see https://linkerd.io/2.11/checks/#l5d-version-control for hints
‼ control plane and cli versions match
control plane running 2.13.1 but cli running stable-2.11.0
see https://linkerd.io/2.11/checks/#l5d-version-control for hints
linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
some proxies are not running the current version:
* linkerd-destination-855c469f76-8rlrg (2.13.1)
* linkerd-destination-855c469f76-h96wg (2.13.1)
* linkerd-destination-855c469f76-hcx2h (2.13.1)
* linkerd-identity-5b76b77589-jhxqm (2.13.1)
* linkerd-identity-5b76b77589-sj2p6 (2.13.1)
* linkerd-identity-5b76b77589-xv8rw (2.13.1)
* linkerd-proxy-injector-6b48b56cd-6fnrw (2.13.1)
* linkerd-proxy-injector-6b48b56cd-724z6 (2.13.1)
* linkerd-proxy-injector-6b48b56cd-8pnrx (2.13.1)
see https://linkerd.io/2.11/checks/#l5d-cp-proxy-version for hints
‼ control plane proxies and cli versions match
linkerd-destination-855c469f76-8rlrg running 2.13.1 but cli running stable-2.11.0
see https://linkerd.io/2.11/checks/#l5d-cp-proxy-cli-version for hints
linkerd-ha-checks
-----------------
‼ pod injection disabled on kube-system
kube-system namespace needs to have the label config.linkerd.io/admission-webhooks: disabled if injector webhook failure policy is Fail
see https://linkerd.io/2.11/checks/#l5d-injection-disabled for hints
Status check results are √
Environment
- Kubernetes Version: 1.27.9
- Cluster Environment: AKS
- Host OS: Node OS is CBL-Mariner/Linux, kernel version 5.15.153.1-2.cm2, containerd://1.6.26
- Linkerd version: 2.13.1
Possible solution
Do not log the contents/payload/headers of the incoming and outgoing requests ever even for debug or trace log levels.
Only log your linkerd application code trace logs with no content from requests.
Think of linkerd-proxy containers as an external service or SDK, which is handling customer data and so do not log customer's data in logs or give warning in the documentation explicitly here: https://linkerd.io/2.15/reference/proxy-log-level/ and give expose a config to scrub any customer data.
Additional context
No response
Would you like to work on fixing this bug?
None