Description
What is the issue?
I want to deploy linkerd-viz with helm using external certificates, managed through cert-manager. Any way I try this, tap no longer works.
How can it be reproduced?
Deploy certificates:
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: linkerd-viz-issuer
namespace: linkerd-viz
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: tap
namespace: linkerd-viz
spec:
secretName: tap-k8s-tls
duration: 2400h
renewBefore: 1200h
issuerRef:
name: linkerd-viz-issuer
kind: Issuer
commonName: tap.linkerd-viz.svc
dnsNames:
- tap.linkerd-viz.svc
isCA: false
privateKey:
rotationPolicy: Always
algorithm: RSA
usages:
- server auth
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: tap-injector
namespace: linkerd-viz
spec:
secretName: tap-injector-k8s-tls
duration: 2400h
renewBefore: 1200h
issuerRef:
name: linkerd-viz-issuer
kind: Issuer
commonName: tap-injector.linkerd-viz.svc
dnsNames:
- tap-injector.linkerd-viz.svc
isCA: false
privateKey:
rotationPolicy: Always
algorithm: RSA
usages:
- server auth
Deploy linkerd-viz chart with the following values:
tap:
injectCaFrom: tap
externalSecret: true
tapInjector:
injectCaFrom: tap-injector
externalSecret: trueUsing injectCaFromSecret does not work either and shows the same errors:
tap:
injectCaFromSecret: tap-k8s-tls
externalSecret: true
tapInjector:
injectCaFromSecret: tap-injector-k8s-tls
externalSecret: trueI've also tried to hardcode self-signed certificates using crtPEM and keyPEM. However, the caBundle is still the auto-generated ca which changes during each deploy. I cannot overwrite it; as soon as I set it, I must activate injectCaFromSecret. Effectively not being able to use crtPEM and keyPEM is probably a separate issue though.
Not setting these values above lets tap work properly. However, I really need to be able to specify my own certificate; either by hard-coding it through the values or injecting them with cert-manager.
Logs, error output, etc
The following is spammed in the tap logs:
time="2025-05-24T12:56:44Z" level=info msg="running version dev-undefined"
time="2025-05-24T12:56:44Z" level=info msg="starting admin server on :9998"
time="2025-05-24T12:56:44Z" level=info msg="Using trust domain: cluster.local"
time="2025-05-24T12:56:44Z" level=info msg="waiting for caches to sync"
time="2025-05-24T12:56:45Z" level=info msg="caches synced"
time="2025-05-24T12:56:45Z" level=info msg="starting tap API server on :8089" addr=":8089" component=tap
2025/05/24 13:18:10 http: TLS handshake error from 10.2.8.12:45156: EOF
2025/05/24 13:18:10 http: TLS handshake error from 10.2.8.12:45174: EOF
2025/05/24 13:18:10 http: TLS handshake error from 10.2.8.12:45176: EOF
2025/05/24 13:18:10 http: TLS handshake error from 10.2.8.12:45184: EOF
2025/05/24 13:18:10 http: TLS handshake error from 10.2.8.12:45168: EOF
2025/05/24 13:18:10 http: TLS handshake error from 10.2.8.12:45208: EOF
2025/05/24 13:18:10 http: TLS handshake error from 10.2.8.12:45196: EOF
2025/05/24 13:18:10 http: TLS handshake error from 10.2.8.12:45224: EOF
2025/05/24 13:18:10 http: TLS handshake error from 10.2.8.12:45230: EOF
2025/05/24 13:18:10 http: TLS handshake error from 10.2.8.12:45240: remote error: tls: bad certificate
2025/05/24 13:18:10 http: TLS handshake error from 10.2.8.12:45250: EOF
2025/05/24 13:18:10 http: TLS handshake error from 10.2.8.12:45254: remote error: tls: bad certificate
2025/05/24 13:18:11 http: TLS handshake error from 10.2.8.12:47558: EOF
2025/05/24 13:18:11 http: TLS handshake error from 10.2.8.12:47566: remote error: tls: bad certificate
output of linkerd check -o short
linkerd-version
---------------
‼ cli is up-to-date
is running version 25.4.4 but the latest edge version is 25.5.4
see https://linkerd.io/2/checks/#l5d-version-cli for hints
control-plane-version
---------------------
‼ control plane is up-to-date
is running version 25.4.4 but the latest edge version is 25.5.4
see https://linkerd.io/2/checks/#l5d-version-control for hints
linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
some proxies are not running the current version:
* linkerd-destination-5fcb7ff84d-5nn54 (edge-25.4.4)
* linkerd-destination-5fcb7ff84d-9d7h8 (edge-25.4.4)
* linkerd-destination-5fcb7ff84d-d472f (edge-25.4.4)
* linkerd-identity-6c479bb57b-fp6td (edge-25.4.4)
* linkerd-identity-6c479bb57b-mnst8 (edge-25.4.4)
* linkerd-identity-6c479bb57b-xdv2s (edge-25.4.4)
* linkerd-proxy-injector-6cf4d89d46-crdj9 (edge-25.4.4)
* linkerd-proxy-injector-6cf4d89d46-nm8zs (edge-25.4.4)
* linkerd-proxy-injector-6cf4d89d46-rcpjc (edge-25.4.4)
see https://linkerd.io/2/checks/#l5d-cp-proxy-version for hints
linkerd-viz
-----------
× tap API server has valid cert
cert is not issued by the trust anchor: x509: certificate signed by unknown authority
see https://linkerd.io/2/checks/#l5d-tap-cert-valid for hints
Status check results are ×
The error is misleading, as the certificate generated by cert-manager is equivalent to the certificate generated with {{- $ca := genSelfSignedCert $host (list) (list $host) 365 }}.
Environment
- Kubernetes Version: v1.32.2
- Cluster environment: ovh
- Host OS: arch linux
- linkerd version: edge-25.4.4
Possible solution
Letting tap error and not using it in the dashboard.
Additional context
No response
Would you like to work on fixing this bug?
maybe