8000 tsh authentication handshake failed: tls: failed to verify certificate: x509 · Issue #54336 · gravitational/teleport · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

tsh authentication handshake failed: tls: failed to verify certificate: x509 #54336

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
midnight47 opened this issue Apr 27, 2025 · 0 comments
Closed

tsh authentication handshake failed: tls: failed to verify certificate: x509 #54336

midnight47 opened this issue Apr 27, 2025 · 0 comments

Comments

@midnight47
Copy link

Expected behavior:
normal tsh login

Current behavior:
ERROR: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"

Bug details:
I install helm chart teleport 17.4.5

full installation
helm repo add teleport https://charts.releases.teleport.dev helm repo update kubectl create ns teleport

create self signed certificate:

cat ca_openssl.cnf
`
[ v3_ca ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = teleport.test.local
DNS.2 = *.teleport.test.local
DNS.3 = *.teleport.cluster.local
cat teleport_openssl.cnf
[ req ]
default_bits = 4096
prompt = no
default_md = sha256
distinguished_name = dn
req_extensions = req_ext

[ dn ]
C = RU
ST = YourState
L = YourCity
O = YourOrganization
CN = teleport.test.local

[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = teleport.test.local
DNS.2 = *.teleport.test.local
DNS.3 = *.teleport.cluster.local
`

openssl genrsa -out ca.key 4096 openssl req -x509 -sha256 -new -key ca.key -days 10000 -out ca.crt openssl genrsa -out teleport.key 4096 openssl req -new -key teleport.key -out teleport.csr -config teleport_openssl.cnf openssl x509 -req -in teleport.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out teleport.crt -days 10000 -sha256 -extfile ca_openssl.cnf -extensions v3_ca
create secrets from this cert
`
kubectl -n teleport create secret generic teleport-ca-cert --from-file=ca.pem=./ca.crt

kubectl create secret generic teleport-tls
--from-file=tls.crt=./teleport.crt
--from-file=tls.key=./teleport.key
--from-file=ca.crt=./ca.crt
-n teleport
on my servers and clients I added this certificate as trusted
cp teleport.crt ca.crt /usr/local/share/ca-certificates/
update-ca-certificates
`

that is my teleport values
`
clusterName: teleport.test.local
proxyListenerMode: multiplex

log:
level: DEBUG

teleport:
dataDir: /var/lib/teleport

authService:
enabled: true
storage:
type: dir

proxyService:
enabled: true
publicAddr: teleport.test.local:443
extraVolumes:
- name: teleport-ca
secret:
secretName: teleport-ca-cert
extraVolumeMounts:
- name: teleport-ca
mountPath: /etc/ssl/certs
readOnly: true
env:
- name: SSL_CERT_FILE
value: /etc/ssl/certs/ca.pem

kubeService:
enabled: true
kubeClusterName: "cluster.local"

sshService:
enabled: false

service:
type: ClusterIP

persistence:
enabled: true
accessModes:
- ReadWriteMany
size: 2Gi
storageClassName: nfs-client

ingress:
enabled: true
spec:
ingressClassName: nginx
rules:
- host: teleport.test.local
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: teleport-cluster
port:
number: 443

annotations:
ingress:
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

tls:
existingSecretName: teleport-tls
existingCASecretName: teleport-ca-cert

auth:
tls:
enabled: true
certFile: /etc/teleport/certs/tls.crt
keyFile: /etc/teleport/certs/tls.key
caFile: /etc/teleport/certs/ca.crt
extraVolumes:
- name: teleport-tls
secret:
secretName: teleport-tls
extraVolumeMounts:
- name: teleport-tls
mountPath: /etc/teleport/certs
readOnly: true
`

I use ingress-controller with this values:
`
controller:
kind: Deployment
replicaCount: 2
resources:
limits:
cpu: 1000m
memory: 1Gi
requests:
cpu: 100m
memory: 256Mi
ingressClassResource:
default: false
enabled: true
name: "nginx"
metrics:
enabled: true
external:
enabled: true
annotations:
config:
disable-ipv6: "true"
disable-ipv6-dns: "true"
enable-access-log-for-default-backend: "false"
http2-max-field-size: 8k
keep-alive: "65"
large-client-header-buffers: 16 64k
limit-conn-status-code: "429"
limit-req-status-code: "429"
load-balance: ewma
client_max_body_size: 300m
proxy-body-size: 300m
error-log-level: error
log-format-escape-json: "true"
log-format-upstream: '{"bytes_sent": "$bytes_sent", "vhost": "$host", "request_proto": "$server_protocol", "remote_addr": "$remote_addr", "proxy_add_x_forwarded_for": "$proxy_add_x_forwarded_for", "remote_user": "$remote_user", "time_local": "$time_local", "request_method": "$request_method", "request_uri": "$uri", "request_args": "$args", "request" : "$request", "status": "$status", "body_bytes_sent": "$body_bytes_sent", "http_referer": "$http_referer", "http_user_agent": "$http_user_agent", "request_length": "$request_length", "request_time" : "$request_time", "upstream_addr": "$upstream_addr", "upstream_response_length": "$upstream_response_length", "upstream_response_time": "$upstream_response_time", "upstream_status": "$upstream_status", "X-Business-Error": "$upstream_http_x_business_error", "upstream_header_time": "$upstream_header_time", "upstream_connect_time": "$upstream_connect_time","connections_waiting": "$connections_waiting", "connections_active": "connections_active"}'
map-hash-bucket-size: "128"
proxy-next-upstream: error timeout http_500 http_502 http_503 http_504
server-tokens: "false"
ssl-protocols: TLSv1.2 TLSv1.3
ssl-session-cache: "true"
ssl-session-cache-size: 20m
ssl-session-timeout: 30m
use-forwarded-headers: "true"
use-gzip: "true"
use-proxy-protocol: "true"
worker-cpu-affinity: auto
worker-processes: "2"
allow-snippet-annotations: "true"
annotations-risk-level: Critical
extraArgs:
enable-ssl-passthrough: "true"
ingressClassResource:
name: nginx
enabled: true
default: false
controllerValue: "k8s.io/ingress-nginx"

ingressClass: nginx

service:
enabled: true
type: LoadBalancer

admissionWebhooks:
enabled: true

defaultBackend:
enabled: true
I install ingress controller like this
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx
--namespace ingress-nginx --create-namespace
--version 4.12.1
-f values.yaml
`

my installation of teleport:
helm upgrade --install teleport-cluster teleport/teleport-cluster -n teleport -f values.yaml --version 17.4.5
create user
kubectl exec -it -n teleport deploy/teleport-cluster-auth -- tctl users add admin --roles=editor,auditor,access

ok I can login to web interface add OTP for my mobile.

after I decide check connection with tsh utill. I install it:
curl https://goteleport.com/static/install-connect.sh | bash -s 17.4.5

and get command to connect
tsh login --proxy=teleport.test.local:443 --auth=local --user=admin teleport.test.local
I get this error:
`
tsh login --proxy=teleport.test.local:443 --auth=local --user=admin teleport.test.local --debug
2025-04-27T14:11:30.038+06:00 INFO [CLIENT] no host login given. defaulting to root client/api.go:1260
2025-04-27T14:11:30.070+06:00 WARN [CLIENT] [KEY AGENT] Unable to connect to SSH agent on socket "": dial unix: missing address client/api.go:4837
2025-04-27T14:11:30.071+06:00 DEBU [TSH] Pinging the proxy to fetch listening addresses for non-web ports. common/tsh.go:4171
2025-04-27T14:11:30.107+06:00 DEBU [CLIENT] not using loopback pool for remote proxy addr: teleport.test.local:443 client/api.go:4796
2025-04-27T14:11:30.122+06:00 DEBU Attempting request to Proxy web api method:GET host:teleport.test.local:443 path:/webapi/ping/local trace_id:32b54d8257056b89036da979c4107aee span_id:76339a26f9fa21c2 webclient/webclient.go:153
2025-04-27T14:11:30.553+06:00 DEBU ALPN connection upgrade test complete address:teleport.test.local:443 upgrade_required:false trace_id:32b54d8257056b89036da979c4107aee span_id:76339a26f9fa21c2 client/alpn_conn_upgrade.go:96
2025-04-27T14:11:30.553+06:00 DEBU Attempting request to Proxy web api method:GET host:teleport.test.local:443 path:/webapi/find trace_id:32b54d8257056b89036da979c4107aee span_id:36cf96ebfaca1e02 webclient/webclient.go:153
2025-04-27T14:11:30.714+06:00 DEBU [CLIENT] Attempting to login with new software private keys. client/api.go:4011
Enter password for Teleport user admin:
2025-04-27T14:11:39.006+06:00 DEBU [CLIENT] not using loopback pool for remote proxy addr: teleport.test.local:443 client/api.go:4796
2025-04-27T14:11:39.008+06:00 DEBU [CLIENT] HTTPS client init(proxyAddr=teleport.test.local:443, insecure=false, extraHeaders=map[]) client/weblogin.go:553
Enter an OTP code from a device:
2025-04-27T14:11:50.430+06:00 DEBU [KEYAGENT] Deleting obsolete stored keyring with index {ProxyHost:teleport.test.local Username:admin ClusterName:teleport.test.local}. client/keyagent.go:550
2025-04-27T14:11:50.553+06:00 DEBU [KEYSTORE] Adding known host teleport.test.local with proxy teleport.test.local client/trusted_certs_store.go:395
2025-04-27T14:11:50.556+06:00 INFO [KEYAGENT] Loading SSH key for user "admin" and cluster "teleport.test.local". client/keyagent.go:198

ERROR REPORT:
Original Error: trace.aggregate connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
Stack Trace:
github.com/gravitational/teleport/lib/client/api.go:4248 github.com/gravitational/teleport/lib/client.(*TeleportClient).ConnectToRootCluster
github.com/gravitational/teleport/tool/tsh/common/tsh.go:2088 github.com/gravitational/teleport/tool/tsh/common.onLogin
github.com/gravitational/teleport/tool/tsh/common/tsh.go:1517 github.com/gravitational/teleport/tool/tsh/common.Run
github.com/gravitational/teleport/tool/tsh/common/tsh.go:651 github.com/gravitational/teleport/tool/tsh/common.Main
github.com/gravitational/teleport/tool/tsh/main.go:26 main.main
runtime/proc.go:272 runtime.main
runtime/asm_amd64.s:1700 runtime.goexit
User Message: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
in log of auth service I get kubectl logs -f -n teleport deployments/teleport-cluster-auth
2025-04-27T09:01:24.921Z DEBU [AUTH:1] Reconciling autoupdate_agent_rollout pid:7.1 component:rollout-controller rollout/controller.go:126
2025-04-27T09:01:32.945Z INFO emitting audit event event_type:mfa_auth_challenge.create fields:map[challenge_allow_reuse:false challenge_scope:CHALLENGE_SCOPE_LOGIN cluster_name:teleport.test.local code:T1015I ei:0 event:mfa_auth_challenge.create time:2025-04-27T09:01:32.945Z trace.component:audit uid:c8bf01be-b041-43dc-88d2-626ab72f4b6c user:admin user_kind:1] events/emitter.go:287
2025-04-27T09:01:37.402Z INFO emitting audit event event_type:user.login fields:map[addr.remote:10.233.67.223:37136 cluster_name:teleport.test.local code:T1000I ei:0 event:user.login method:local mfa_device:map[mfa_device_name:otp-device mfa_device_type:TOTP mfa_device_uuid:1fefd5e1-d955-456b-8f96-6aefb793d9ff] required_private_key_policy:none success:true time:2025-04-27T09:01:37.403Z trace.component:audit uid:edb976d8-81a7-4981-8925-5008938f87da user:admin user_agent:Go-http-client/1.1 user_origin:1] events/emitter.go:287
2025-04-27T09:01:37.402Z DEBU Generated user key with expiry. allowed_logins:[-teleport-nologin-447ce5d0-9abb-4a66-8e89-180014b3e6a4 -teleport-internal-join] valid_before_unix_ts:1745787697 valid_before:2025-04-27T21:01:37.402Z keygen/keygen.go:159
2025-04-27T09:01:37.403Z DEBU [CA] Generating TLS certificate common_name:admin dns_names:[] key_usage:1 not_after:2025-04-27 21:01:37.402812083 +0000 UTC tlsca/ca.go:1327
2025-04-27T09:01:37.403Z INFO emitting audit event event_type:cert.create fields:map[cert_type:user cluster_name:teleport.test.local code:TC000I ei:0 event:cert.create identity:map[client_ip:10.233.67.223 expires:2025-04-27T21:01:37.402812083Z logins:[-teleport-nologin-447ce5d0-9abb-4a66-8e89-180014b3e6a4 -teleport-internal-join] prev_identity_expires:0001-01-01T00:00:00Z private_key_policy:none roles:[editor auditor access] route_to_cluster:teleport.test.local teleport_cluster:teleport.test.local traits:map[aws_role_arns: azure_identities: db_names: db_roles: db_users: gcp_service_accounts: host_user_gid:[] host_user_uid:[] kubernetes_groups: kubernetes_users: logins: windows_logins:] user:admin] time:2025-04-27T09:01:37.404Z trace.component:audit uid:1d821715-18e7-4823-ae74-342225d63fcb] events/emitter.go:287
`

If I use option '--insecure'

tsh login --proxy=teleport.test.local:443 --auth=local --user=admin teleport.test.local --insecure --debug

`
2025-04-27T15:04:17.510+06:00 INFO [CLIENT] no host login given. defaulting to root client/api.go:1260
2025-04-27T15:04:17.517+06:00 WARN [CLIENT] [KEY AGENT] Unable to connect to SSH agent on socket "": dial unix: missing address client/api.go:4837
2025-04-27T15:04:17.517+06:00 DEBU [TSH] Pinging the proxy to fetch listening addresses for non-web ports. common/tsh.go:4171
2025-04-27T15:04:17.523+06:00 DEBU [CLIENT] not using loopback pool for remote proxy addr: teleport.test.local:443 client/api.go:4796
2025-04-27T15:04:17.535+06:00 DEBU Attempting request to Proxy web api method:GET host:teleport.test.local:443 path:/webapi/ping/local trace_id:ec8c682a2de0d9296d2e7b39132ff632 span_id:d19a30e207170692 webclient/webclient.go:153
2025-04-27T15:04:17.590+06:00 DEBU ALPN connection upgrade test complete address:teleport.test.local:443 upgrade_required:false trace_id:ec8c682a2de0d9296d2e7b39132ff632 span_id:d19a30e207170692 client/alpn_conn_upgrade.go:96
2025-04-27T15:04:17.743+06:00 DEBU Attempting request to Proxy web api method:GET host:teleport.test.local:443 path:/webapi/find trace_id:ec8c682a2de0d9296d2e7b39132ff632 span_id:296d0a001d5fcde8 webclient/webclient.go:153
2025-04-27T15:04:17.805+06:00 DEBU [CLIENT] Attempting to login with new software private keys. client/api.go:4011
Enter password for Teleport user admin:
2025-04-27T15:04:25.026+06:00 DEBU [CLIENT] not using loopback pool for remote proxy addr: teleport.test.local:443 client/api.go:4796
2025-04-27T15:04:25.028+06:00 DEBU [CLIENT] HTTPS client init(proxyAddr=teleport.test.local:443, insecure=true, extraHeaders=map[]) client/weblogin.go:553
WARNING: You are using insecure connection to Teleport proxy https://teleport.test.local:443
Enter an OTP code from a device:
2025-04-27T15:04:35.435+06:00 DEBU [KEYAGENT] Deleting obsolete stored keyring with index {ProxyHost:teleport.test.local Username:admin ClusterName:teleport.test.local}. client/keyagent.go:550
2025-04-27T15:04:35.694+06:00 DEBU [KEYSTORE] Adding known host teleport.test.local with proxy teleport.test.local client/trusted_certs_store.go:395
2025-04-27T15:04:35.697+06:00 INFO [KEYAGENT] Loading SSH key for user "admin" and cluster "teleport.test.local". client/keyagent.go:198

ERROR REPORT:
Original Error: trace.aggregate rpc error: code = Unknown desc = unexpected HTTP status code received from server: 302 (Found); malformed header: missing HTTP content-type
Stack Trace:
github.com/gravitational/teleport/lib/client/api.go:4248 github.com/gravitational/teleport/lib/client.(*TeleportClient).ConnectToRootCluster
github.com/gravitational/teleport/tool/tsh/common/tsh.go:2088 github.com/gravitational/teleport/tool/tsh/common.onLogin
github.com/gravitational/teleport/tool/tsh/common/tsh.go:1517 github.com/gravitational/teleport/tool/tsh/common.Run
github.com/gravitational/teleport/tool/tsh/common/tsh.go:651 github.com/gravitational/teleport/tool/tsh/common.Main
github.com/gravitational/teleport/tool/tsh/main.go:26 main.main
runtime/proc.go:272 runtime.main
runtime/asm_amd64.s:1700 runtime.goexit
User Message: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 302 (Found); malformed header: missing HTTP content-type
`

in auth log I got
2025-04-27T09:04:15.490Z DEBU [AUTH:1] Reconciling autoupdate_agent_rollout pid:7.1 component:rollout-controller rollout/controller.go:126 2025-04-27T09:04:25.179Z INFO emitting audit event event_type:mfa_auth_challenge.create fields:map[challenge_allow_reuse:false challenge_scope:CHALLENGE_SCOPE_LOGIN cluster_name:teleport.test.local code:T1015I ei:0 event:mfa_auth_challenge.create time:2025-04-27T09:04:25.18Z trace.component:audit uid:8ca8f15d-db07-48f6-ade8-c20fd44dce16 user:admin user_kind:1] events/emitter.go:287 2025-04-27T09:04:35.426Z INFO emitting audit event event_type:user.login fields:map[addr.remote:10.233.67.223:41118 cluster_name:teleport.test.local code:T1000I ei:0 event:user.login method:local mfa_device:map[mfa_device_name:otp-device mfa_device_type:TOTP mfa_device_uuid:1fefd5e1-d955-456b-8f96-6aefb793d9ff] required_private_key_policy:none success:true time:2025-04-27T09:04:35.427Z trace.component:audit uid:7999fec9-d79c-4441-9acd-51e85ce51f2b user:admin user_agent:Go-http-client/1.1 user_origin:1] events/emitter.go:287 2025-04-27T09:04:35.427Z DEBU Generated user key with expiry. allowed_logins:[-teleport-nologin-a2108874-4738-4b7e-957d-06787dd4d274 -teleport-internal-join] valid_before_unix_ts:1745787875 valid_before:2025-04-27T21:04:35.427Z keygen/keygen.go:159 2025-04-27T09:04:35.428Z DEBU [CA] Generating TLS certificate common_name:admin dns_names:[] key_usage:1 not_after:2025-04-27 21:04:35.427349031 +0000 UTC tlsca/ca.go:1327 2025-04-27T09:04:35.429Z INFO emitting audit event event_type:cert.create fields:map[cert_type:user cluster_name:teleport.test.local code:TC000I ei:0 event:cert.create identity:map[client_ip:10.233.67.223 expires:2025-04-27T21:04:35.427349031Z logins:[-teleport-nologin-a2108874-4738-4b7e-957d-06787dd4d274 -teleport-internal-join] prev_identity_expires:0001-01-01T00:00:00Z private_key_policy:none roles:[editor auditor access] route_to_cluster:teleport.test.local teleport_cluster:teleport.test.local traits:map[aws_role_arns:<nil> azure_identities:<nil> db_names:<nil> db_roles:<nil> db_users:<nil> gcp_service_accounts:<nil> host_user_gid:[] host_user_uid:[] kubernetes_groups:<nil> kubernetes_users:<nil> logins:<nil> windows_logins:<nil>] user:admin] time:2025-04-27T09:04:35.429Z trace.component:audit uid:6821952a-993e-4def-82d7-4382ac025f9f] events/emitter.go:287 2025-04-27T09:05:11.408Z DEBU [AUTH:1] Reconciling autoupdate_agent_rollout pid:7.1 component:rollout-controller rollout/controller.go:126
in ingress log I got
{"bytes_sent": "206", "vhost": "74656c65706f72742e746573742e6c6f63616c.teleport.cluster.local", "request_proto": "HTTP/2.0", "remote_addr": "10.233.122.192", "proxy_add_x_forwarded_for": "10.233.122.192", "remote_user": "", "time_local": "27/Apr/2025:09:04:35 +0000", "request_method": "POST", "request_uri": "/teleport.trust.v1.TrustService/GetCertAuthorities", "request_args": "", "request" : "POST /teleport.trust.v1.TrustService/GetCertAuthorities HTTP/2.0", "status": "302", "body_bytes_sent": "0", "http_referer": "", "http_user_agent": "tsh/17.4.5 grpc-go/1.68.0", "request_length": "172", "request_time" : "0.013", "upstream_addr": "10.233.79.32:3080", "upstream_response_length": "0", "upstream_response_time": "0.012", "upstream_status": "302", "X-Business-Error": "", "upstream_header_time": "0.012", "upstream_connect_time": "0.011","connections_waiting": "0", "connections_active": "connections_active"}
so because of this ingress log and domain (hash).teleport.cluster.local I added alternative domain when I created self signed certificate

and I created additional ingress
`
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: teleport-cluster-proxy-wildcard
namespace: teleport
annotations:
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
ingressClassName: nginx
rules:
- host: '.teleport.cluster.local'
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: teleport-cluster
port:
number: 443
tls:
- hosts:
- '.teleport.cluster.local'
secretName: teleport-tls

`

also I checked how it work without
tls: - hosts: - '*.teleport.cluster.local' secretName: teleport-tls
nothing change.

my version of k8s = v1.24.0

please help with settings.
@gravitational gravitational locked and limited conversation to collaborators Apr 27, 2025
@stevenGravy stevenGravy converted this issue into discussion #54338 Apr 27, 2025

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@midnight47
0