Oci transport refinement #11711

souleb · 2023-01-10T22:21:48Z

What this PR does / why we need it:
If implemented, users will be able to use custom certificates and CA when interacting with OCI registries.

This is convenient for user interacting with registries that hold custom signed certificates. By providing the CA the server certificate can be validate during handshakes.

This also enable interacting with insecure registries.

The following flags can be used when using the helm cli with OCI Registries:

cert-file, identify registry client using this SSL certificate file
key-file, identify registry client using this SSL key file
ca-file, verify certificates of HTTPS-enabled servers using this CA bundle
insecure-skip-tls-verify, skip tls certificate checks

Special notes for your reviewer:
For the oras client, we use a functional option to provide a custom httpClient that can then be passed to authorizer and resolver. This is flexible, and sdk users can construct their client with any needed custom transport.

We tested this by setting up a docker registry in an ec2 server with a certificate and CA generated with openSSL.

Regression test have been manually performed for the following helm cmds:

helm push
helm pull
helm show
helm template
helm install
helm upgrade

If applicable:

this PR contains documentation
this PR contains unit tests
this PR has been tested for backwards compatibility

cc @sabre1041

keskad · 2023-01-11T10:01:17Z

I have tested manually this branch at commit 5c40996dad38d70060b0df0e8d9674f5f8acb268 using following commands:

helm pull --untar --untardir sample-app/charts oci://xxx.yyy.zzz/sample-test/charts/sample-app sample-app --version 1.0.0 --password=zzzzzzzzzzzzz --username=robot$cccc --ca-file /opt/ca/xyz.crt --insecure-skip-tls-verify

helm pull --untar --untardir sample-app/charts oci://xxx.yyy.zzz/sample-test/charts/sample-app sample-app --version 1.0.0 --password=zzzzzzzzzzzzz --username=robot$cccc --ca-file /opt/ca/xyz.crt

helm pull --untar --untardir sample-app/charts oci://xxx.yyy.zzz/sample-test/charts/sample-app sample-app --version 1.0.0 --password=zzzzzzzzzzzzz --username=robot$cccc --insecure-skip-tls-verify

Part of the output shows that it connects to the server properly (TLS works):

DEBU[0000] fetch response received                       host=xxx.yyy.zzz response.header.connection=keep-alive response.header.content-length=98 response.header.content-type="application/json; charset=utf-8" response.header.date="Wed, 11 Jan 2023 09:40:28 GMT" response.header.server=nginx response.header.set-cookie="sid=xxxxxxxxx; Path=/; HttpOnly" response.header.x-request-id=xxxxx response.status="404 Not Found" url="https://xxx.yyy.zzz/v2/sample-test/charts/sample-app/manifests/1.0.0"

sabre1041 · 2023-01-12T05:42:58Z

@keskad Were your sample commands successful even though you may have received a 404 response within the underlying ORAS client library code?

TerryHowe · 2023-02-09T17:44:35Z

Is this PR going anywhere?

TerryHowe · 2023-02-09T18:17:53Z

We've been building and running off this branch for over a month to take advantage of ability to have a root CA for private registries passed in for the helm install. Our customers use Harbor and other registries to mirror charts and images for performance reasons and to support air gapped environments.

TerryHowe · 2023-02-10T01:48:31Z

make test-coverage succeeds on this branch for me. Is that circleci problem ephemeral?

keskad · 2023-02-10T07:57:54Z

@sabre1041 Sorry, I will today test it :)

souleb · 2023-02-10T10:13:30Z

make test-coverage succeeds on this branch for me. Is that circleci problem ephemeral?

The circleci problem is due to the TCP/IP implementation I believe. Hexadecimal representation for an IP address is sometime not accepted. We use an hex loopback address for testing TLS. We do this hack because the docker client fallback to http for localhost, but does not take into account non decimal representation of the address.

This work just fine with github actions, and I understood that @sabre1041 is migrating all tests there.

Is this PR going anywhere?

We have tested this with @sabre1041. As this is a consequent change, I believe 2 maintainers review and approval are necessary.

scottrigby · 2023-02-13T23:51:27Z

Nice work @souleb and @sabre1041 🤌 🚛

Thanks @keskad, for testing.

@TerryHowe, thanks for joining the last Helm dev meeting, giving your results, and commenting about them. Glad it's been working! Let's get it in.

Dotting i's and crossing t's, I ran the CI commands in this config file locally from the point of failure in CircleCI. Note the last step deploy is skipped on PRs). Both:

Locally on MacOS

In Docker cimg/go:1.18 image. Here's what I tried with Docker. Someone want to help verify this?

$ gh checkout pr 11711
# For DOCKER_ env variables, see Keybase
$ docker run --rm -it \
    -e GOCACHE="/tmp/go/cache" \
    -e GOLANGCI_LINT_VERSION="1.46.2" \
    -e DOCKER_USER=**** \
    -e DOCKER_PASS=**** \
    -v $(pwd):/home/circleci/helm.sh/helm -w /home/circleci/helm.sh/helm cimg/go:1.18

`make test-coverage`

✅ Succeeded locally on MacOS
❌ Failed when run from within Docker cimg/go:1.18 image. This may be due to what @souleb said about the TCP/IP implementation in Oci transport refinement #11711 (comment).

Click to expand output

127.0.0.1 - - [13/Feb/2023:23:18:13 +0000] "GET /v2/testrepo/local-subchart/tags/list HTTP/1.1" 200 52 "" "Helm/3.10"
time="2023-02-13T23:18:13.7588425Z" level=warning msg="error parsing level \"\": not a valid logrus Level: \"\", using \"info\"\t"
time="2023-02-13T23:18:13.760661Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version=go1.18.10
time="2023-02-13T23:18:13.7610851Z" level=info msg="redis not configured" go.version=go1.18.10
time="2023-02-13T23:18:13.7609037Z" level=info msg="Starting upload purge in 10m0s" go.version=go1.18.10
time="2023-02-13T23:18:13.7636833Z" level=info msg="restricting TLS version to tls1.2 or higher" go.version=go1.18.10
time="2023-02-13T23:18:13.7637402Z" level=info msg="restricting TLS cipher suites to: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_AES_128_GCM_SHA256,TLS_CHACHA20_POLY1305_SHA256,TLS_AES_256_GCM_SHA384" go.version=go1.18.10
time="2023-02-13T23:18:13.7681008Z" level=info msg="listening on [::]:33967, tls" go.version=go1.18.10
time="2023-02-13T23:18:13.8152698Z" level=info msg="Error logging in to endpoint, trying next endpoint" error="Get \"https://0x7f000001:33967/v2/\": dial tcp: lookup 0x7f000001 on 192.168.65.5:53: no such host"
time="2023-02-13T23:18:13.8415987Z" level=info msg="Error logging in to endpoint, trying next endpoint" error="Get \"https://0x7f000001:33967/v2/\": dial tcp: lookup 0x7f000001 on 192.168.65.5:53: no such host"
time="2023-02-13T23:18:13.8652873Z" level=info msg="Error logging in to endpoint, trying next endpoint" error="Get \"https://0x7f000001:33967/v2/\": dial tcp: lookup 0x7f000001 on 192.168.65.5:53: no such host"
--- FAIL: TestTLSRegistryClientTestSuite (0.31s)
    --- FAIL: TestTLSRegistryClientTestSuite/Test_0_Login (0.10s)
        client_tls_test.go:51: 
            	Error Trace:	/home/circleci/helm.sh/helm/pkg/registry/client_tls_test.go:51
            	Error:      	Expected nil, but got: &url.Error{Op:"Get", URL:"https://0x7f000001:33967/v2/", Err:(*net.OpError)(0xc0007b4370)}
            	Test:       	TestTLSRegistryClientTestSuite/Test_0_Login
            	Messages:   	no error logging into registry with good credentials
        client_tls_test.go:56: 
            	Error Trace:	/home/circleci/helm.sh/helm/pkg/registry/client_tls_test.go:56
            	Error:      	Expected nil, but got: &url.Error{Op:"Get", URL:"https://0x7f000001:33967/v2/", Err:(*net.OpError)(0xc000886370)}
            	Test:       	TestTLSRegistryClientTestSuite/Test_0_Login
            	Messages:   	no error logging into registry with good credentials, insecure mode
    --- FAIL: TestTLSRegistryClientTestSuite/Test_1_Push (0.05s)
        utils_test.go:215: 
            	Error Trace:	/home/circleci/helm.sh/helm/pkg/registry/utils_test.go:215
            	            				/home/circleci/helm.sh/helm/pkg/registry/client_tls_test.go:60
            	Error:      	Expected nil, but got: &fmt.wrapError{msg:"failed to do request: Head \"https://0x7f000001:33967/v2/testrepo/boop/blobs/sha256:0140f0a2f6b9a56b5feb46431c23f8906369faaa16fc64e2c7f0c462072099ad\": dial tcp: lookup 0x7f000001 on 192.168.65.5:53: no such host", err:(*url.Error)(0xc000818000)}
            	Test:       	TestTLSRegistryClientTestSuite/Test_1_Push
            	Messages:   	no error pushing non-strict ref (bad basename), with strict mode disabled
        utils_test.go:224: 
            	Error Trace:	/home/circleci/helm.sh/helm/pkg/registry/utils_test.go:224
            	            				/home/circleci/helm.sh/helm/pkg/registry/client_tls_test.go:60
            	Error:      	Expected nil, but got: &fmt.wrapError{msg:"failed to do request: Head \"https://0x7f000001:33967/v2/testrepo/examplechart/blobs/sha256:43f83b0de8f2b93e7d8441434e4d950731cc661ed710dd8b1846b907dbb7a4b3\": dial tcp: lookup 0x7f000001 on 192.168.65.5:53: no such host", err:(*url.Error)(0xc000818000)}
            	Test:       	TestTLSRegistryClientTestSuite/Test_1_Push
            	Messages:   	no error pushing non-strict ref (bad tag), with strict mode 
8000
disabled
        utils_test.go:233: 
            	Error Trace:	/home/circleci/helm.sh/helm/pkg/registry/utils_test.go:233
            	            				/home/circleci/helm.sh/helm/pkg/registry/client_tls_test.go:60
            	Error:      	Expected nil, but got: &fmt.wrapError{msg:"failed to do request: Head \"https://0x7f000001:33967/v2/testrepo/local-subchart/blobs/sha256:d2ab7310924fa816445fa7dec0a829b005ece8592219e9714c9572ec47847c9a\": dial tcp: lookup 0x7f000001 on 192.168.65.5:53: no such host", err:(*url.Error)(0xc000818090)}
            	Test:       	TestTLSRegistryClientTestSuite/Test_1_Push
            	Messages:   	no error pushing good ref
        utils_test.go:236: 
            	Error Trace:	/home/circleci/helm.sh/helm/pkg/registry/utils_test.go:236
            	            				/home/circleci/helm.sh/helm/pkg/registry/client_tls_test.go:60
            	Error:      	Expected nil, but got: &fmt.wrapError{msg:"failed to do request: Head \"https://0x7f000001:33967/v2/testrepo/local-subchart/manifests/0.1.0\": dial tcp: lookup 0x7f000001 on 192.168.65.5:53: no such host", err:(*url.Error)(0xc0008f09c0)}
            	Test:       	TestTLSRegistryClientTestSuite/Test_1_Push
            	Messages:   	no error pulling a simple chart
        utils_test.go:251: 
            	Error Trace:	/home/circleci/helm.sh/helm/pkg/registry/utils_test.go:251
            	            				/home/circleci/helm.sh/helm/pkg/registry/client_tls_test.go:60
            	Error:      	Expected nil, but got: &fmt.wrapError{msg:"failed to do request: Head \"https://0x7f000001:33967/v2/testrepo/signtest/blobs/sha256:b0a02b7412f78ae93324d48df8fcc316d8482e5ad7827b5b238657a29a22f256\": dial tcp: lookup 0x7f000001 on 192.168.65.5:53: no such host", err:(*url.Error)(0xc0004baa80)}
            	Test:       	TestTLSRegistryClientTestSuite/Test_1_Push
            	Messages:   	no error pushing good ref with prov
        utils_test.go:254: 
            	Error Trace:	/home/circleci/helm.sh/helm/pkg/registry/utils_test.go:254
            	            				/home/circleci/helm.sh/helm/pkg/registry/client_tls_test.go:60
            	Error:      	Expected nil, but got: &fmt.wrapError{msg:"failed to do request: Head \"https://0x7f000001:33967/v2/testrepo/signtest/manifests/0.1.0\": dial tcp: lookup 0x7f000001 on 192.168.65.5:53: no such host", err:(*url.Error)(0xc0004bb3e0)}
            	Test:       	TestTLSRegistryClientTestSuite/Test_1_Push
            	Messages:   	no error pulling a simple chart
        suite.go:77: test panicked: runtime error: invalid memory address or nil pointer dereference
            goroutine 319 [running]:
            runtime/debug.Stack()
            	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
            github.com/stretchr/testify/suite.failOnPanic(0xc00051a9c0, {0xc14d20, 0x1337ea0})
            	/home/circleci/go/pkg/mod/github.com/stretchr/testify@v1.8.0/suite/suite.go:77 +0x3b
            github.com/stretchr/testify/suite.Run.func1.1()
            	/home/circleci/go/pkg/mod/github.com/stretchr/testify@v1.8.0/suite/suite.go:161 +0x252
            panic({0xc14d20, 0x1337ea0})
            	/usr/local/go/src/runtime/panic.go:838 +0x207
            helm.sh/helm/v3/pkg/registry.testPush(0xc0000e1600)
            	/home/circleci/helm.sh/helm/pkg/registry/utils_test.go:259 +0xcbd
            helm.sh/helm/v3/pkg/registry.(*TLSRegistryClientTestSuite).Test_1_Push(0x0?)
            	/home/circleci/helm.sh/helm/pkg/registry/client_tls_test.go:60 +0x1b
            reflect.Value.call({0xc0000adbc0?, 0xc000300f40?, 0x13?}, {0xcfacb4, 0x4}, {0xc000393e70, 0x1, 0x1?})
            	/usr/local/go/src/reflect/value.go:556 +0x845
            reflect.Value.Call({0xc0000adbc0?, 0xc000300f40?, 0xc0000e1600?}, {0xc000055e70, 0x1, 0x1})
            	/usr/local/go/src/reflect/value.go:339 +0xbf
            github.com/stretchr/testify/suite.Run.func1(0xc00051a9c0)
            	/home/circleci/go/pkg/mod/github.com/stretchr/testify@v1.8.0/suite/suite.go:175 +0x4b6
            testing.tRunner(0xc00051a9c0, 0xc00001bb00)
            	/usr/local/go/src/testing/testing.go:1439 +0x102
            created by testing.(*T).Run
            	/usr/local/go/src/testing/testing.go:1486 +0x35f
    --- FAIL: TestTLSRegistryClientTestSuite/Test_2_Pull (0.04s)
        utils_test.go:295: 
            	Error Trace:	/home/circleci/helm.sh/helm/pkg/registry/utils_test.go:295
            	            				/home/circleci/helm.sh/helm/pkg/registry/client_tls_test.go:64
            	Error:      	Expected nil, but got: &fmt.wrapError{msg:"failed to do request: Head \"https://0x7f000001:33967/v2/testrepo/local-subchart/manifests/0.1.0\": dial tcp: lookup 0x7f000001 on 192.168.65.5:53: no such host", err:(*url.Error)(0xc000819f20)}
            	Test:       	TestTLSRegistryClientTestSuite/Test_2_Pull
            	Messages:   	no error pulling a simple chart
        utils_test.go:305: 
            	Error Trace:	/home/circleci/helm.sh/helm/pkg/registry/utils_test.go:305
            	            				/home/circleci/helm.sh/helm/pkg/registry/client_tls_test.go:64
            	Error:      	Expected nil, but got: &fmt.wrapError{msg:"failed to do request: Head \"https://0x7f000001:33967/v2/testrepo/local-subchart/manifests/0.1.0\": dial tcp: lookup 0x7f000001 on 192.168.65.5:53: no such host", err:(*url.Error)(0xc000318990)}
            	Test:       	TestTLSRegistryClientTestSuite/Test_2_Pull
            	Messages:   	no error pulling a chart with prov when no prov exists, ignoring missing
        utils_test.go:327: 
            	Error Trace:	/home/circleci/helm.sh/helm/pkg/registry/utils_test.go:327
            	            				/home/circleci/helm.sh/helm/pkg/registry/client_tls_test.go:64
            	Error:      	Expected nil, but got: &fmt.wrapError{msg:"failed to do request: Head \"https://0x7f000001:33967/v2/testrepo/signtest/manifests/0.1.0\": dial tcp: lookup 0x7f000001 on 192.168.65.5:53: no such host", err:(*url.Error)(0xc000766570)}
            	Test:       	TestTLSRegistryClientTestSuite/Test_2_Pull
            	Messages:   	no error pulling a chart with prov
        suite.go:77: test panicked: runtime error: invalid memory address or nil pointer dereference
            goroutine 397 [running]:
            runtime/debug.Stack()
            	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
            github.com/stretchr/testify/suite.failOnPanic(0xc0004a31e0, {0xc14d20, 0x1337ea0})
            	/home/circleci/go/pkg/mod/github.com/stretchr/testify@v1.8.0/suite/suite.go:77 +0x3b
            github.com/stretchr/testify/suite.Run.func1.1()
            	/home/circleci/go/pkg/mod/github.com/stretchr/testify@v1.8.0/suite/suite.go:161 +0x252
            panic({0xc14d20, 0x1337ea0})
            	/usr/local/go/src/runtime/panic.go:838 +0x207
            helm.sh/helm/v3/pkg/registry.testPull(0xc0000e1600)
            	/home/circleci/helm.sh/helm/pkg/registry/utils_test.go:332 +0x967
            helm.sh/helm/v3/pkg/registry.(*TLSRegistryClientTestSuite).Test_2_Pull(0x0?)
            	/home/circleci/helm.sh/helm/pkg/registry/client_tls_test.go:64 +0x1b
            reflect.Value.call({0xc0000adbc0?, 0xc000300f58?, 0x13?}, {0xcfacb4, 0x4}, {0xc000392e70, 0x1, 0x1?})
            	/usr/local/go/src/reflect/value.go:556 +0x845
            reflect.Value.Call({0xc0000adbc0?, 0xc000300f58?, 0xc0000e1600?}, {0xc000220670, 0x1, 0x1})
            	/usr/local/go/src/reflect/value.go:339 +0xbf
            github.com/stretchr/testify/suite.Run.func1(0xc0004a31e0)
            	/home/circleci/go/pkg/mod/github.com/stretchr/testify@v1.8.0/suite/suite.go:175 +0x4b6
            testing.tRunner(0xc0004a31e0, 0xc00001bc20)
            	/usr/local/go/src/testing/testing.go:1439 +0x102
            created by testing.(*T).Run
            	/usr/local/go/src/testing/testing.go:1486 +0x35f
    --- FAIL: TestTLSRegistryClientTestSuite/Test_3_Tags (0.01s)
        utils_test.go:369: 
            	Error Trace:	/home/circleci/helm.sh/helm/pkg/registry/utils_test.go:369
            	            				/home/circleci/helm.sh/helm/pkg/registry/client_tls_test.go:68
            	Error:      	Expected nil, but got: &url.Error{Op:"Get", URL:"https://0x7f000001:33967/v2/testrepo/local-subchart/tags/list", Err:(*net.OpError)(0xc0007b40f0)}
            	Test:       	TestTLSRegistryClientTestSuite/Test_3_Tags
            	Messages:   	no error retrieving tags
        utils_test.go:370: 
            	Error Trace:	/home/circleci/helm.sh/helm/pkg/registry/utils_test.go:370
            	            				/home/circleci/helm.sh/helm/pkg/registry/client_tls_test.go:68
            	Error:      	Not equal: 
            	            	expected: 1
            	            	actual  : 0
            	Test:       	TestTLSRegistryClientTestSuite/Test_3_Tags
    --- FAIL: TestTLSRegistryClientTestSuite/Test_4_Logout (0.00s)
        client_tls_test.go:76: 
            	Error Trace:	/home/circleci/helm.sh/helm/pkg/registry/client_tls_test.go:76
            	Error:      	Expected nil, but got: &errors.errorString{s:"not logged in"}
            	Test:       	TestTLSRegistryClientTestSuite/Test_4_Logout
            	Messages:   	no error logging out of registry
FAIL
coverage: 77.9% of statements
FAIL	helm.sh/helm/v3/pkg/registry	3.680s
FAIL
make: *** [Makefile:113: test-coverage] Error 1

`make test build`

❌ Failed locally on MacOS. Guessing this is due to my golangci-lint version. Can someone please verify this?
❌ Failed when run from within Docker cimg/go:1.18 image

Click to expand output

FAIL
make: *** [Makefile:107: test-unit] Error 1
circleci@55606cd12c82:~/helm.sh/helm$ exit

keskad · 2023-02-24T08:54:49Z

Sorry for late reply.

I have checked it and I can login & push & pull using binary built from oci-transport-refinement branch.
On my side I can confirm the --insecure-skip-tls-verify works, --ca-file does not work for me (it may be my fault - I don't know, the same cert works with curl --cacert ... https://mydomain.org).

sabre1041 · 2023-03-03T03:58:36Z

@souleb #11676 was merged. Would you be able to rebase so that we can get the build to pass and get closer to integrating this in?

If implemented, users will be able to use custom certificates and CA to while interacting with OCI registries. Signed-off-by: Soule BA <bah.soule@gmail.com>

Signed-off-by: Andrew Block <andy.block@gmail.com>

If enabled the registryClient is set using a helper that accepts the TLS flags. This keeps the client creation consistent accross the different commands. Signed-off-by: Soule BA <bah.soule@gmail.com>

sabre1041 · 2023-03-03T14:10:57Z

finally green! @scottrigby @mattfarina any chance you could take a look at this?

TerryHowe · 2023-03-06T21:30:53Z

Scott's comment 3 weeks ago about make test build failing on Mac, I saw the same thing, but it was a go-lint problem with 1.20.

TerryHowe · 2023-03-06T21:37:03Z

As to Damian's comment about the ca-file option, works for me:

tlhowe@3c22fb41240e bin % ./helm pull oci://195.17.xxx.xxx:30003/l0g8r8j6/eks-anywhere-packages --version 0.0.0-e0b458ff6381c03251680a50e7d8ad8dbdf42136 --ca-file ../../eks-anywhere/rootCA.pem
Pulled: 195.17.xxx.xxx:30003/l0g8r8j6/eks-anywhere-packages:0.0.0-e0b458ff6381c03251680a50e7d8ad8dbdf42136
Digest: sha256:01061bad03817a3911533a29d8e75b5832c18153baba23065eae668dc40ade1d
tlhowe@3c22fb41240e bin % ls *z
eks-anywhere-packages-0.0.0-e0b458ff6381c03251680a50e7d8ad8dbdf42136.tgz

Running Harbor

TerryHowe · 2023-03-10T02:03:27Z

login

% helm registry login --username $REGISTRY_USERNAME --password $REGISTRY_PASSWORD $REGISTRY --ca-file ./rootCA.pem 
WARNING: Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded
% helm registry login --username $REGISTRY_USERNAME --password $REGISTRY_PASSWORD $REGISTRY --insecure            
WARNING: Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded

Push

% helm push --ca-file ./rootCA.pem foo-0.1.0.tgz oci://${REGISTRY}/foo/foo        
Pushed: 195.17.xxx.xxx:30003/foo/foo/foo:0.1.0
Digest: sha256:ec454fceae2fb04431b0830885d24b2074092da647b47c36147f87687fb278b7
% helm push --insecure-skip-tls-verify foo-0.1.0.tgz oci://${REGISTRY}/foo/foo  
Pushed: 195.17.xxx.xxx:30003/foo/foo/foo:0.1.0
Digest: sha256:ec454fceae2fb04431b0830885d24b2074092da647b47c36147f87687fb278b7
% helm push foo-0.1.0.tgz oci://${REGISTRY}/foo/foo  
Error: failed to do request: Head "https://195.17.xxx.xxx:30003/v2/foo/foo/foo/blobs/sha256:af9c03c64b1d3d1c76a09d75faf3fd1c8b9ccc97ce0d31e464bccf68fe236674": x509: “tlhowe@3c22fb41240e.ant.amazon.com (Howe)” certificate is not trusted

Show

% helm show chart --insecure-skip-tls-verify  oci://${REGISTRY}/foo/foo/foo 
Pulled: 195.17.xxx.xxx:30003/foo/foo/foo:0.1.0
Digest: sha256:ec454fceae2fb04431b0830885d24b2074092da647b47c36147f87687fb278b7
apiVersion: v2
appVersion: 1.16.0
description: A Helm chart for Kubernetes
name: foo
type: application
version: 0.1.0

% helm show chart --ca-file ./rootCA.pem  oci://${REGISTRY}/foo/foo/foo 
Pulled: 195.17.xxx.xxx:30003/foo/foo/foo:0.1.0
Digest: sha256:ec454fceae2fb04431b0830885d24b2074092da647b47c36147f87687fb278b7
apiVersion: v2
appVersion: 1.16.0
description: A Helm chart for Kubernetes
name: foo
type: application
version: 0.1.0

% helm show chart   oci://${REGISTRY}/foo/foo/foo 
Error: Get "https://195.17.xxx.xxx:30003/v2/foo/foo/foo/tags/list": x509: “tlhowe@3c22fb41240e.ant.amazon.com (Howe)” certificate is not trusted

Install

% helm install --generate-name --ca-file ./rootCA.pem oci://195.17.xxx.xxx:30003/foo/foo/foo
Pulled: 195.17.xxx.xxx:30003/foo/foo/foo:0.1.0
Digest: sha256:ec454fceae2fb04431b0830885d24b2074092da647b47c36147f87687fb278b7
NAME: foo-1678413424
LAST DEPLOYED: Thu Mar  9 18:57:07 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=foo,app.kubernetes.io/instance=foo-1678413424" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT

% helm install --generate-name --insecure-skip-tls-verify oci://${REGISTRY}/foo/foo/foo
Pulled: 195.17.xxx.xxx:30003/foo/foo/foo:0.1.0
Digest: sha256:ec454fceae2fb04431b0830885d24b2074092da647b47c36147f87687fb278b7
NAME: foo-1678413517
LAST DEPLOYED: Thu Mar  9 18:58:39 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=foo,app.kubernetes.io/instance=foo-1678413517" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT

% helm install --generate-name oci://${REGISTRY}/foo/foo/foo
Error: INSTALLATION FAILED: Get "https://195.17.xxx.xxx:30003/v2/foo/foo/foo/tags/list": x509: “sammy@3c22fb41240e.ant.amazon.com (Howe)” certificate is not trusted

scottrigby

Nice work Soulé! And Andy! For this combined PR with both of your work in it. I reviewed the code again. The tests added look great. All tests pass now too. 👏

Thanks, @TerryHowe, for the end-user feedback from AWS and for manually testing these as well! #11711 (comment). I have manually tested a lot of this now too and it passes. (also Thanks @keskad for your prior testing in earlier iterations).

Approving ✅ We'll need another maintainer who didn't work on this PR to approve this one as well before merging.

scottrigby · 2023-03-14T05:17:43Z

Thanks @jdolitsky! 💖 We're good to merge. This should be included in 3.12.0.

dreamerkr · 2023-04-11T13:08:17Z

Title: Push to OCI insecure registry err
code brach: main
command:
./helm push --debug --insecure-skip-tls-verify influxdb-4.3.2.tgz oci://192.168.254.17:5000/helm-charts
err message:
WARN[0000] reference for unknown type: application/vnd.cncf.helm.chart.content.v1.tar+gzip
DEBU[0000] checking and pushing to url="https://192.168.254.17:5000/v2/helm-charts/influxdb/blobs/sha256:876b8b7611c8e60d0bd25bccedbe311fa66471b17ac9b09c9d5cd8fb15cb0558"
DEBU[0000] do request request.header.accept="application/vnd.cncf.helm.chart.content.v1.tar+gzip, /" request.header.user-agent=Helm/3.11 request.method=HEAD url="https://192.168.254.17:5000/v2/helm-charts/influxdb/blobs/sha256:876b8b7611c8e60d0bd25bccedbe311fa66471b17ac9b09c9d5cd8fb15cb0558"
WARN[0000] reference for unknown type: application/vnd.cncf.helm.config.v1+json
DEBU[0000] checking and pushing to url="https://192.168.254.17:5000/v2/helm-charts/influxdb/blobs/sha256:10e6ddc66ebd9db8394bf8818e7fe863e3f1d7ebdb98e14717793128315fcac6"
DEBU[0000] do request request.header.accept="application/vnd.cncf.helm.config.v1+json, /" request.header.user-agent=Helm/3.11 request.method=HEAD url="https://192.168.254.17:5000/v2/helm-charts/influxdb/blobs/sha256:10e6ddc66ebd9db8394bf8818e7fe863e3f1d7ebdb98e14717793128315fcac6"
Error: failed to do request: Head "https://192.168.254.17:5000/v2/helm-charts/influxdb/blobs/sha256:876b8b7611c8e60d0bd25bccedbe311fa66471b17ac9b09c9d5cd8fb15cb0558": http: server gave HTTP response to HTTPS client
helm.go:84: [debug] failed to do request: Head "https://192.168.254.17:5000/v2/helm-charts/influxdb/blobs/sha256:876b8b7611c8e60d0bd25bccedbe311fa66471b17ac9b09c9d5cd8fb15cb0558": http: server gave HTTP response to HTTPS client

rstribrn · 2023-07-26T10:32:57Z

Does not work with "helm dep update":

helm version
version.BuildInfo{Version:"v3.12.1", GitCommit:"f32a527a060157990e2aa86bf45010dfb3cc8b8d", GitTreeState:"clean", GoVersion:"go1.20.4"}

helm dep update

Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "xxx-releases" chart repository
...Successfully got an update from the "xxx-snaphots" chart repository
Update Complete. ⎈Happy Helming!⎈
Error: could not retrieve list of tags for repository oci://xxx/bitnamicharts: Get "https://xxx/v2/bitnamicharts/redis-cluster/tags/list": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "xxx_TLS_CA_20180205")

This also does not work (gives same result):
helm dependency update --kube-insecure-skip-tls-verify

And of course:
helm dependency update --insecure-skip-tls-verify
Error: unknown flag: --insecure-skip-tls-verify

(BTW: Same behaviour with "helm dep build")

pull-request-size bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jan 10, 2023

scottrigby added this to the 3.12.0 milestone Feb 13, 2023

scottrigby mentioned this pull request Feb 14, 2023

Convert remaining CircleCI config to GitHub Actions #11820

Closed

1 task

hiddeco mentioned this pull request Feb 15, 2023

Unable to specify CA for Helm OCI repository fluxcd/flux2#3417

Closed

1 task

souleb and others added 6 commits March 3, 2023 07:33

Enable custom certificates option for OCI

b0ecb21

If implemented, users will be able to use custom certificates and CA to while interacting with OCI registries. Signed-off-by: Soule BA <bah.soule@gmail.com>

Added support for insecure OCI registries

08593c8

Signed-off-by: Andrew Block <andy.block@gmail.com>

Added insecure option to login subcommand

154f37e

Signed-off-by: Andrew Block <andy.block@gmail.com>

Removed conditional

ec5e29e

Signed-off-by: Andrew Block <andy.block@gmail.com>

Reimplemented change in httpgetter for insecure TLS option

c94306f

Signed-off-by: Andrew Block <andy.block@gmail.com>

Provide a helper to set the registryClient in cmd

11738dd

If enabled the registryClient is set using a helper that accepts the TLS flags. This keeps the client creation consistent accross the different commands. Signed-off-by: Soule BA <bah.soule@gmail.com>

sabre1041 force-pushed the oci-transport-refinement branch from 5c40996 to 11738dd Compare March 3, 2023 13:33

scottrigby approved these changes Mar 10, 2023

View reviewed changes

jdolitsky approved these changes Mar 13, 2023

View reviewed changes

scottrigby merged commit d27d1bd into main Mar 14, 2023

souleb mentioned this pull request Mar 14, 2023

Add support for custom certs in HelmRepositories of Type oci fluxcd/source-controller#723

Closed

mattfarina deleted the oci-transport-refinement branch March 14, 2023 15:07

shizhMSFT mentioned this pull request Mar 22, 2023

X509 Authentication with OCI registry oras-project/oras#491

Closed

mattfarina mentioned this pull request Mar 22, 2023

Bump the Go version #11896

Merged

3 tasks

souleb mentioned this pull request Mar 23, 2023

rely on mockdns to resolve 0x7f000001 instead of default resolver #11938

Closed

3 tasks

dreamerkr mentioned this pull request Apr 11, 2023

Push to OCI insecure registry err #11982

Closed

aryan9600 mentioned this pull request May 17, 2023

[RFC-004] Add .spec.insecure to HelmRepository fluxcd/source-controller#957

Closed

scottrigby mentioned this pull request Jul 21, 2023

oci: Add flag --plain-http to enable working with HTTP registries #12128

Merged

3 tasks

byDimasik mentioned this pull request May 23, 2025

Allow providing CA certificate for helm extensions that use OCI registries k0sproject/k0s#5877

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Oci transport refinement #11711

Oci transport refinement #11711

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Oci transport refinement #11711

Oci transport refinement #11711

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

make test-coverage

make test build

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

`make test-coverage`

`make test build`