Open
Description
I am trying to use a workload identity to authenticate to my Azure storage account.
I am able to reach the storage account when using AZBLOB_ACCOUNT_KEY, but not with AZBLOB_USE_MANAGED_IDENTITY and a service account to inject the authentication information via AAD Workload Identity webhook.
The storage account API returns de following error: Authentication information is not given in the correct format. Check the value of Authorization
Environment variables are correctly set by the workload identity webhook:
$ env | grep AZURE
AZURE_TENANT_ID=***
AZURE_FEDERATED_TOKEN_FILE=/var/run/secrets/azure/tokens/azure-identity-token
AZURE_AUTHORITY_HOST=https://login.microsoftonline.com/
AZURE_CLIENT_ID=***
Here is the azblob part of my config:
azblob:
endpoint_schema: https
endpoint_suffix: core.windows.net
account_name: ***
account_key: ""
sas: ""
use_managed_identity: true
container: clickhouse-backup-client-alpha
assume_container_exists: true
path: ""
object_disk_path: ""
compression_level: 1
compression_format: tar
sse_key: ""
buffer_count: 3
max_parts_count: 256
timeout: 4h
debug: true
The error log:
2025-03-25 17:17:42.170 WRN pkg/storage/general.go:252 > BackupList bd.Walk return error: -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/azure-storage-blob-go@v0.15.0/azblob/zc_storage_er
===== RESPONSE ERROR (ServiceCode=) =====
Description=Authentication information is not given in the correct format. Check the value of Authorization header.
RequestId:31a0acfe-101e-000c-1ca9-9d5a67000000
Time:2025-03-25T17:17:42.1598077Z, Details:
Code: InvalidAuthenticationInfo
GET https://***.blob.core.windows.net/clickhouse-backup-client-alpha?comp=list&delimiter=%2F&restype=container&timeout=14401
Authorization: REDACTED
User-Agent: [Azure-Storage/0.15 (go1.24.1; linux)]
X-Ms-Client-Request-Id: [64bc33df-9f04-4e6f-4ae6-82a6e55edda0]
X-Ms-Version: [2020-10-02]
--------------------------------------------------------------------------------
RESPONSE Status: 400 Authentication information is not given in the correct format. Check the value of Authorization header.
Content-Length: [298]
Content-Type: [application/xml]
Date: [Tue, 25 Mar 2025 17:17:41 GMT]
Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
X-Ms-Request-Id: [31a0acfe-101e-000c-1ca9-9d5a67000000]
And the debug stacktrace:
goroutine 222 [running]:
github.com/Azure/azure-storage-blob-go/azblob.stack()
github.com/Azure/azure-storage-blob-go@v0.15.0/azblob/zc_policy_request_log.go:158 +0x5e
github.com/Azure/azure-storage-blob-go/azblob.NewPipeline.NewRequestLogPolicyFactory.func3.1({0x269f610, 0xc0004778f0}, {0x0?})
github.com/Azure/azure-storage-blob-go@v0.15.0/azblob/zc_policy_request_log.go:108 +0x61d
github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0x224f100?, {0x269f610?, 0xc0004778f0?}, {0xd?})
github.com/Azure/azure-pipeline-go@v0.2.3/pipeline/core.go:43 +0x29
github.com/Azure/azure-storage-blob-go/azblob.(*tokenCredentialWithRefresh).New.(*tokenCredential).New.func1({0x269f610, 0xc0004778f0}, {0x0?})
github.com/Azure/azure-storage-blob-go@v0.15.0/azblob/zc_credential_token.go:144 +0x15b
github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0x269f5a0?, {0x269f610?, 0xc0004778f0?}, {0xc0002ddb18?})
github.com/Azure/azure-pipeline-go@v0.2.3/pipeline/core.go:43 +0x29
github.com/Azure/azure-storage-blob-go/azblob.NewPipeline.NewRetryPolicyFactory.func2.1({0x269f5a0, 0xc00044c0a0}, {0xc0000598a8?})
github.com/Azure/azure-storage-blob-go@v0.15.0/azblob/zc_policy_retry.go:204 +0x8c7
github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0xc0006f1d40?, {0x269f5a0?, 0xc00044c0a0?}, {0xc0007a8c90?})
github.com/Azure/azure-pipeline-go@v0.2.3/pipeline/core.go:43 +0x29
github.com/Azure/azure-storage-blob-go/azblob.NewPipeline.NewUniqueRequestIDPolicyFactory.func1.1({0x269f5a0, 0xc00044c0a0}, {0xc0006c4701?})
github.com/Azure/azure-storage-blob-go@v0.15.0/azblob/zc_policy_unique_request_id.go:22 +0xde
github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0x2138200?, {0x269f5a0?, 0xc00044c0a0?}, {0xa?})
github.com/Azure/azure-pipeline-go@v0.2.3/pipeline/core.go:43 +0x29
github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1({0x269f5a0, 0xc00044c0a0}, {0xc0006c6128?})
github.com/Azure/azure-storage-blob-go@v0.15.0/azblob/zc_policy_telemetry.go:34 +0x11e
github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0xc0008d1180?, {0x269f5a0?, 0xc00044c0a0?}, {0x1?})
github.com/Azure/azure-pipeline-go@v0.2.3/pipeline/core.go:43 +0x29
github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do(0xc0007a8b70?, {0x269f5a0, 0xc00044c0a0}, {0x2673980?, 0xc0003a6c60?}, {0xc0007a8b78?})
github.com/Azure/azure-pipeline-go@v0.2.3/pipeline/core.go:129 +0x4a
github.com/Azure/azure-storage-blob-go/azblob.containerClient.ListBlobHierarchySegment({{{{0xc0007a8b70, 0x5}, {0x0, 0x0}, 0x0, {0xc0007a8b78, 0x28}, {0xc0004b67c0, 0x1f}, {0x0, ...}, ...}, ...}}, ...)
github.com/Azure/azure-storage-blob-go@v0.15.0/azblob/zz_generated_container.go:693 +0x46b
github.com/Azure/azure-storage-blob-go/azblob.ContainerURL.ListBlobsHierarchySegment({{{{{...}, {...}, 0x0, {...}, {...}, {...}, 0x0, 0x0, {...}, {...}, ...}, ...}}}, ...)
github.com/Azure/azure-storage-blob-go@v0.15.0/azblob/url_container.go:251 +0x165
github.com/Altinity/clickhouse-backup/v2/pkg/storage.(*AzureBlob).WalkAbsolute(0xc0007be7e0, {0x269f5a0, 0xc00044c0a0}, {0x26db118, 0x1}, 0x0, 0xc0008d11c0)
github.com/Altinity/clickhouse-backup/v2/pkg/storage/azblob.go:271 +0x314
github.com/Altinity/clickhouse-backup/v2/pkg/storage.(*AzureBlob).Walk(0xc0007be7e0, {0x269f5a0, 0xc00044c0a0}, {0x26633b8?, 0x476c19?}, 0x0, 0xc0008d11c0)
github.com/Altinity/clickhouse-backup/v2/pkg/storage/azblob.go:251 +0xa5
github.com/Altinity/clickhouse-backup/v2/pkg/storage.(*BackupDestination).BackupList(0xc0005640f0, {0x269f5a0, 0xc00044c0a0}, 0x1, {0x0, 0x0})
github.com/Altinity/clickhouse-backup/v2/pkg/storage/general.go:176 +0x2ae
github.com/Altinity/clickhouse-backup/v2/pkg/backup.(*Backuper).GetRemoteBackups(0xc000476150?, {0x269f5a0, 0xc00044c0a0}, 0x1)
github.com/Altinity/clickhouse-backup/v2/pkg/backup/list.go:365 +0x2bf
github.com/Altinity/clickhouse-backup/v2/pkg/server.(*APIServer).httpListHandler(0xc000346000, {0x269d350, 0xc0007be000}, 0xc000450140)
github.com/Altinity/clickhouse-backup/v2/pkg/server/server.go:836 +0x40f
net/http.HandlerFunc.ServeHTTP(0xc000476230?, {0x269d350?, 0xc0007be000?}, 0xc0004f99e8?)
net/http/server.go:2294 +0x29
github.com/Altinity/clickhouse-backup/v2/pkg/server.(*APIServer).basicAuthMiddleware-fm.(*APIServer).basicAuthMiddleware.func1({0x269d350, 0xc0007be000}, 0xc000450140)
github.com/Altinity/clickhouse-backup/v2/pkg/server/server.go:298 +0x569
net/http.HandlerFunc.ServeHTTP(0xc000450000?, {0x269d350?, 0xc0007be000?}, 0x0?)
net/http/server.go:2294 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00015afc0, {0x269d350, 0xc0007be000}, 0xc0001ca500)
github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2
net/http.serverHandler.ServeHTTP({0xc0000d00f0?}, {0x269d350?, 0xc0007be000?}, 0x1?)
net/http/server.go:3301 +0x8e
net/http.(*conn).serve(0xc00058c360, {0x269f568, 0xc0000d0720})
net/http/server.go:2102 +0x625
created by net/http.(*Server).Serve in goroutine 129
net/http/server.go:3454 +0x485
Metadata
Metadata
Assignees
Labels
No labels