8000 AWS SigV4 canonicalization does not handle pre-encoded URL components correctly · Issue #16927 · curl/curl · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

AWS SigV4 canonicalization does not handle pre-encoded URL components correctly #16927

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
jhartCoalfire opened this issue Apr 2, 2025 · 6 comments
Open

AWS SigV4 canonicalization does not handle pre-encoded URL components correctly #16927

jhartCoalfire opened this issue Apr 2, 2025 · 6 comments
Labels
authentication help wanted HTTP

Comments

@jhartCoalfire
Copy link
jhartCoalfire commented Apr 2, 2025 •
edited
Loading

I did this

I'm observing that when a URL contains already-encoded characters (e.g. %3A), the canonical request generated by cURL's AWS SigV4 signing (in lib/http_aws_sigv4.c, particularly in the canon_string() function) does not match the expected canonical request, leading to signature mismatches.

For example, when a URL containing %3A is used, the canonical string produced seems to miss the inclusion of the x-amz-security-token header in the signed headers, resulting in a signature error.

I have compared this behavior with awscurl, which handles the encoding correctly. I suspect that the logic in canon_string() (lines XYZ–ABC in lib/vauth/aws_sigv4.c) is not correctly normalizing already-encoded characters.

Could you please review this section and consider a fix so that pre-encoded URL components are handled consistently?

I expected the following

curl "https://s3tables.us-east-1.amazonaws.com:443/tables/arn%3Aaws%3As3tables%3Aus-east-1%3A022954301426%3Abucket%2Fjasoehartablebucket/jasoeharnamespace/jasoehartable/encryption" \
                                              --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
                                              -H "x-amz-security-token: $AWS_SESSION_TOKEN" \
                                              --aws-sigv4 "aws:amz:us-east-1:s3tables" \
                                              -H 'Accept-Encoding: gzip, deflate, br' \
                                              -H 'User-Agent: agent'

Running the above command fails to handle the %3A encoded characters already present in the URL. If I un-encode the URL, the request fails to the service. Encoding it however fails signing.

The error coming back:

{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The Canonical String for this request should have been
'GET
/tables/arn%253Aaws%253As3tables%253Aus-east-1%253A022954301426%253Abucket%252Fjasoehartablebucket/jasoeharnamespace/jasoehartable/encryption

So it looks like it is encoding the '%' again and failing to recognize the '%3A' as a valid hex digit

Example with single quote and unencoded characters

curl 'https://s3tables.us-east-1.amazonaws.com:443/tables/arn:aws:s3tables:us-east-1:022954301426:bucket/jasoehartablebucket/table/0e65f8e0-0d5c-4791-8010-120e0ebf8bfc/jasoeharnamespace/jasoehartable/encryption' \
            --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
            -H "x-amz-security-token: $AWS_SESSION_TOKEN" \
            --aws-sigv4 "aws:amz:us-east-1:s3tables" \
            -H 'Accept-Encoding: gzip, deflate, br' \
            -H 'User-Agent: agent'

{"message":"Unable to determine service/operation name to be authorized"}

curl/libcurl version

curl 8.13

operating system

Ubuntu 22.04
@bagder
Copy link
Member
bagder commented Apr 3, 2025

I spent 10 minutes just now trying to find out how colons should be treated in this weirdo system but I failed. I don't know the correct way.

The canon_string function encodes colons as %3A so it should get the same signature independent of source format. I think.

@jhartCoalfire
Copy link
Author

If there is any further information I can provide, let me know. This was discovered while I was creating a burp suite extension to transform APIs to curl with sigv4 signing and observed the above behavior when testing.

@bagder bagder mentioned this issue Apr 4, 2025
Closed
@bagder
Copy link
Member
bagder commented Apr 4, 2025

We need to find the detailed explanation/spec for how the canonicalization of that part of the URL is meant to work for sigv4. I couldn't find it.

Then we need to make sure the code does what the spec says. Presumably by someone using adjusted code against a live server - I personally don't have anything I can test this against.

@jay
Copy link
Member
jay commented Apr 4, 2025

@outscale-mgo

@nbaws
Copy link
Contributor
nbaws commented Apr 4, 2025

I can fix this one if you like.

@bagder
Copy link
Member
bagder commented Apr 4, 2025

I can fix this one if you like.

That would be totally awesome! 🥳
6BF3

@nbaws nbaws mentioned this issue Apr 22, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
authentication help wanted HTTP
Milestone
No milestone
Development

No branches or pull requests
4 participants
@bagder @jay @nbaws @jhartCoalfire
0