enhancement(aws_s3 sink): Add ability to configure request errors to retry #23206

jchap-pnnl · 2025-06-13T22:07:06Z

Summary

Adds a new field, a retry_logic struct, to S3SinkConfig that allows user to specify types of response errors to retry. Users can specify specific status codes of error responses of failed requests they want to be automatically retried, or they can specify all failed requests to be automatically retried.

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
No

How did you test this PR?

Ran existing integration tests and verified they passed.
Manually tested to verify the new functionality. Generated 403 errors by changing the access_key_id to an invalid one. Verified the failed authentication service was retried when the configuration required it to and didn't when the configuration didn't require it to. Ran tests with different values for retry_logic, retry_all_errors, and errors_to_retry:
- retry_logic: omitted |
  - retry_all_errors: true | false | omitted
  - errors_to_retry: [] | [403] | omitted

Used this configuration:

# vector.yaml
sources:
  generate_syslog:
    type: "demo_logs"
    format: "syslog"
    count: 10

transforms:
  remap_syslog:
    inputs:
      - "generate_syslog"
    type: "remap"
    source: |
      structured = parse_syslog!(.message)
      . = merge(., structured)

sinks:
  s3:
    type: aws_s3
    encoding:
      codec: "json"
    region: <actual region>
    inputs:
      - remap_syslog
    bucket: <bucket name>
    auth:
      access_key_id: <key id value>
      secret_access_key: <key value>

    retry_logic:
      retry_all_errors: true
      errors_to_retry: [403]

Considered adding a new test to src/sinks/aws_s3/integration_tests.rs, but didn't see a good option for detecting service retries. Considered a test using a timer that could measure how long a successful request takes and determine if retries are happening if the failed request takes longer. We rejected this option because the results could vary depending on the load of system resources or other factors. But please let us know if you recommend a testing approach that would work.

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the "no-changelog" label to this PR.

Notes

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
The CI checks run only after we manually approve them.
- We recommend adding a pre-push hook, please see this template.
- Alternatively, we recommend running the following locally before pushing to the remote branch:
  - cargo fmt --all
  - cargo clippy --workspace --all-targets -- -D warnings
  - cargo nextest run --workspace (alternatively, you can run cargo test --all)
  - ./scripts/check_changelog_fragments.sh
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run cargo vdev build licenses to regenerate the license inventory and commit the changes (if any). More details here.

Closes: Consider expanding the cases where Vector retries requests #10870

…l errors. Added retry_all_errors to S3SinkConfig and S3RetryLogic structs. Setting retry_all_errors to the default value in the generate_config function. Added self.retry_all_errors to the condition in the is_retriable_error function. (vectordotdev#10870)

…. Changed retry_all_errors to Option<bool>.

… Added configured_to_retry and check_response functions to s3_common/config.rs. Added configured_to_retry call to is_retriable_error result in RetryLogic. (vectordotdev#10870)

…nd replaced them with retry_logic in S3SinkConfig. (vectordotdev#10870)

…dotdev#10870)

…0870)

…ectordotdev#10870)

…ogic rather than unwrap_or. (vectordotdev#10870)

bits-bot · 2025-06-13T22:07:11Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

Chapman, Joe B seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

website/cue/reference/components/sinks/base/aws_s3.cue

…_logic trait in S3SinkConfig. (vectordotdev#10870)

thomasqueirozb · 2025-06-17T19:02:40Z

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "vector"
-version = "0.48.0"


Please don't bump the vector version, we have our own release process

Hmm, I'm not sure what to do about this. I bumped up the version because without it, running cargo vdev check version gives me this error:

Is there another way to resolve that error without changing the vector version locally?

cargo vdev check version is not a required CI check AFAIK, you can ignore this and revert the changes to Cargo.* files

make check-version is recommended to be included in the pre-push hooks. I can skip that check if that's the best way to avoid the version requirement error.

I've reverted the change.

thomasqueirozb · 2025-06-17T19:05:22Z

src/sinks/s3_common/config.rs

+    /// A vector list of status codes matching specific error types that trigger failed
+    /// service retry attempts. The list is ignored if `retry_all_errors` is true.


Suggested change

/// A vector list of status codes matching specific error types that trigger failed

/// service retry attempts. The list is ignored if `retry_all_errors` is true.

/// A list of HTTP status codes matching specific error types that trigger failed

/// service retry attempts. This list is ignored if `retry_all_errors` is true.

…umentation comment for errors_to_retry. (vectordotdev#10870)

Chapman, Joe B added 18 commits May 15, 2025 15:13

Added errors_to_retry field to S3SinkConfig as an Option<Vec<String>>…

bff7cef

…. Changed retry_all_errors to Option<bool>.

Changed errors_to_retry from Option<Vec<String>> to Option<Vec<u16>>.…

0a13b5a

… Added configured_to_retry and check_response functions to s3_common/config.rs. Added configured_to_retry call to is_retriable_error result in RetryLogic. (vectordotdev#10870)

Moved retry_all_errors and errors_to_retry into S3RetryLogic struct a…

55cea19

…nd replaced them with retry_logic in S3SinkConfig. (vectordotdev#10870)

Added retry_logic to s3_flush_on_exhaustion integration test. (vector…

d57043f

…dotdev#10870)

Merge branch 'master' into feature/failed-response-retry

fa7da2f

Fixed format and build issues in s3_common/config.rs. (vectordotdev#1…

545f8de

…0870)

Fixed panic error if retry_logic is None in sinks/aws_s3/config.rs. (v…

8c02620

…ectordotdev#10870)

Merge branch 'master' into feature/failed-response-retry

09b9121

Changed build_processor function to call unwrap_or_default on retry_l…

8f85195

…ogic rather than unwrap_or. (vectordotdev#10870)

Fixed formatting issue in build_processor function. (vectordotdev#10870)

8219fbd

Updated docs for retry_logic. (vectordotdev#10870)

05b2981

Added changelog fragment. (vectordotdev#10870)

eae0ee2

Fixed documentation note for S3RetryLogic. (vectordotdev#10870)

f25505a

Updated component docs. (vectordotdev#10870)

3f4072a

Updated the feature authors list for the changelog. (vectordotdev#10870)

17fa04d

Merge branch 'master' into feature/failed-response-retry

fe540c9

Increased the version in Cargo.toml. (vectordotdev#10870)

99e3db4

jchap-pnnl requested review from a team as code owners June 13, 2025 22:07

github-actions bot added domain: sinks Anything related to the Vector's sinks domain: external docs Anything related to Vector's external, public documentation labels Jun 13, 2025

urseberry reviewed Jun 16, 2025

View reviewed changes

website/cue/reference/components/sinks/base/aws_s3.cue Outdated Show resolved Hide resolved

website/cue/reference/components/sinks/base/aws_s3.cue Outdated Show resolved Hide resolved

website/cue/reference/components/sinks/base/aws_s3.cue Outdated Show resolved Hide resolved

Chapman, Joe B added 2 commits June 17, 2025 09:42

Merge branch 'master' into feature/failed-response-retry

c428202

Adjusted wording in documentation comments for S3RetryLogic and retry…

372715c

…_logic trait in S3SinkConfig. (vectordotdev#10870)

thomasqueirozb requested changes Jun 17, 2025

View reviewed changes

thomasqueirozb added the meta: awaiting author Pull requests that are awaiting their author. label Jun 17, 2025

Chapman, Joe B added 2 commits June 17, 2025 15:30

Reverted vector version change in Cargo.toml. Adjusted wording in doc…

18163e0

…umentation comment for errors_to_retry. (vectordotdev#10870)

Updated autogenerated documentation for aws_s3. (vectordotdev#10870)

9e775eb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

enhancement(aws_s3 sink): Add ability to configure request errors to retry #23206

enhancement(aws_s3 sink): Add ability to configure request errors to retry #23206

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

		/// A vector list of status codes matching specific error types that trigger failed
		/// service retry attempts. The list is ignored if `retry_all_errors` is true.

enhancement(aws_s3 sink): Add ability to configure request errors to retry #23206

Are you sure you want to change the base?

enhancement(aws_s3 sink): Add ability to configure request errors to retry #23206

Conversation

Uh oh!

Summary

Change Type

Is this a breaking change?

How did you test this PR?

Does this PR include user facing changes?

Notes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!