8000 Use secrets in operators by IyeOnline · Pull Request #5197 · tenzir/tenzir · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Use secrets in operators #5197

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 27 commits into from
Jun 30, 2025
Merged

Use secrets in operators #5197

merged 27 commits into from
Jun 30, 2025

Conversation

IyeOnline
Copy link
Contributor
@IyeOnline IyeOnline commented May 14, 2025 •
edited
Loading

This is the followup to #5065, actually making use of secrets in operators.

Outstanding operators/connectors:

@raxyte:

  • from_http
  • http (transform)
  • from_opensearch (nothing to be done)
  • from_velociraptor (nothing to be done)
  • AMQP (load/save)

@IyeOnline:

  • ABS (load/save)
  • CURL-based (http&ftp load/save)
  • save_email
  • google_cloud_pub_sub (load/save) (nothing to be done)
  • GCS (load/save)
  • S3 (load/save)
  • Kafka (options record may contain secrets)
  • from_file
  • fluent-bit (load/save)
  • to_asl
  • to_azure_log_analytics
  • to_clickhouse
  • to_google_secops
  • to_google_cloud_logging
  • to_opensearch
  • to_snowflake
  • to_splunk (done in the initial PR as a test)

Operators not currently considered:

  • to_hive (postponed because of upcomming rewrite)
  • load_file/save_file (local path)
  • SQS (queue name)
  • TCP/UDP (local endpoint)
  • ZMQ (local endpoint)

@IyeOnline IyeOnline added the connector Loader and saver label May 15, 2025
@IyeOnline IyeOnline force-pushed the topic/secrets branch 2 times, most recently from ed24e41 to 6709515 Compare May 19, 2025 10:22
@IyeOnline IyeOnline force-pushed the topic/secrets-in-operators branch from 8868984 to 6d62550 Compare May 20, 2025 15:22
@IyeOnline IyeOnline force-pushed the topic/secrets-in-operators branch from 6d62550 to 7046d7e Compare May 20, 2025 18:46
IyeOnline added a commit that referenced this pull request May 21, 2025
@IyeOnline
Disable new secret feature by default
8b3b9a4 
Since we merge secrets without any support in operators, we enable
`tenzir.legacy-secret-model` by default for now. This also reverts/hides
all docs and changelog entries, which will be in
#5197
@IyeOnline IyeOnline force-pushed the topic/secrets-in-operators branch from 7046d7e to 859c64d Compare May 21, 2025 08:14
@IyeOnline IyeOnline mentioned this pull request May 21, 2025
Merged
Base automatically changed from topic/secrets to main May 21, 2025 10:26
IyeOnline added a commit that referenced this pull request May 21, 2025
@IyeOnline
Add first class secret type (#5065)
68655ef 
This adds a first class `secret` type to the TQL language that can
"store" secrets.

- See tenzir/issues#2781

The feature is currently disabled by default, to be re-enabled
#5197
@IyeOnline IyeOnline force-pushed the topic/secrets-in-operators branch from 7d037b7 to 32a0298 Compare June 3, 2025 09:09
@github-actions GitHub Actions
Copy link
github-actions bot commented Jun 3, 2025 •
edited
Loading

📚 Documentation Preview

🧹 Preview deployment has been cleaned up

The documentation preview for this PR has been removed since the PR was closed.

@IyeOnline IyeOnline force-pushed the topic/secrets-in-operators branch from 2cf3c39 to eb7d4f0 Compare June 3, 2025 16:01
@raxyte raxyte force-pushed the topic/secrets-in-operators branch from eb7d4f0 to 4a492a0 Compare June 4, 2025 10:00
@IyeOnline IyeOnline force-pushed the topic/secrets-in-operators branch 10 times, most recently from 8a36350 to edf2377 Compare June 10, 2025 12:04
@raxyte raxyte force-pushed the topic/secrets-in-operators branch 2 times, most recently from c6c55bc to 481520b Compare June 10, 2025 14:24
@IyeOnline IyeOnline force-pushed the topic/secrets-in-operators branch 2 times, most recently from dab2787 to 943ae99 Compare June 11, 2025 11:31
@raxyte raxyte force-pushed the topic/secrets-in-operators branch from 943ae99 to 944e0d2 Compare June 11, 2025 12:49
IyeOnline and others added 12 commits June 26, 2025 17:42
@IyeOnline
Adapt load_gcs and save_gcs to use secrets
345d360
@IyeOnline
Adapt load_s3 and save_s3 to use secrets
1ac0bb8
@IyeOnline
Adapt load_azure_blob_storage to use secrets
c8f3cc6 
Includes `save_azure_blob_storage`
@raxyte @IyeOnline
Adapt {load,save}_amqp to use secrets
4f53937
@IyeOnline
Adapt from_file to use secrets
148b046
@IyeOnline
Adapt {from,to}_fluent_bit to use secrets
eb0aaef
@raxyte @IyeOnline
Adapt from_http to use secrets
657ae37
@raxyte @IyeOnline
Adapt http operator to use secrets
684de1f
@IyeOnline
Implement format expressions for secrets
69c6661 
A format expression containing a `secret` will yield a `secret`.
@IyeOnline
Integrate review feedback for secrets-in-operators
2bd57e3
@IyeOnline
Refactor secret censoring
d991e03 
Secret censoring now happens in the execution node's diagnostic handler.
This has the advantage, that we cannot forget it. In order to not be
overly aggressive, it now only censors the full values of managed secrets,
ignoring incomplete parts.
@IyeOnline
Refactor secret resolution model, take 3
ea35c22 
This now enforces yielding and ensures the error cases are safe by not
re-scheduling if a callback fails.
@IyeOnline IyeOnline force-pushed the topic/secrets-in-operators branch 6 times, most recently from c34b0d6 to 7ce3dbb Compare June 27, 2025 13:45
jachris
jachris approved these changes Jun 27, 2025
View reviewed changes
jachris
jachris reviewed Jun 27, 2025
View reviewed changes
IyeOnline added 3 commits June 27, 2025 18:12
@IyeOnline
Integrate feedback for secrets-in-operators, take 2
a5da8c5
@IyeOnline
Adapt python operator to use secrets
19b3743 
This also removes the TQL1 `python` operator.
@IyeOnline
Adapt shell to use secrets
34a92dd 
This also removes the TQL1 `shell` and `apply` operators and ports tests
that used these to TQL2.
@IyeOnline IyeOnline force-pushed the topic/secrets-in-operators branch from 7ce3dbb to 34a92dd Compare June 27, 2025 16:13
jachris
jachris reviewed Jun 27, 2025
View reviewed changes
@IyeOnline IyeOnline enabled auto-merge June 30, 2025 11:36
@IyeOnline IyeOnline merged commit 9938a14 into main Jun 30, 2025
60 checks passed
@IyeOnline IyeOnline deleted the topic/secrets-in-operators branch June 30, 2025 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dominiklohmann dominiklohmann dominiklohmann left review comments

@jachris jachris jachris approved these changes

Copilot code review Copilot Copilot left review comments

Assignees
No one assigned
Labels
connector Loader and saver
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@IyeOnline @dominiklohmann @jachris @raxyte
0