8000 Support batching in webhook · Issue #22066 · risingwavelabs/risingwave · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Support batching in webhook #22066

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
hasyimibhar opened this issue May 30, 2025 · 1 comment · May be fixed by #22075
Open

Support batching in webhook #22066

hasyimibhar opened this issue May 30, 2025 · 1 comment · May be fixed by #22075
Labels
type/feature Type: New feature.
Milestone
release-2.5

Comments

@hasyimibhar
Copy link
hasyimibhar commented May 30, 2025 •
edited
Loading

Is your feature request related to a problem? Please describe.

Currently, webhook source ingest data per row. We have a use case where we generate events on client side, queue it in memory, and after certain thresholds, flush it downstream in a single batched http call. We do this to minimize roundtrip, as each flush can contain up to hundreds of small events.

Describe the solution you'd like

The easiest way I can think of to support this is by using JSONL, where each row is separated by a newline. One way I can think of is to introduce is as a separate connector called webhook_batched:

CREATE TABLE wbhtable (
  data JSONB
) WITH (
  connector = 'webhook_batched'
) VALIDATE SECRET test_secret AS secure_compare(
  headers->>'x-signature',
  encode(hmac(test_secret, data, 'sha1'), 'hex')
);

Describe alternatives you've considered

The alternative is to modify webhook to support multiple rows. But using JSONL will break backwards compatibility, as it's natural for webhook JSON payload to contain newline.

Additional context

If this idea makes sense, I am willing to contribute.
@hasyimibhar hasyimibhar added the type/feature Type: New feature. label May 30, 2025
@github-actions github-actions bot added this to the release-2.5 milestone May 30, 2025
@hasyimibhar hasyimibhar linked a pull request Jun 2, 2025 that will close this issue
Open
8 tasks
@hasyimibhar
Copy link
Author

PR submitted. Instead of introducing another connector type called webhook_batched, I added this as is_batched config instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/feature Type: New feature.
Projects
None yet
Milestone
release-2.5
Development

Successfully merging a pull request may close this issue.

feat(webhook): add support for webhook batching hasyimibhar/risingwave
1 participant
@hasyimibhar
0