8000 EOWC: close `RANGE`/`SESSION` window with watermark · Issue #17119 · risingwavelabs/risingwave · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

EOWC: close RANGE/SESSION window with watermark #17119

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
stdrc opened this issue Jun 5, 2024 · 0 comments
Open

EOWC: close RANGE/SESSION window with watermark #17119

stdrc opened this issue Jun 5, 2024 · 0 comments
Assignees
@stdrc
Labels
A-eowc Area: Emit on window close. type/enhancement Type: Enhancement for an existing feature.

Comments

@stdrc
Copy link
Member
stdrc commented Jun 5, 2024 •
edited
Loading

Example:

=> create table t (
    tm timestamp,
    foo int,
    bar int,
    watermark for tm as tm - interval '5 minutes'
) append only;

=> create materialized view mv as
select
    *,
    max(foo) over (partition by bar order by tm range between current row and '10 minute' following) as m
from t
emit on window close;

=> insert into t values
  ('2023-05-06 16:51:00', 1, 100)
, ('2023-05-06 16:56:00', 8, 100)
, ('2023-05-06 17:30:00', 3, 200)
, ('2023-05-06 17:35:00', 5, 100)
, ('2023-05-06 17:59:00', 4, 100)
, ('2023-05-06 18:01:00', 6, 200)
;

=> select * from mv;
         tm          | foo | bar | m 
---------------------+-----+-----+---
 2023-05-06 16:51:00 |   1 | 100 | 8
 2023-05-06 16:56:00 |   8 | 100 | 8
(2 rows)

In the above example, when 17:59:00 and 18:01:00 are inserted, these two rows are not visible to OverWindow. What are visible to OverWindow are the first 4 rows. Now for 17:30:00 and 17:35:00 (belonging to partition 200 and 100 respectively), they don't know if there'll be future rows that are in their corresponding range window (e.g. [17:30:00, 17:40:00]), so they can't be outputted. However in fact, because of the existence of 18:01:00, OverWindow should've received the 17:56:00 watermark, indicating that there'll be no row before that in the future. So in fact the 17:30:00 and 17:35:00 can be outputted very safely.

This problem doesn't affect the correctness of the result. All rows will finally output correctly if the stream is healthy and all concerned partitions are being actively updated. But in the case that some partitions are seldomly updated, the result can be delayed for an indefinite long time.
@stdrc stdrc added the type/feature Type: New feature. label Jun 5, 2024
@stdrc stdrc changed the title eowc: close RANGE/SESSION window with watermark EOWC: close RANGE/SESSION window with watermark Jun 5, 2024
@github-actions github-actions bot added this to the release-1.10 milestone Jun 5, 2024
@stdrc stdrc self-assigned this Jun 5, 2024
@stdrc stdrc modified the milestones: release-1.10, release-1.11 Jul 10, 2024
@stdrc stdrc removed this from the release-2.0 milestone Aug 22, 2024
@stdrc stdrc added A-eowc Area: Emit on window close. type/enhancement Type: Enhancement for an existing feature. and removed type/feature Type: New feature. labels May 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stdrc stdrc

Labels
A-eowc Area: Emit on window close. type/enhancement Type: Enhancement for an existing feature.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@stdrc
0