-
Notifications
You must be signed in to change notification settings - Fork 651
feat(sink): enable user-defined primary key for upsert sink #8610
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
7f0bc7b
enable user to specify sink pk
xx01cyx 683e512
dashboard
xx01cyx fbb2d92
fix ut
xx01cyx 909cda3
merge main and resolve conflicts
xx01cyx c621143
rename pk to plan_pk
xx01cyx 624aeb2
enable sink e2e test
xx01cyx c047965
add TODO to reuse conductor instance
xx01cyx 434c808
add comment about sink primary key format
xx01cyx 3319abe
rename sink.mode to type for iceberg sink
xx01cyx 83c9828
fix iceberg sink test
xx01cyx 6fa59cc
fix iceberg sink test
xx01cyx d3a4efa
remove unnecessary code
xx01cyx File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,35 +1,20 @@ | ||
| statement ok | ||
| create table t1 (v1 int, v2 int); | ||
|
|
||
| statement error No primary key for the upsert sink | ||
| create sink s1 from t1 with (connector = 'console'); | ||
|
|
||
| statement ok | ||
| create sink s1 as select v1, v2, _row_id from t1 with (connector = 'console'); | ||
|
|
||
| statement ok | ||
| create table t2 (v1 int, v2 int primary key); | ||
|
|
||
| statement ok | ||
| create sink s2 from t2 with (connector = 'console'); | ||
|
|
||
| statement error No primary key for the upsert sink | ||
| create sink s3 as select avg(v1) from t2 with (connector = 'console'); | ||
| create table t (v1 int, v2 int); | ||
|
|
||
| statement ok | ||
| create sink s3 as select avg(v1) from t2 with (connector = 'console', format = 'append_only', force_append_only = 'true'); | ||
| create sink s1 from t with (connector = 'console'); | ||
|
|
||
| statement ok | ||
| create sink s4 as select avg(v1), v2 from t2 group by v2 with (connector = 'console'); | ||
| create sink s2 as select avg(v1), v2 from t group by v2 with (connector = 'console'); | ||
|
|
||
| statement error The sink cannot be append-only | ||
| create sink s5 from t2 with (connector = 'console', format = 'append_only'); | ||
| create sink s3 from t with (connector = 'console', type = 'append-only'); | ||
|
|
||
| statement ok | ||
| create sink s5 from t2 with (connector = 'console', format = 'append_only', force_append_only = 'true'); | ||
| create sink s3 from t with (connector = 'console', type = 'append-only', force_append_only = 'true'); | ||
|
|
||
| statement error Cannot force the sink to be append-only | ||
| create sink s6 from t2 with (connector = 'console', format = 'upsert', force_append_only = 'true'); | ||
| create sink s4 from t with (connector = 'console', type = 'upsert', force_append_only = 'true'); | ||
|
|
||
| statement ok | ||
| drop sink s1 | ||
|
|
@@ -41,13 +26,4 @@ statement ok | |
| drop sink s3 | ||
|
|
||
| statement ok | ||
| drop sink s4 | ||
|
|
||
| statement ok | ||
| drop sink s5 | ||
|
|
||
| statement ok | ||
| drop table t1 | ||
|
|
||
| statement ok | ||
| drop table t2 | ||
| drop table t | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure whether we should use
primary.keyorprimary_key?primary.keyseems more consistent with other configuration.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO
a.bimplies thatbis a subcategory (or sub-configuration) undera, so I preferprimary_key. Remember we also have aforce_append_onlyoption. cc. @tabVersion @st1page @fuyufjh