Discussion: Use ORDER BY as part of MV's distribution key #19321
Comments
|
An example to show what the dist key looks like now on main: create table test_dist (v1 int, v2 int, v3 int, primary key (v1, v2));
create table test_dist2 (v1 int, v2 int, v3 int, primary key (v1, v2));
create table test_dist3 (v1 int, v2 int, v3 int, primary key (v1, v2));
describe test_dist;
-- primary key v1, v2
-- distribution key v1, v2
create materialized view test_dist_mv as select * from test_dist order by v1, v3;
describe test_dist_mv;
-- primary key v1, v3, v2
-- distribution key v1, v2
drop MATERIALIZED view test_join_dist;
create MATERIALIZED view test_join_dist as
select t1.v3 as a, t2.v3 as b, t3.v3 as c
from test_dist as t1
join test_dist2 as t2
on t1.v1 = t2.v1
join test_dist3 as t3
on t2.v1 = t3.v1;
describe test_join_dist;
-- primary key test_dist.v1, test_dist.v2, test_dist2.v2, test_dist3.v2, test_dist2.v1
-- distribution test_dist.v1, test_dist.v2, test_dist2.v2, test_dist3.v2, test_dist2.v1
create MATERIALIZED view test_join_dist2 as
select t1.v3 as a, t2.v3 as b, t3.v3 as c
from test_dist as t1
join test_dist2 as t2
on t1.v1 = t2.v1
join test_dist3 as t3
on t2.v1 = t3.v1
order by t1.v3, t1.v2;
describe test_join_dist2;
-- primary key test_dist.v3, test_dist.v2, test_dist.v1, test_dist2.v2, test_dist3.v2, test_dist2.v1
-- distribution key test_dist2.v1
ideally the dist key for test_dist and dist_join_dist2 can be: describe test_dist_mv;
-- primary key v1, v3, v2
-- distribution key v1, v3, ...
describe test_join_dist2;
-- primary key test_dist.v3, test_dist.v2, test_dist.v1, test_dist2.v2, test_dist3.v2, test_dist2.v1
-- distribution key test_dist.v3, test_dist.v2, ... |
|
This issue has been open for 60 days with no activity. If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the You can also confidently close this issue as not planned to keep our backlog clean. |
|
I reproduced a bug from this: #21668. I think ORDER BY can be part of MV's distribution key. But it must contain the full stream key as well to ensure updates don't become dis-ordered if we re-shuffle back to original distribution. Consider the following case of StreamTableScan after 27efddf: The dist is the user order key. It gets shuffled to different actors initially, and the update on ... and then shuffled back to the same actor with: So we get: Where B should be after C, and we encounter a double-insert error. |
If we include the stream key, like: I think the locality will also be affected with the presence of stream key. So we cannot achieve the original objective of having data distributed purely by order key. It can help avoid data skew however? |
|
We'd better check the dist key for index also, to see if we have the same problem. |
I think this is not accurate. The fundamental rule is that, records with same stream key value must be shuffled to the same partition, thus, have the same distribution key value. Following this...
So in the case of #21668, if we enrich the stream key to also include column It appears that the goal of this issue is correct ("as part of"), but the implementation in PR #20176 is incorrect ("as"). |
That works |
|
But this issue description is not accurate as a result:
Rather than MV pk, it needs to be subset of MV |
Doesn't this just mean that MV stream key should be equal to MV pk, since MV pk will be |
Uh oh!
There was an error while loading. Please reload this page.
context: https://risingwave-labs.slack.com/archives/C034TRPKN1F/p1731041466238309?thread_ts=1731032250.800469&cid=C034TRPKN1F
stream_keyas MV's distribution key #12824distribued by <column_list>syntax as well, allowing users to specify the distribution column manually"Some nodes:
dist_keyjust need to be subset ofpk, not prefix.pkisorder_by | stream_keydist_keyisupstream_dist_key(subset ofstream_key). And we changed to wholestream_key.distributed by. Since index is essentially the same as MV, there seem to be no reason not to also support it for MVThe text was updated successfully, but these errors were encountered: