Update pgcopydb sentinel in the main follow process. #521 8000
Conversation
To avoid infinite looping when endpos has been reached it's important to update our endpos value to the sentinel's one even in the main follow process. In passing, review some error messages. Also refrain from stopping early from reading data from a PIPE when a terminating signal is received, we should finish reading as per the comments in the code.
When the ENDPOS internal message is received from the PIPEs in replay mode, check the current sentinel.endpos value on the source database to make sure the ENDPOS message matches with the current setting, and that being the case stop processing.
927c29d to
dcc51ad
Compare
| @@ -990,6 +1014,13 @@ follow_wait_subprocesses(StreamSpecs *specs) | |||
| } | |||
| } | |||
|
|
|||
| /* update current sentinel values (endpos) */ | |||
| if (!follow_get_sentinel(specs, &(specs->sentinel), false)) | |||
There was a problem hiding this comment.
@dimitri I'm seeing lots of print(with -vv) with the latest version due to this. IIUC, now we call follow_get_sentinel for every 150ms?
play_lsn 0/0
2023-11-07 06:50:17 895197 SQL pgsql.c:490 Connecting to [source] "postgres://tsdbadmin@tsdb-22e8519b-internal-90d0.a.timescaledb.io:26479/defaultdb?sslmode=require&keepalives=1&keepalives_idle=10&keepalives_interval=10&keepalives_count=60"
2023-11-07 06:50:17 895197 SQL pgsql.c:1471 [SOURCE 4026526] select startpos, endpos, apply, write_lsn, flush_lsn, replay_lsn from pgcopydb.sentinel;
2023-11-07 06:50:17 895197 SQL pgsql.c:400 Disconnecting from [source] "postgres://tsdbadmin@tsdb-22e8519b-internal-90d0.a.timescaledb.io:26479/defaultdb?sslmode=require&keepalives=1&keepalives_idle=10&keepalives_interval=10&keepalives_count=60"
2023-11-07 06:50:17 895197
There was a problem hiding this comment.
Yes we do. We could call it less often I suppose, I will have a look, thanks for your feedback.
To avoid infinite looping when endpos has been reached it's important to update our endpos value to the sentinel's one even in the main follow process.
In passing, review some error messages.
Also refrain from stopping early from reading data from a PIPE when a terminating signal is received, we should finish reading as per the comments in the code.