8000 Refactor code to fetch current LSN position from Postgres. by dimitri · Pull Request #608 · dimitri/pgcopydb · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Refactor code to fetch current LSN position from Postgres. #608

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jan 2, 2024
Merged

Refactor code to fetch current LSN position from Postgres. #608

merged 1 commit into from
Jan 2, 2024

Conversation

dimitri
Copy link
Owner
@dimitri dimitri commented Jan 2, 2024

No description provided.

@dimitri dimitri added this to the v0.15 milestone Jan 2, 2024
@dimitri dimitri self-assigned this Jan 2, 2024
@dimitri dimitri merged commit 4273b4e into main Jan 2, 2024
@dimitri dimitri deleted the refactor/fetch-current-lsn branch January 2, 2024 11:17
arajkumar
arajkumar reviewed Jan 3, 2024
View reviewed changes
src/bin/pgcopydb/catalog.c
" write_lsn pg_lsn, flush_lsn pg_lsn, replay_lsn pg_lsn)"
" write_lsn pg_lsn, flush_lsn pg_lsn, replay_lsn pg_lsn)",

"create table lsn_tracking(source pg_lsn, target pg_lsn)"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you planning to move file based lsn tracking to sqlite?

(I don't see this table being used anywhere.)

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ooops, missed the clean-up for that. The idea crossed my mind yeah, but I wanted to clean-up some other aspects first (summary information and top-level timings in the SQLite database so that pgcopydb list summary can represent multi-steps activity and also be used concurrently to the main command, or after the fact).

@arajkumar
Copy link
Contributor

I also see lots of warnings after moving sentinel to sqlite. Should we change the log level to debug?

10:04:16.611 2223479 WARN   Skipping sentinel replay_lsn update: failed to find a durable LSN matching current flushLSN
10:04:16.792 2223479 WARN   Skipping sentinel replay_lsn update: failed to find a durable LSN matching current flushLSN
10:04:16.975 2223479 WARN   Skipping sentinel replay_lsn update: failed to find a durable LSN matching current flushLSN
10:04:17.160 2223479 WARN   Skipping sentinel replay_lsn update: failed to find a durable LSN matching current flushLSN
10:04:17.360 2223479 WARN   Skipping sentinel replay_lsn update: failed to find a durable LSN matching current flushLSN
10:04:17.536 2223479 WARN   Skipping sentinel replay_lsn update: failed to find a durable LSN matching current flushLSN
10:04:26.113 2223477 INFO   Reported write_lsn 3/4247FD60, flush_lsn 3/42297318, replay_lsn 3/41E6C410
10:04:33.200 2223479 WARN   Skipping sentinel replay_lsn update: failed to find a durable LSN matching current flushLSN
10:04:36.113 2223477 INFO   Reported write_lsn 3/4247FDC8, flush_lsn 3/4247FD60, replay_lsn 3/41E6C410
10:04:45.322 2223479 WARN   Skipping sentinel replay_lsn update: failed to find a durable LSN matching current flushLSN
10:04:47.115 2223477 INFO   Reported write_lsn 3/4247FDF8, flush_lsn 3/4247FDC8, replay_lsn 3/41E6C410
10:04:53.536 2223479 WARN   Skipping sentinel replay_lsn update: failed to find a durable LSN matching current flushLSN
10:04:57.114 2223477 INFO   Reported write_lsn 3/4247FE60, flush_lsn 3/4247FDF8, replay_lsn 3/41E6C410
10:05:05.565 2223479 WARN   Skipping sentinel replay_lsn update: failed to find a durable LSN matching current flushLSN
10:05:07.115 2223477 INFO   Reported write_lsn 3/4247FF60, flush_lsn 3/4247FE60, replay_lsn 3/41E6C410

@dimitri
Copy link
Owner Author
dimitri commented Jan 3, 2024

I also see lots of warnings after moving sentinel to sqlite. Should we change the log level to debug?

Either that or find a way to avoid the warning by having the data we're looking for? ;-)

@arajkumar
Copy link
Contributor
arajkumar commented Jan 4, 2024 •
edited
Loading

@dimitri What happens with my tests is that the source being ingested with parallel connections, but the target being replicated with single connection causing the huge lag between source & target.
Due to this, we couldn't the matching target insert lsn for the flush_lsn from source.

I'm wondering, why can't we just use the LSN from pg_replication_origin_progress as durable LSN(replay_lsn)?

@arajkumar
Copy link
8BBD
Contributor

Ahh, I think we have to fetch LSN from target to compare against to find the durable LSN, not the source. Let me fix it.

@arajkumar arajkumar mentioned this pull request Jan 4, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@arajkumar arajkumar arajkumar left review comments

Assignees

@dimitri dimitri

Labels
None yet
Projects
None yet
Milestone
v0.15
Development

Successfully merging this pull request may close these issues.
2 participants
@dimitri @arajkumar
0