8000 Sentinel is updated for every commit by arajkumar · Pull Request #774 · dimitri/pgcopydb · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Sentinel is updated for every commit #774

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
May 13, 2024
Merged

Sentinel is updated for every commit #774

merged 1 commit into from
May 13, 2024

Conversation

arajkumar
Copy link
Contributor
@arajkumar arajkumar commented May 13, 2024 •
edited
Loading

We used to track last sentinel sync time and update sentinel for every 1 sec. However, the change[1] which adopted sqlite to track sentinel removed the last updated time assignment[2] leading to sentinel write
for every commit message causes replay slowness by almost 6 times.

Solution: Track last sentinel update time

Without the fix, the following snippet will be always executed because 1 < (now - context->sentinelSyncTime) will be always true!

pgcopydb/src/bin/pgcopydb/ld_replay.c

Lines 170 to 179 in 812a108

if (1 < (now - context->sentinelSyncTime))
{
bool findDurableLSN = true;
if (!stream_apply_sync_sentinel(context, findDurableLSN))
{
/* errors have already been logged */
return false;
}
}

[1] #601
[2] https://github.com/dimitri/pgcopydb/pull/601/files#diff-481cd5fcd15742c022e2a18916c7a508cc7c7d46c264084daab85200e1b10004L477

Before this PR,

cat ~/dimitri/bench_new.sql | time pgcopydb stream apply --resume --not-consistent -
14:24:27.034 1550353 INFO   Running pgcopydb version 0.0.13.5.g6a1c2f8.dirty from "/home/ubuntu/dimitri/pgcopydb/src/bin/pgcopydb/pgcopydb"
14:24:27.082 1550353 INFO   Using work dir "/tmp/pgcopydb"
14:24:27.155 1550353 INFO   Setting up previous LSN from replication origin "pgcopydb" progress at 0/0, overriding previous value D8/959B48C0
14:24:27.156 1550353 INFO   Catchup-up with changes from LSN 0/0
pgcopydb stream apply --resume --not-consistent -  7.93s user 6.78s system 9% cpu 2:30.66 total

After this PR,

cat ~/dimitri/bench_new.sql | time pgcopydb stream apply --resume --not-consistent -
14:47:05.704 1551251 INFO   Running pgcopydb version 0.0.13.5.g6a1c2f8.dirty from "/home/ubuntu/dimitri/pgcopydb/src/bin/pgcopydb/pgcopydb"
14:47:05.752 1551251 INFO   Using work dir "/tmp/pgcopydb"
14:47:05.824 1551251 INFO   Setting up previous LSN from replication ori
8000
gin "pgcopydb" progress at 0/0, overriding previous value D8/95A663C8
14:47:05.825 1551251 INFO   Catchup-up with changes from LSN 0/0
pgcopydb stream apply --resume --not-consistent -  8.76s user 3.24s system 57% cpu 20.883 total

Gain:

(2m, 30 sec - 21 sec) / 21 sec = ~6.1 times

@arajkumar
Sentinel is updated for every commit
ff5d349 
We used to track last sentinel sync time and update sentinel for every
1 sec. However, the change[1] which adopted sqlite to track sentinel
  removed the last updated time assignment[2] leading to sentinel write
for every commit message causes replay slowness by almost 10 times.

Solution: Track last sentinel update time

[1] dimitri#601
[2] https://github.com/dimitri/pgcopydb/pull/601/files#diff-481cd5fcd15742c022e2a18916c7a508cc7c7d46c264084daab85200e1b10004L477

Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
@dimitri dimitri added bug Something isn't working enhancement New feature or request labels May 13, 2024
@dimitri dimitri added this to the v0.16 milestone May 13, 2024
@dimitri
Copy link
Owner
dimitri commented May 13, 2024

Fixes #754

@dimitri dimitri merged commit f4b1c1c into dimitri:main May 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dimitri dimitri dimitri approved these changes

Assignees

@arajkumar arajkumar

Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Milestone
v0.16
Development

Successfully merging this pull request may close these issues.
2 participants
@arajkumar @dimitri
0