Use our SQLite catalogs for the pgcopydb sentinel table. #601
8000Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
9d4f142 to
ea8f6f7
Compare
The main use-case for the sentinel table is to communicate progress made by the three independent worker processes for stream, transform, and replay. It is also needed to communicate new startpos, endpos and apply values with those processes from another pgcopydb (interactive) command, a process that's not in the process tree. This was previously solved by creating and using a pgcopydb.sentinel table on the source database. Now that we have a SQLite database internal to the pgcopydb run, we can use that instead.
In particular, we need to share a common directory mount point in the test and inject containers, because the inject container is setting the endpos for the test container where the main tests are running. For that, in my testing the best option is to use a host directory. This also makes it very easy to then use sqlite3 and inspect what's going on from the outside.
2dc02db to
d16ec8b
Compare
arajkumar
reviewed
Jan 2, 2024
arajkumar
added a commit
to arajkumar/pgcopydb
that referenced
this pull request
May 13, 2024
We used to track last sentinel sync time and update sentinel for every 1 sec. However, the change[1] which adopted sqlite to track sentinel removed the last updated time assignment[2] leading to sentinel write for every commit message causes replay slowness by almost 10 times. Solution: Track last sentinel update time [1] dimitri#601 [2] https://github.com/dimitri/pgcopydb/pull/601/files#diff-481cd5fcd15742c022e2a18916c7a508cc7c7d46c264084daab85200e1b10004L477 Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
dimitri
pushed a commit
that referenced
this pull request
May 13, 2024
We used to track last sentinel sync time and update sentinel for every 1 sec. However, the change[1] which adopted sqlite to track sentinel removed the last updated time assignment[2] leading to sentinel write for every commit message causes replay slowness by almost 10 times. Solution: Track last sentinel update time [1] #601 [2] https://github.com/dimitri/pgcopydb/pull/601/files#diff-481cd5fcd15742c022e2a18916c7a508cc7c7d46c264084daab85200e1b10004L477 Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The main use-case for the sentinel table is to communicate progress made by
the three independent worker processes for stream, transform, and replay.
It is also needed to communicate new startpos, endpos and apply values with
those processes from another pgcopydb (interactive) command, a process
that's not in the process tree.
This was previously solved by creating and using a pgcopydb.sentinel table
on the source database. Now that we have a SQLite database internal to the
pgcopydb run, we can use that instead.