Implement "live" streaming of changes from source to target. #185
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
8d7e10b to
6b8f353
Compare
This patch introduces support for Unix Pipes and an internal streaming API
where changes flow through the system rather than being batched in files.
The following command line can be used to replay changes in a "live" fashion
from the source to the target database.
$ pgcopydb stream receive --to-stdout
| pgcopydb stream transform - -
| pgcopydb stream apply -
The command `pgcopydb stream replay` is scheduled to glue together those
internal parts, and then the `pgcopydb clone --follow` will be extended to
automatically switch from catchup mode to replay mode when suitable.
Share the code that reads through a unix pipe (stream) line by line, introducing a user callback facility with a (void *) context.
This avoids partial reads of a part of a line, now we read a line at a time, unless it doesn't fit in our 128 MB read buffer, a case that is handled in the code already (albeit quite poorly at the time being). Also improve logging, install set -o pipefail in the test suite.
74f1bb6 to
f8a9348
Compare
arajkumar
added a commit
to arajkumar/pgcopydb
that referenced
this pull request
Dec 10, 2024
24a7d6d Fix incorrect table locking during copy (dimitri#184) 0b069e3 Validate wal2json plugin and enable numeric as string if supported (dimitri#183) aca861f Unique constraint is not a default replica identity (dimitri#182) Signed-off-by: Arunprasad Rajkumar <ar.arunprasad@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch introduces support for Unix Pipes and an internal streaming API where changes flow through the system rather than being batched in files. The following command line can be used to replay changes in a "live" fashion from the source to the target database.
The command
pgcopydb stream replayis scheduled to glue together those internal parts, and then thepgcopydb clone --followwill be extended to automatically switch from catchup mode to replay mode when suitable.