best way to 'tap' into commited changes #134

guykatz · 2024-02-06T11:29:05Z

guykatz
Feb 6, 2024

hi all, i think I already asked something like this or similar but i couldnt find it..
is there a way to 'tap' into the changes that are commited (succesfuly) to the storage?
some sort of a transaction log i can read, some sort of event API i can listen to?
thanks

hg-ms · 2024-02-07T06:25:18Z

hg-ms
Feb 7, 2024
Collaborator

Hello,
There is a small tool to convert the binary content from the transaction log files to text.
To build the TransactionFileConverter as standalone jar you can use the maven profile transaction-file-converter:

eclipse-store\storage\storage>mvn -Ptransaction-file-converter clean package

Now you should find the transaction-file-converter.jar in the target directory.

The transaction log files are the *.sft files you can find in each channel’s directory, to convert them simple pass a *sft file to the transaction-file-converter.jar:

java -jar transaction-file-converter.jar transactions_0.sft

The transaction log files contain a time stamp, the type of change and some file length information.

4 replies

guykatz Feb 7, 2024
Author

thanks @hg-ms
what I am looking for is to know what actually changed in some granularity (prefferably smaller then bigger).
looking at the transaction log as text is see the following info via the header:
Type Timestamp Time Delta (ms) Timestamp(long) Resulting Length Length Change Current Head File Significant File Special Offset
but I dont undertsand how to know what exactly was created, for example, in the following log line:
CREATION 2023-10-24 13:30:33.501 4528 1698143433501000000 0 0 1 2
i understand something was created but how can i know what exactly?
thanks!

hg-ms Feb 7, 2024
Collaborator

The log entries a described at the start of StorageTransactionsAnalysis.java
The Creation Entry means that:

a file was created
at 2023-10-24 13:30:33.501
4528ms after the last log entry
1698143433501000000 is the raw timestamp as long
the new file has a length of 0 bytes
there was a length change of 0 bytes
the current had file (the file the channel append data) is file no 1
the new create file is file no 2

But I guess this is not the kind of changes you are interested in.

There is a way to get all objects that will be persisted, but it is a bit hacky:
To do so you need to create a storer and cast it to BinaryStorer.Default.
Between the store and commit you can retrieve all objects that will be persisted.

BinaryStorer.Default storer = (Default) storage.createLazyStorer();
storer.store(root);
storer.iterateMergeableEntries((oid, instance) -> {System.out.println("oid: " + oid + " instance: " + instance);} );
storer.commit();

guykatz Feb 8, 2024
Author

thanks @hg-ms
do you think it might be of interest for you guys to incorporate some sort of a listener API for this inside the code?
I am thinking about potential use cases like outbox pattern, Change Data Capture integration etc...
this would be beneficial I think...

hg-ms Feb 8, 2024
Collaborator

8000

You are right, it can be useful to have some convenient option to get the persisted objects.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

best way to 'tap' into commited changes #134

{{title}}

Replies: 1 comment 4 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

best way to 'tap' into commited changes #134

guykatz Feb 6, 2024

Replies: 1 comment · 4 replies

hg-ms Feb 7, 2024 Collaborator

guykatz Feb 7, 2024 Author

hg-ms Feb 7, 2024 Collaborator

guykatz Feb 8, 2024 Author

hg-ms Feb 8, 2024 Collaborator

guykatz
Feb 6, 2024

Replies: 1 comment 4 replies

hg-ms
Feb 7, 2024
Collaborator

guykatz Feb 7, 2024
Author

hg-ms Feb 7, 2024
Collaborator

guykatz Feb 8, 2024
Author

hg-ms Feb 8, 2024
Collaborator