MSC2716: Incrementally importing history into existing rooms #2716

ara4n · 2020-08-04T01:52:25Z

@tulir

proposals/2716-importing-history-into-existing-rooms.md

lieuwex · 2020-08-29T14:16:32Z

ara4n · 2020-11-10T19:37:15Z

tulir

proposals/2716-importing-history-into-existing-rooms.md

tulir · 2020-11-10T21:14:27Z

proposals/2716-importing-history-into-existing-rooms.md

+
+## Unstable prefix
+
+Feels unnecessary.


It might be useful to have an unstable feature flag to check if the homeserver supports this

proposals/2716-importing-history-into-existing-rooms.md

kegsay · 2020-12-17T15:58:22Z

proposals/2716-importing-history-into-existing-rooms.md

+     | \___________________________________
+     |            \                        \
+     |             \                        \
+live timeline    previous 1000 messages   another block of ancient history


Backfilling via /messages works by walking back up prev_events. If the DAG looks like this, we'll never hit different eras so /messages will return 0 events.

EDIT: Actually it uses depth which will interleave instead. /get_missing_events will however walk up prev events, so all these lovely eras will never make it to other federated servers.

I don't think this works because the /messages endpoint has no idea when to jump to a different era. That endpoint topologically walks the DAG (in Dendrite it does this by depth), meaning if you actually did this you would get interleaved events as each era's events start producing the same depth values. This at least returns the events in the forks, but not where you want them.

In addition, doing this would produce forwards extremities at the end of each era, which servers will attempt to merge.

so my expectation is that a homeserver should calculate the an appropriate depth when importing history like this, probably by tiebreaking based on origin_server_ts. Where does Dendrite get its depth param from? As it certainly shouldn't be trusting the one it receives over federation, because of https://github.com/matrix-org/matrix-doc/issues/1229.

also, the fact that we create forward extremities at the end of each era which then get merged by the next message sent in the room was intended to be a feature, not a bug.

From our meeting, the depth is assumed from the stream ID and can be spoofed. I may not have the details correct but we did discuss fudging it.

So having rediscussed this IRL: Dendrite (and Synapse) currently get their depth parameters used for ordering from the wire. Ideally, we'd calculate the depth parameter instead - which could be easy, if we mandate that blocks of old history are always loaded contiguously in reverse chronological order. As a quick fudge to test the approach however we could set depth=1 for these events, and hopefully the default ordering will be sufficient (we think it is on synapse, but dendrite might need a tweak).

kegsay

kegsay · 2020-12-17T17:33:26Z

proposals/2716-importing-history-into-existing-rooms.md

+     | \___________________________________
+     |            \                        \
+     |             \                        \
+live timeline    previous 1000 messages   another block of ancient history


I don't think this works because the /messages endpoint has no idea when to jump to a different era. That endpoint topologically walks the DAG (in Dendrite it does this by depth), meaning if you actually did this you would get interleaved events as each era's events start producing the same depth values. This at least returns the events in the forks, but not where you want them.

kegsay · 2020-12-17T17:35:10Z

proposals/2716-importing-history-into-existing-rooms.md

+     | \___________________________________
+     |            \                        \
+     |             \                        \
+live timeline    previous 1000 messages   another block of ancient history


In addition, doing this would produce forwards extremities at the end of each era, which servers will attempt to merge.

proposals/2716-importing-history-into-existing-rooms.md

flokli · 2022-11-14T14:58:03Z

proposals/2716-batch-send-historical-messages.md

+with `?prev_event_id` pointing at that floating state to auth the event and where we
+want to insert the event.
+
+Another way of doing this might be to store the different eras of the room as


Has there been any discussion around being able to import history from the client?

Assume someone was previously active on IRC, and creates a Matrix account, then joins an IRC channel via matrix-appservice-irc.
That person might already have had some room history or DMs (that the AS is unaware of), and it would be nice if there was a way to "import" these old logs from the IRC client into some room state, and then see it consistently in all their Matrix clients.

This would need to be stored somewhere server-side, so it's available across all devices, but it might not necessarily be shared to other room members (even though two parties might decide to migrate their conversation history that has happened on IRC to a Matrix DM room)

@flokli The new /batch_send endpoint here is in the client API -> POST /_matrix/client/v1/rooms/<roomID>/batch_send?prev_event_id=<message3-eventID> but is currently restricted to application service tokens currently.

If there are good use cases for it, we could easily remove this restriction to allow any room admin (person with proper power levels) to do it.

Thanks for the clarification!

I agree, I would be very much interested in this too, being able to import old chats to an appservice would be quite relevant.

@asymmetric

turt2live · 2023-01-10T03:46:25Z

proposals/2716-batch-send-historical-messages.md

+Because we also can't use the `historical` power level for controlling who can
+send these events in the existing room version, we always persist but instead
+only process and give meaning to the `m.room.insertion`, `m.room.batch`, and
+`m.room.marker` events when the room `creator` sends them. This caveat/rule only
+applies to existing room versions.


Is this using soft rejection on the server-side to determine valid-ness of events? Currently it reads a bit like an event auth change, which can't happen with respect to existing room versions.

It doesn't reject or soft-fail any events in existing room versions.

If the homeserver sees one of those event types from the room creator, the homeserver will process as necessary to make the imported history display when paginating over that area of the room.

Otherwise, it just appears as any other event in the room, nothing happens, and the imported history isn't visible. If the homeserver ever adds MSC2716 support, that history suddenly is unlocked for that server once they re-process the events in the room (probably as a background job).

twouters · 2023-01-10T14:54:32Z

proposals/2716-batch-send-historical-messages.md

+Synapse, these are called `outlier`s and won't be visible in the chat history
+which also allows us to insert multiple batches without having a bunch of `@mxid
+joined the room` noise between each batch. **The state will not be resolved into
+the current state of the room.**


Current behavior of synapse is undesirable because it fails backfilled joins for members who have already joined the room.

Does the spec need to clarify that the client implementation must validate the current state of the room before sending a batch event, or should the spec mandate server implementations to ignore batched member state events that are already fulfilled in the current state of the room?

What fails about the member already being joined to the room? What error are you seeing? I would assume this should work because a m.room.member transition from join -> join is allowed, https://spec.matrix.org/v1.4/client-server-api/#mroommember

For clarity, it would be good to see your exact request that is failing.

Back to your topic of how to resolve things; we would want the conflict resolution to be general and the same for all event types. Doing something different for m.room.member would be a recipe for complexity foot-guns.

In most cases, bridges have to include invite events, and join -> invite is not allowed.

Ignoring such state events feels like it'd be the easiest to implement on the server, but it doesn't necessarily need to be built into state_events_at_start at all, it could be a new members field that the server uses to add invite and join events as necessary, or it could even just detect senders of the events in the batch. From the bridge point of view, all of those options are easy to implement.

Beeper (hungryserv) does the last option: it gets the list of users from the event senders, filters away any users who are already in the room and then creates member events for the missing users. (we also don't have any use cases for state events, so the server forbids using state_events_at_start entirely)

MadLittleMods · 2023-03-16T23:58:13Z

proposals/2716-batch-send-historical-messages.md

+**Power level:**
+
+ - `historical` (does not need prefixing because it's already under an
+   experimental room version)


This language is derived/inspired by MSC2285

Suggested change

experimental room version)

### While the MSC is unstable

During this period, to detect server support clients should check for the

presence of the `org.matrix.msc2716` flag in `unstable_features` on `/versions`.

Clients are also required to use the unstable prefixes (see [unstable

prefix](#unstable-prefix)) during this time.

### Once the MSC is merged but not in a spec version

Once this MSC is merged, but is not yet part of the spec, clients should rely on

the presence of the `org.matrix.msc2716.stable` flag in `unstable_features` to

determine server support. If the flag is present, clients are required to use

stable prefixes (see [unstable prefix](#unstable-prefix)).

### Once the MSC is in a spec version

Once this MSC becomes a part of a spec version, clients should rely on the

presence of the spec version, that supports the MSC, in `versions` on

`/versions`, to determine support. Servers are encouraged to keep the

`org.matrix.msc2716.stable` flag around for a reasonable amount of time

to help smooth over the transition for clients. "Reasonable" is intentionally

left as an implementation detail, however the MSC process currently recommends

*at most* 2 months from the date of spec release.

moritzdietz · 2023-03-28T12:06:11Z

MadLittleMods · 2023-03-28T18:51:19Z

moritzdietz · 2023-03-30T10:49:08Z

Avamander · 2023-03-30T11:42:11Z

James-Mat · 2023-04-10T11:31:26Z

MadLittleMods · 2023-04-11T23:29:56Z

@PaarthShah

Beryesa · 2025-01-03T20:26:06Z

ara4n added the proposal A matrix spec change proposal label Aug 4, 2020

note that we don't solve lazyloading history from ASes

3a03172

turt2live added kind:feature MSC for not-core and not-maintenance stuff proposal-in-review labels Aug 4, 2020

add another alternative

5e6b7b9

Half-Shot self-requested a review August 5, 2020 00:00

turt2live self-requested a review August 5, 2020 06:02

s/parent/prev_event/ for consistency with SS API

9451439

Half-Shot reviewed Aug 11, 2020

View reviewed changes

proposals/2716-importing-history-into-existing-rooms.md Outdated Show resolved Hide resolved

emorrp1 mentioned this pull request Nov 7, 2020

Support importing history from Signal chatrooms mautrix/signal#1

Closed

martindale mentioned this pull request Nov 10, 2020

Restore Archive russelldavies/matrix-archive#15

Open

turt2live removed their request for review November 10, 2020 05:28

MadLittleMods mentioned this pull request Nov 10, 2020

Gitter feature parity overview issue element-hq/roadmap#26

Open

48 tasks

tulir reviewed Nov 10, 2020

View reviewed changes

kegsay reviewed Dec 17, 2020

View reviewed changes

kegsay requested changes Dec 17, 2020

View reviewed changes

MadLittleMods reviewed Dec 17, 2020

View reviewed changes

proposals/2716-importing-history-into-existing-rooms.md Outdated Show resolved Hide resolved

MadLittleMods mentioned this pull request Dec 23, 2020

MSC2923: Connect Matrix room to another Matrix room #2923

Draft

2 tasks

MadLittleMods mentioned this pull request Jan 6, 2021

WIP: MSC2716: Incremental existing history matrix-org/dendrite#1650

Closed

2 tasks

MadLittleMods reviewed Jan 19, 2021

View reviewed changes

proposals/2716-importing-history-into-existing-rooms.md Outdated Show resolved Hide resolved

MadLittleMods mentioned this pull request Jan 28, 2021

Add tests for MSC2716 and backfilling history matrix-org/complement#68

Merged

11 tasks

MadLittleMods added a commit to matrix-org/synapse that referenced this pull request Jan 28, 2021

Add endpoints for backfilling history (MSC2716)

22c038e

Work on matrix-org/matrix-spec-proposals#2716 Complement MR: matrix-org/complement#68

MadLittleMods mentioned this pull request Jan 28, 2021

Add endpoints for backfilling history (MSC2716) matrix-org/synapse#9247

Merged

9 tasks

MadLittleMods added a commit to matrix-org/complement that referenced this pull request Jan 28, 2021

Add tests for MSC2716 and backfilling history

15ba451

Work on MSC2716: matrix-org/matrix-spec-proposals#2716

MadLittleMods reviewed Jan 30, 2021

View reviewed changes

proposals/2716-importing-history-into-existing-rooms.md Outdated Show resolved Hide resolved

flokli reviewed Nov 14, 2022

View reviewed changes

MadLittleMods mentioned this pull request Dec 15, 2022

Fix possible variable shadow in create_new_client_event matrix-org/synapse#14575

Merged

H-Shay mentioned this pull request Dec 16, 2022

Determine an effective way to differentiate between internal batched event creation and batch_send batched events matrix-org/synapse#14689

Open

turt2live reviewed Jan 10, 2023

View reviewed changes

twouters reviewed Jan 10, 2023

View reviewed changes

MadLittleMods reviewed Mar 16, 2023

View reviewed changes

MadLittleMods mentioned this pull request Apr 6, 2023

Follow tombstone and predecessor history matrix-org/matrix-viewer#59

Closed

6 tasks

MadLittleMods added 2 commits April 13, 2023 16:31

Make it more clear what the drawbacks are

e4193ff

Pulling from my summary comments at: - #2716 (comment) - #2716 (comment)

with should be without

1fc8b6b

tulir mentioned this pull request May 15, 2023

Backfilling portal after the fact mautrix/discord#89

Closed

MadLittleMods mentioned this pull request May 24, 2023

Process previously failed backfill events in the background matrix-org/synapse#15585

Merged

5 tasks

MadLittleMods mentioned this pull request Jun 8, 2023

Remove experimental MSC2716 implementation to incrementally import history into existing rooms matrix-org/synapse#15748

Merged

7 tasks

MadLittleMods mentioned this pull request Jun 16, 2023

Remove MSC2716 tests to incrementally import history into existing rooms matrix-org/complement#640

Merged

1 task

matrixbot mentioned this pull request Dec 21, 2023

Determine an effective way to differentiate between internal batched event creation and batch_send batched events element-hq/synapse#14689

Open

toger5 mentioned this pull request May 21, 2024

MSC4140: Delayed events #4140

Open

4 tasks

anoadragon453 removed their request for review June 24, 2024 13:10

flying-scorpio mentioned this pull request Jul 25, 2024

Incorrect order of messages in the Matrix when using concurrency in the script verdigado/rocketchat2matrix#22

Open

MSC2716: Incrementally importing history into existing rooms #2716

Are you sure you want to change the base?

MSC2716: Incrementally importing history into existing rooms #2716

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!