8000 Feat 265 *platform wide* realtime support by eldadfux · Pull Request #692 · appwrite/appwrite · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Feat 265 *platform wide* realtime support #692

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 36 commits into from

Conversation

eldadfux
Copy link
Member
@eldadfux eldadfux commented Oct 20, 2020

What does this PR do?

Adding realtime server for allowing live updates and interactions with the Appwrite platform.

The server will include multiple realtime protocols support including (in order of priority):

  • Websocket - in progress
  • MQTT
  • Socket.io
  • SSE

This PR is still a work in progress. Stuff still yet to be implemented:

  • Abuse control check on connection start (X connections per Y hours, X connection per user)
  • Limit payload size
  • Defining message structure: { status: "ok"|"error", event: EVENT_NAME, data: }
  • JWT Authentication (passed in path / or in message) **related to Support For Verifying JWT Token using the Server API #511 **
  • List of channels
  • Channels subscriptions (per resource / group of resources (collections / files))
  • Logout on current session delete event
  • Integrate messaging worker for event triggering
  • Add e2e tests
  • Update installation script compose file

Test Plan

TODO: add new set of tests to both test connection establishment and stream functionality. Trigger mock event, assert payload is correct on each listener.

Related PRs and Issues

#511
#265
#509

@@ -30,6 +30,19 @@
$response->json(['version' => APP_VERSION_STABLE]);
}, ['response']);

App::get('/v1/health/realtime')
Copy link
Member Author
@eldadfux eldadfux Oct 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This route is only used for testing and will be deleted later.

@TorstenDittmann
Copy link
Contributor

May I ask where the need for multiple real-time protocols comes from?

@eldadfux
Copy link
Member Author
eldadfux commented Oct 20, 2020

@TorstenDittmann not much of a need, but more of a possibility. Seems to be pretty easy to support multiple protocols using Swoole.

This also have a very good fit with Appwrite agenda of being tech and platform agonistic, and might allow new use-cases. There were a few mentions in the community about MQTT and Websocket, I guess that will allow easier integration with Appwrite for platform that are already comfortable with a specific protocol or client SDK.

Obviously we'll start by supporting 1 protocol.

@TorstenDittmann
Copy link
Contributor

I see nothing explicitly wrong with this code for now, since it is a starting point anyway.

The plan is that all possible functions that are normally requested via HTTP are also accessible via the realtime protocol, right?

Also, have you figured out a plan to manage connections and subscriptions yet?

Amazing work! 👍

@eldadfux
Copy link
Member Author

The plan is that all possible functions that are normally requested via HTTP are also accessible via the realtime protocol, right?

@TorstenDittmann I'm still thinking about it, not sure how the message request will look like, maybe something similar to GraphQL or GraphQL itself? I was thinking of starting with a read-only mode where users can subscribe and receive events and use it in combination with the REST API. What do you think?

Also, have you figured out a plan to manage connections and subscriptions yet?

This is what I'm thinking about. I'm gonna use a hashmap of all the users where the key is going to be the connected user ID and the value is going to be a hashmap of channel names and the user connection ID, like this: user_id => [channels => ['channel1' => true, 'channel2' => true], connection => x].

Every new event message will do a hash lookup, which should be pretty fast, find all relevant users, check if they have subscribed to the messages, and then send the message if relevant (if the user has subscribed for it).

The performance here should be O(N), where N is the number of "read" permissions in a resource [file/document/collection] - (@TorstenDittmann let me know if I'm missing something). Maybe we should add a limit to the max number of permissions?

On each connection establishment, we'll verify the user has privileges to access that channel. We'll remove the connection ID on every connection close event or on specific failure errors.

For channel logic, I had this in mind: (but I'd love some feedback)

Channel Description
account All account related events (session create, name update...)
collections Any update/delete/create events for collections where user has read access
collections.[ID] Any update/delete/create events to a given collection where user has read access
collections.[ID].documents Any update/delete/create events to any document in a given collection where user has read access
documents Any update/delete/create events to documents where user has read access
documents.[ID] Any update/delete/create events to a given document where user has read access
files Any update/delete/create events to file where user has read access
files.[ID] Any update/delete/create events to a given file where user has read access

@joshdvir
Copy link

@eldadfux I think that adding realtime feature to Appwrite will be a great addition, just a few things to keep in mind:

  1. MQTT is not really a realtime protocol, it's a lightweight messaging protocol I'm not sure what use case are you trying to add here with it.
  2. SSE only allows 1 way communication, only the server is sending messages to the client which can be limiting, I prefer Websockets by far.
  3. I like the approach you took when you wanted to add a new feature you added a new component to complement that feature, I think this should also apply here, Scaling Websockets is different than scaling HTTP services so I think the component should be a new one that can be scaled independently and not part of the main application.
    Moreover in the past I used https://centrifugal.github.io/centrifugo/ as a websockets server with much success and adding this type of component will save a lot of development time since the main application would just need to "manage" it and define the protocol to use.
    Centrifugo also support SockJS which is like Socket.io.

Let me know what you think

Josh

@eldadfux
Copy link
Member Author

Thanks for the feedback @joshdvir!

About the points you raised:

  1. MQTT can be used to communicate between Appwrite and a developer custom backend located on the same network or datacenter. Also one of our main agendas is to be tech and platform agnostic in order to allow max customisation and flexibility with Appwrite.

  2. Actually that can be good enough for a lot of use cases where you just need a listener. Even with Websocket will probably start by supporting on way communication and on phase 2 we'll add 2-way communication, but that hasn't been decided yet.

  3. Completely agree. The realtime server is going to be an independent & stateless Docker container which will be super easy to both scale and debug. We're using Swoole for the Websocket server mainly because we already use it for HTTP (with great performance) and it allow us to easily reuse a lot of our codebase.

@eldadfux eldadfux changed the base branch from 0.7.x to dev February 21, 2021 17:21
@eldadfux
Copy link
Member Author

@TorstenDittmann are we OK to close this PR in favor of #948 ?

@TorstenDittmann
Copy link
Contributor

@TorstenDittmann are we OK to close this PR in favor of #948 ?

Yes, we can close this one.

@eldadfux eldadfux closed this Apr 15, 2021
@eldadfux eldadfux deleted the feat-265-realtime-support branch May 17, 2021 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants
0