8000 feat: file sync by AlexXi19 · Pull Request #1416 · tensorchord/envd · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

feat: file sync #1416

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 36 commits into from
Apr 21, 2023
Merged

feat: file sync #1416

merged 36 commits into from
Apr 21, 2023

Conversation

AlexXi19
Copy link
Member
@AlexXi19 AlexXi19 commented Jan 16, 2023

File sync

Syncs the user's local project directory to a pod in the kubernetes cluster started by envd run with envd-server.

Steps

  • Installs syncthing binary on the user's device
  • Starts local syncthing process
  • Connects to remote syncthing process in the pod
    • Does so by port forwarding the syncthing api
  • Connects the local device and the remote device by applying a config
    • Local and remote forward of syncthing discovery ports
  • Adds syncing folders by applying a config

Waiting for config to apply

Applying configurations through the syncthing rest api is async so we use the event api and wait until we see a config apply success event

What's not included and known limitations

  • Currently can only work for one project at a time because all syncthing ports are hardcoded and not fetched using a getFreePorts. This change should be pretty easy

Syncthing process management

  • The process will exit on ctrl-c and ctrl-d

What's next

  • Sync for multiple projects
  • envd attach for starting up the sync and port forwarding again

Testing

Would like some opinion on what's are the final tests to keep in the repo.

  • Manual walking through the process and seeing if it works.
  • Miscellaneous testing written during development to progress check on intermediate functionalities.

AlexXi19 and others added 23 commits December 24, 2022 22:57
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: Alex Xi <68758451+AlexXi19@users.noreply.github.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: Alex Xi <68758451+AlexXi19@users.noreply.github.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
@AlexXi19 AlexXi19 marked this pull request as ready for review January 17, 2023 08:26
@gaocegege
Copy link
Member

Thanks for your contribution! 🎉 👍

Is this ready to review?

@VoVAllen
Copy link
Member

@gaocegege it's ready

@VoVAllen
Copy link
Member

Things to be discussed:

  • How user can specify the behavior of code folder, sync vs git clone? Through CLI or envd file?

Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
AlexXi19 and others added 2 commits January 22, 2023 23:41
Signed-off-by: Alex Xi <68758451+AlexXi19@users.noreply.github.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
@AlexXi19
Copy link
Member Author

addressed comments

cc @kemingy

Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Copy link
Member
@VoVAllen VoVAllen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also add syncthing option at the envd-server side

And also modify the server part, to ensure the git clone logic is not affected is syncthing option is not specified

@VoVAllen
Copy link
Member

And also please resolve the conflicts. Thanks!

@gaocegege
Copy link
Member

/cc @kemingy

Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
@AlexXi19
Copy link
Member Author

@VoVAllen waiting on envd-server release to get latest types.

@gaocegege
Copy link
Member

I will create a new release today.

Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
@AlexXi19 AlexXi19 changed the title File sync feat: file sync Feb 21, 2023
@AlexXi19
Copy link
Member Author

@VoVAllen updated dependencies, now its all passing :)

Signed-off-by: AlexXi19 <alex2001314jjj@gmail.com>
Copy link
Member
@gaocegege gaocegege left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Could we merge this?

/cc @kemingy @VoVAllen

103FC
@kemingy
Copy link
Member
kemingy commented Apr 20, 2023

LGTM

Could we merge this?

/cc @kemingy @VoVAllen

LGTM

@AlexXi19
Copy link
Member Author

Will resolve conflicts later today.

@VoVAllen VoVAllen added this pull request to the merge queue Apr 21, 2023
Merged via the queue into tensorchord:main with commit e56981a Apr 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants
0