8000 what if i have a 10GB file to upload? · Issue #38 · sethgrid/pester · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

what if i have a 10GB file to upload? #38

8000 New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
fredwangwang opened this issue Oct 15, 2018 · 3 comments
Open

what if i have a 10GB file to upload? #38

fredwangwang opened this issue Oct 15, 2018 · 3 comments

Comments

@fredwangwang
Copy link

Seems like the logic is going to cache the file in-memory, which is super expensive to a simple HTTP POST.

@sethgrid
Copy link
Owner

I've been moving across the county; a bit swamped.
Any suggestions? PRs welcome :)

@fredwangwang
Copy link
Author

sorry, i dont monitor my email that much...

Some thinkings:
Provide additional layer to wrap some common file upload protocols. For example: plain post upload, multipart upload.

Then the wrapping layer can ask the user for raw file input instead of io.reader. The advantage of this is every time the upload fails and retries, the file can be rewind to the start of the stream. No data is lost, no need to store in-mem as well.

But talking this out loud here, it is probably not wise to keep retrying upload large files anyways. Some resumable upload protocol should be used instead of "brute force uploading". So adding a warning to stderr when detecting the user has a really large payload probably good enough?

@derekbassett
Copy link

Would it be possible in this case to detect an io.ReadSeeker? That way you could seek back to the beginning and read again?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants
0