Closed
Description
When streaming cat output into a shell pipeline, s5cmd buffers the whole file before passing it to the next application. This makes its speed pointless, as now the rest of the pipeline has to wait for complete file to be received. It also makes it buffer the whole file, which may be large, in memory.
It would be useful to have it stream the file as soon as it becomes available, as well as to allow restricting the amount of memory it uses for buffering.
Now it just gets killed by the OS once it has buffered enough data.
Or am I missing somehting?
Metadata
Metadata
Assignees
Labels
No labels