Slow file move within the same bucket #896
Unanswered
michaltarana
asked this question in
Q&A
Replies: 1 comment 1 reply
-
|
Might have been the same as in #916? |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
@michaltarana , maybe try again following that PR? I will be releasing today, so you can wait for that if you wish. |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hi, I am using s3fs 2024.9.0 to handle files on an S3 bucket operated on a CEPH cluster. One of the operations I need to perform is renaming a file, i.e. move within the same bucket, using
S3FileSytem_instance.mv('bucket/old_name', 'bucket/new_name')
While it works fine for smaller files, the files over 5GB move very slowly and I get back a ReadTimeOutError exception from botocore.
The same operation on the same files, using aws-cli mv takes only several seconds. However, s3fs is not done even after 15 minutes when the read_timeout exception is raised. Of course, I could try to increase the read_timeout config_kwarg. However, I am wondering, why is the s3fs so much slower with this operation on a large file than aws cli?
I would be very thankful for any insight.
Beta Was this translation helpful? Give feedback.
All reactions