8000 Issue Uploading Large Files · Issue #1395 · minio/minio-js · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
8000

Issue Uploading Large Files #1395

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
AndrewKahr opened this issue Mar 23, 2025 · 6 comments · Fixed by #1396
Closed

Issue Uploading Large Files #1395

AndrewKahr opened this issue Mar 23, 2025 · 6 comments · Fixed by #1396

Comments

@AndrewKahr
Copy link

I am trying to upload large files (10-20GB) using minio-js to a local minio instance, however, I am seeing it fail with the following error:

warning: The value of "size" is out of range. It must be >= 0 && <= 4294967296. Received 16_493_197_601

I am using the following to upload:

const mc = newMinio({
        accessKey,
        secretKey
      });

await mc.fPutObject(bucket, object, archivePath, {});

I've also seen it throw the following error:

data is too long

But in this scenario, the file is smaller (roughly 2GB), so I'm not sure what it's referring to in terms of file size limits.

I am using minio-js v8.0.5, node20, and the minio server is RELEASE.2025-03-12T18-04-18Z (go1.24.1 windows/amd64)

Guidance would be much appreciated and let me know if any other information would be helpful!
@prakashsvmx
Copy link
Member

minio-js does not print any warning. please double check.

mc admin trace -v ALIAS - ?

@AndrewKahr
Copy link
Author

Running that seems to indicate that minio-js isn't even reaching out to the server as there are no calls for puts.

In scenarios where an upload did succeed, I would see many put messages so I know at least the server side is working and the client is able to communicate with the server.

@prakashsvmx
Copy link
Member

So it is most likely proxy, lb issue

@AndrewKahr
Copy link
Author

There is no proxy or lb between the client and the server. The client is in a VM and the server is running on the host and they communicate directly with each other. The client also has no issue downloading large objects, it only seems to affect uploading. This also previously worked with a build of my code from half a year ago. I only recently updated all the packages and rebuilt it when I started seeing the issue.

@AndrewKahr
Copy link
Author

I went ahead and did some additional debugging:
Minio v8.0.4 works, however, v8.0.5 breaks. I was also able to get stack traces:

RangeError [ERR_OUT_OF_RANGE]: The value of "size" is out of range. It must be >= 0 && <= 4294967296. Received 9_627_333_636
    at Function.allocUnsafe (node:buffer:403:3)
    at Function.concat (node:buffer:592:25)
    at ReadStream.<anonymous> (C:\dev\index.js:71835:106)
    at ReadStream.emit (node:events:519:28)
    at endReadableNT (node:internal/streams/readable:1696:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

Doing some Googling indicates this is a limitation of Buffer sizes in Node 20.

This makes me think there was some regression with the stream reader.

v22 increases the limit on 64 bit systems to 8PB so just as a test, I switched to node22 and instead got a different stack trace:

RangeError: data is too long
    at Hash.update (node:internal/crypto/hash:119:22)
    at hashBinary (C:\dev\index.js:71057:45)
    at Client.uploadBuffer (C:\dev\index.js:69397:32)
    at Client.putObject (C:\dev\index.js:69384:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async Client.fPutObject (C:\dev\index.js:69330:12)
    at async saveCache (C:\dev\index.js:109955:13)

Ultimately, switching to v22 wouldn't be ideal though as Github Actions action files do not support it yet and we'd prefer to stay on Node20 as a result.

Unfortunately, nothing stands out in the diff between v8.0.4 and v8.0.5 relating to streams but I'm hoping this can help direct the debugging direction. Let me know if any additional information would be helpful!

@prakashsvmx
Copy link
Member
prakashsvmx commented Mar 24, 2025 •
edited
Loading

Thank you @AndrewKahr ..
i think the issue is with compatibility of stream.readableLength . i will send a fix.
also fputObject is using lstat will update to stat

1GB file works fine.. 10GB does not work currently

@prakashsvmx prakashsvmx mentioned this issue Mar 24, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix large object upload and and update functional test for 0 byte file upload prakashsvmx/minio-js
2 participants
@AndrewKahr @prakashsvmx
0