8000 Training GPU Utilization and Other Kind of Dataset Format · Issue #18 · LitingLin/LoRAT · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Training GPU Utilization and Other Kind of Dataset Format #18

New issue

7E50 Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
JaaackHongggg opened this issue Apr 15, 2025 · 0 comments
Open

Training GPU Utilization and Other Kind of Dataset Format #18

JaaackHongggg opened this issue Apr 15, 2025 · 0 comments

Comments

@JaaackHongggg
Copy link

Thanks for your great work!

I am wondering what is the average gpu utilization when you train lorat on your device? It seems that you use V100 and 4090 in the paper.

When I train lorat by using your code without any modification, the GPU utilization is only about 60 by using H100. I have tested with different num_train_workers and num_io_threads_per_worker, the GPU utilization is always 60 for almost all the time. I use the cpfs (Cloud Parallel File Storage by aliyun) as the file system, which is designed for high performance computing.

I think that maybe the bottleneck is the disk io, but the cpfs is quick enough. Thus, maybe we should turn to other kinds of dataset format, such as lmdb, webdataset, or parquet, which is designed for high performance computing, instead of reading the image frequently from disk.

Do you have any plan to support other kinds of dataset format? I can give my help, if you can develop the support for other dataset format.

If you have any question, plz feel free to contact me.

Best regards.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@JaaackHongggg
0