[cleanup] Clean up unused scripts and files #80
Conversation
There was a problem hiding this comment.
Hey @BlankCheng, this is a great effort to clean up old and no longer useful scripts.
I would however recommend that we either keep hold of some historical/archived scripts for different sizes of base models and/or algorithms as a reference for when someone wants to run smaller models (I, for one, don't always know what the correct number of nodes are to use for a specific model size) or use DAPO vs GRPO, etc.
If we don't want to keep around a small number of archived scripts, then we should include a guide in the README for how someone would adjust the launch script for their own experiment, etc.
In fact, there is a strong need to update the README anyway since the example runner file/configuration is being removed as well as the entire set of data preprocessing scripts. We need to indicate what is going on there and point to where the data is on each server.
Thanks!
|
Hi, sure I'm writing README in another branch now and will PR when ready. |
Clean up unused scripts and files for a neat main branch.