Make your ML experiment wrapper scripts smarter with...
Install • Tutorial / Demo • Documentation • FAQs • Releases
🌟 Labtasker makes ML experiment wrapper scripts smarter with task prioritization, failure handling, halfway resume and more: just change 1 line of code.
If you like our project, please give us a star ⭐ on GitHub for latest update.
TLDR: Replace for loops in your experiment wrapper script with labtasker to enable features like experiment
parallelization, dynamic task prioritization, failure handling, halfway resume, and more.
for script in eval/eval_model_A.py eval/eval_model_B.py
do
for dataset in visualmrc_test halu_eval foo_eval bar_eval baz_eval
do
- # run sequentially with only 1 GPU 😫
- CUDA_VISIBLE_DEVICES=0 python $script --dataset $dataset
+ # submit the task args once
+ labtasker submit -- --exp_script $script --exp_dataset $dataset
done
done+ # parallelism across any number of workers effortlessly 😄
+ CUDA_VISIBLE_DEVICES=0 labtasker loop -- python '%(exp_script)' --dataset '%(exp_dataset)' &
+ CUDA_VISIBLE_DEVICES=1 labtasker loop -- python '%(exp_script)' --dataset '%(exp_dataset)' &
...
+ CUDA_VISIBLE_DEVICES=7 labtasker loop -- python '%(exp_script)' --dataset '%(exp_dataset)' &🐳 For detailed examples and concepts, check out the documentation.
This demo shows how to easily submit task arguments and run jobs in parallel.
It also features an event listener to monitor task execution in real-time and automate workflows, such as sending emails on task failure.
For more detailed steps, please refer to the content in the Tutorial / Demo.
- ⚙️ Easy configuration and setup.
- 🧩 Versatile and minimalistic design.
- 🔄 Supports both CLI and Python API for task scheduling.
- 🔌 Customizable plugin system.
- ⚡️ Effortless Parallelization: Distribute tasks across multiple GPU workers with just a few lines of code.
- 🛡️ Intelligent Failure Management: Automatically capture exceptions, retry failed tasks, and maintain detailed error logs.
- 🔄 Seamless Recovery: Resume failed experiments with a single command - no more scavenging through logs or directories.
- 🎯 Real-time Prioritization: Changed your mind about experiment settings? Instantly cancel, add, or reschedule tasks without disrupting existing ones.
- 🤖 Workflow Automation: Set up smart event triggers for email notifications or task workflow based on FSM transition events.
- 📊 Streamlined Logging: All stdout/stderr automatically organized in
.labtasker/logs- zero configuration required. - 🧩 Extensible Plugin System: Create custom command combinations or leverage community plugins to extend functionality.
Note
You need a running Labtasker server to use the client tools. See deployment instructions.
pip install labtaskerpip install git+https://github.com/fkcptlst/labtasker.gitUse the following command to quickly setup a labtasker queue for your project:
labtasker initThen, use labtasker submit to submit tasks and use labtasker loop to run tasks across any number of workers.
For detailed information on demo, tutorial, deployment, usage, please refer to the documentation.
See LICENSE for details.