Open
Description
We need the ability to group together a number of related small tasks to run as a single batch job - but still have the suite manages the tasks as separate entities.
Quick points:
- Redesign suite
log/job/file system to handle this. - Allow user to configure job run time independent of tasks, but with tasks being able to subscribe themselves to jobs.
- Allow grouping of independent but similar tasks (e.g. a parallel group of tasks, each of which requires similar amount of resources) and grouping of tasks in a sub-graph.
- Change logic and API to remove the one-task-to-many-jobs assumption. E.g. What do we do if we want to poll, kill or trigger a task/job?
- Handle retry of some task failures in a job.
See also: