Make reporting more metrics for tasks possible #3343
Conversation
| @@ -66,6 +66,9 @@ def handle_task_disabled(self, task, config): | |||
| def handle_task_done(self, task): | |||
| pass | |||
|
|
|||
| def handle_task_statistics(self, task, statistics): | |||
| pass | |||
There was a problem hiding this comment.
Is there a common implementation that can be introduced here? Or within the Datadog and/or Prometheus implementations of MetricsCollector?
If not, and this method needs to remain as an optionally-implemented method, i would ask at a minimum for there to be appropriately informative and discoverable documentation written surrounding its intended use. (I don't believe there are currently any docs/ content around metrics collectors).
There was a problem hiding this comment.
This method is entirely optional. I had not provided any documentation and tests yet, because I was hoping to gather some feedback in #3342 as you suggested in my earlier pull request (#3333 (comment)).
I will gladly add tests and documentation if you are convinced this change is small enough.
|
@mateka , a few PRs were merged today which alter the build and testing process for Luigi, We tried to update this branch with those modification. However, the CI trigger didn't appear to actually kickoff the Github Action workflow. Would you be able to make an adjustment to this branch and push a change so that we can see the Github Action behavior? |
97976c5 to
bbf7cb0
Compare
I pushed new version of this branch: rebased on current master with cherry-picked commit and it works now. I also added some documentation and test for new feature. |
I would like to collect and monitor more metrics from luigi graphs runs.
Description
I added method for passing tasks' run statistics from TaskContext into scheduler and metrics collector.
Motivation and Context
Example and motivation is in Issue #3342.
Have you tested this? If so, how?
I tested in on example graph and it worked.