-
Notifications
You must be signed in to change notification settings - Fork 28
[DOCS-1453] Docs for run status change automations #1329
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
20 commits
Select commit
Hold shift + click to select a range
4df9de2
[DOCS-1453] Docs for run status change automations
mdlinville 417ad21
Stephen's feedback
mdlinville aec67a3
Copyedits after rebase
mdlinville bd71a62
Merge remote-tracking branch 'origin/main' into DOCS-1453
mdlinville 5a3d4d9
Apply suggestions from code review
mdlinville 4b2a7f9
Update content/en/guides/core/automations/create-automations/webhook.md
mdlinville d8369d7
Merge remote-tracking branch 'origin/DOCS-1453' into DOCS-1453
mdlinville d5dec0f
Fix link
mdlinville 2064576
Adjust font size of H4-H6
mdlinville f856166
Merge branch 'main' into DOCS-1453
mdlinville bf02295
Merge remote-tracking branch 'origin/main' into DOCS-1453
mdlinville 5b3f28f
Add images to Event Scopes page
mdlinville c554c35
Add images to Slack and Webhook pages
mdlinville 863786c
Align image dimensions
mdlinville 90a4fd2
Optimised images with calibre/image-actions
github-actions[bot] 565609d
Crop some cruft from an image
mdlinville 6deafd0
Fix image filename
mdlinville 8962496
Merge remote-tracking branch 'origin/DOCS-1453' into DOCS-1453
mdlinville e512166
Optimised images with calibre/image-actions
github-actions[bot] 410d231
Merge branch 'main' into DOCS-1453
mdlinville File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+39.4 KB
assets/images/automations/run_metrics_change_threshold_automation.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -22,6 +22,8 @@ This section describes the scopes and events for an automation in a [Registry]({ | |
1. Navigate to the **Registry** App at https://wandb.ai/registry/. | ||
1. Click the name of a registry, then view and create automations in the **Automations** tab. | ||
|
||
 | ||
|
||
Learn more about [creating automations]({{< relref "create-automations/" >}}). | ||
|
||
### Scopes | ||
|
@@ -40,6 +42,8 @@ This section describes the scopes and events for an automation in a [project]({{ | |
1. Navigate to your W&B project on the W&B App at `https://wandb.ai/<team>/<project-name>`. | ||
1. View and create automations in the **Automations** tab. | ||
|
||
 | ||
|
||
Learn more about [creating automations]({{< relref "create-automations/" >}}). | ||
|
||
### Scopes | ||
|
@@ -55,21 +59,34 @@ This section describes the events related to an artifact that can trigger an aut | |
- **Adding a new alias to a version of an artifact**: Trigger a specific step of your workflow when a new artifact version in a project or collection has a specific label or alias applied. For example, run a series of downstream processing steps when an artifact has the `test-set-quality-check` alias applied. | ||
|
||
### Run events | ||
mdlinville marked this conversation as resolved.
Show resolved
Hide resolved
|
||
From the project's **Automations** tab or directly from a line plot panel, you can create a run metric automation triggered by: | ||
- A metric in a run's history. | ||
- A [system metric]({{< relref "/guides/models/app/settings-page/system-metrics.md" >}}) such as `cpu`, which tracks the percentage of CPU utilization. W&B logs system metrics automatically every 15 seconds. | ||
An automation can be triggered by a change in a [run's status]({{< relref "/guides/models/track/runs/#run-states" >}}) or a change in a [metric value]({{< relref "/guides/models/track/log/#what-data-is-logged-with-specific-wb-api-calls" >}}). | ||
|
||
#### Run status change | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We should add a UI screenshot here There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added below |
||
{{% alert %}} | ||
- Currently available only in [W&B Multi-tenant Cloud]({{< relref "/guides/hosting/#wb-multi-tenant-cloud" >}}). | ||
- A run with **Killed** status cannot trigger an automation. This status indicates that the run was stopped forcibly by an admin user. | ||
{{% /alert %}} | ||
|
||
The notification can trigger a workflow when these events occur: | ||
- **Run metrics threshold met**: When the average, minimum, or maximum logged value for a given metric meets the threshold you specify. | ||
- **Run metrics change threshold met**: When the average, minimum, or maximum logged value for a given metric changes by the threshold you specify. | ||
Trigger a workflow when a run changes its [status]({{< relref "/guides/models/track/runs/_index.md#run-states" >}}) to **Running**, **Finished**, or **Failed**. Optionally, you can further limit the runs that can trigger an automation by filtering by the user that started a run or the run's name. | ||
|
||
To set up a run metric automation, you configure how to compare the metric's value with the threshold you specify. Your choices depend on the event type and on any filters you specify. | ||
 | ||
|
||
Because run status is a property of the entire run, you can create a run status automation only from the the **Automations** page, not from a workspace. | ||
|
||
#### Run metrics change | ||
{{% alert %}} | ||
Run metric automations are currently available only in [W&B Multi-tenant Cloud]({{< relref "/guides/hosting/#wb-multi-tenant-cloud" >}}). | ||
Currently available only in [W&B Multi-tenant Cloud]({{< relref "/guides/hosting/#wb-multi-tenant-cloud" >}}). | ||
{{% /alert %}} | ||
|
||
#### Threshold | ||
Trigger a workflow based on a logged value for a metric, either a metric in a run's history or a [system metric]({{< relref "/guides/models/app/settings-page/system-metrics.md" >}}) such as `cpu`, which tracks the percentage of CPU utilization. W&B logs system metrics automatically every 15 seconds. | ||
|
||
You can create a run metrics automation from the project's **Automations** tab or directly from a line plot panel in a workspace. | ||
|
||
To set up a run metric automation, you configure how to compare the metric's value with the threshold you specify. Your choices depend on the event type and on any filters you specify. | ||
|
||
Optionally, you can further limit the runs that can trigger an automation by filtering by the user that started a run or the run's name. | ||
|
||
##### Threshold | ||
For **Run metrics threshold met** events, you configure: | ||
1. The window of most recently logged values to consider (defaults to 5). | ||
1. Whether to evaluate the **Average**, **Min**, or **Max** value within the window. | ||
|
@@ -83,7 +100,9 @@ For **Run metrics threshold met** events, you configure: | |
|
||
For example, trigger an automation when average `accuracy` is above `.6`. | ||
|
||
#### Change threshold | ||
 | ||
|
||
##### Change threshold | ||
md
10000
linville marked this conversation as resolved.
Show resolved
Hide resolved
mdlinville marked this conversation as resolved.
Show resolved
Hide resolved
|
||
For **Run metrics change threshold met** events, the automation uses two "windows" of values to check whether to start: | ||
|
||
- The _current window_ of recently logged values to consider (defaults to 10). | ||
|
@@ -99,13 +118,15 @@ To create the automation, you configure: | |
- Increases by at least | ||
- Decreases by at least | ||
- Increases or decreases by at least | ||
|
||
For example, trigger an automation when `loss` decreases by at least `.25`. | ||
|
||
For example, trigger an automation when average `loss` decreases by at least `.25`. | ||
|
||
 | ||
mdlinville marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
#### Run filters | ||
This section describes how the automation selects runs to evaluate. | ||
|
||
- By default, any run in the project triggers the animation when the event occurs. To consider only specific runs, specify a run filter. | ||
- By default, any run in the project triggers the automation when the event occurs. To consider only specific runs, specify a run filter. | ||
- Each run is considered individually and can potentially trigger the automation. | ||
- Each run's values are put into a separate window and compared to the threshold separately. | ||
- In a 24 hour period, a particular automation can fire at most once per run. | ||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.