8000 [DOCS-1453] Docs for run status change automations by mdlinville · Pull Request #1329 · wandb/docs · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

[DOCS-1453] Docs for run status change automations #1329

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 20 commits into from
Jul 10, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/images/automations/run_status_change.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/images/automations/webhooks.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 3 additions & 3 deletions assets/scss/_variables_project.scss
Original file line number Diff line number Diff line change
Expand Up @@ -142,9 +142,9 @@ h1, h2, h3, h4, h5, h6 {
h1, .td-content h1 { font-size: 32px !important; }
h2, .td-content h2 { font-size: 24px !important; }
h3, .td-content h3 { font-size: 20px !important; }
h4, .td-content h4 { font-size: 16px !important; }
h5, .td-content h5 { font-size: 14px !important; }
h6, .td-content h6 { font-size: 12px !important; }
h4, .td-content h4 { font-size: 18px !important; }
h5, .td-content h5 { font-size: 17px !important; }
h6, .td-content h6 { font-size: 16px !important; }

// Lead paragraph
.td-content .lead {
Expand Down
7 changes: 5 additions & 2 deletions content/en/guides/core/automations/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,18 @@ For example, an automation can notify a Slack channel when a new version is crea
## Overview
An automation can start when a specific [event]({{< relref "automation-events.md" >}}) occurs in a registry or project.

For an artifact in a [Registry]({{< relref "/guides/core/registry/">}}), an automation can start:
In a [Registry]({{< relref "/guides/core/registry/">}}), an automation can start:
- When a new artifact version is linked to a collection. For example, trigger testing and validation workflows for new candidate models.
- When an alias is added to an artifact version. For example, trigger a deployment workflow when an alias is added to a model version.

For an artifact in a [project]({{< relref "/guides/models/track/project-page.md" >}}), an automation can start:
In a [project]({{< relref "/guides/models/track/project-page.md" >}}), an automation can start:
- When a new version is added to an artifact. For example, start a training job when a new version of a dataset artifact is added to a given collection.
- When an alias is added to an artifact version. For example, trigger a PII redaction workflow when the alias "redaction" is added to a dataset artifact.
- When a metric for a run meets or exceeds a configured threshold.
- When a metric for a run changes by a configured threshold.
- When a run's status changes to **Running**, **Failed**, or **Finished**.

Optionally filter runs by user or run name.

This diagram shows the relationship between automation events and actions.

Expand Down
47 changes: 34 additions & 13 deletions content/en/guides/core/automations/automation-events.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ This section describes the scopes and events for an automation in a [Registry]({
1. Navigate to the **Registry** App at https://wandb.ai/registry/.
1. Click the name of a registry, then view and create automations in the **Automations** tab.

![Screenshot of the Registry Automations tab with an automation](/images/automations/registry_automations_tab.png)

Learn more about [creating automations]({{< relref "create-automations/" >}}).

### Scopes
Expand All @@ -40,6 +42,8 @@ This section describes the scopes and events for an automation in a [project]({{
1. Navigate to your W&B project on the W&B App at `https://wandb.ai/<team>/<project-name>`.
1. View and create automations in the **Automations** tab.

![Screenshot of the Project Automations tab with an automation](/images/automations/project_automations_tab.png)

Learn more about [creating automations]({{< relref "create-automations/" >}}).

### Scopes
Expand All @@ -55,21 +59,34 @@ This section describes the events related to an artifact that can trigger an aut
- **Adding a new alias to a version of an artifact**: Trigger a specific step of your workflow when a new artifact version in a project or collection has a specific label or alias applied. For example, run a series of downstream processing steps when an artifact has the `test-set-quality-check` alias applied.

### Run events
From the project's **Automations** tab or directly from a line plot panel, you can create a run metric automation triggered by:
- A metric in a run's history.
- A [system metric]({{< relref "/guides/models/app/settings-page/system-metrics.md" >}}) such as `cpu`, which tracks the percentage of CPU utilization. W&B logs system metrics automatically every 15 seconds.
An automation can be triggered by a change in a [run's status]({{< relref "/guides/models/track/runs/#run-states" >}}) or a change in a [metric value]({{< relref "/guides/models/track/log/#what-data-is-logged-with-specific-wb-api-calls" >}}).

#### Run status change
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add a UI screenshot here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added below

{{% alert %}}
- Currently available only in [W&B Multi-tenant Cloud]({{< relref "/guides/hosting/#wb-multi-tenant-cloud" >}}).
- A run with **Killed** status cannot trigger an automation. This status indicates that the run was stopped forcibly by an admin user.
{{% /alert %}}

The notification can trigger a workflow when these events occur:
- **Run metrics threshold met**: When the average, minimum, or maximum logged value for a given metric meets the threshold you specify.
- **Run metrics change threshold met**: When the average, minimum, or maximum logged value for a given metric changes by the threshold you specify.
Trigger a workflow when a run changes its [status]({{< relref "/guides/models/track/runs/_index.md#run-states" >}}) to **Running**, **Finished**, or **Failed**. Optionally, you can further limit the runs that can trigger an automation by filtering by the user that started a run or the run's name.

To set up a run metric automation, you configure how to compare the metric's value with the threshold you specify. Your choices depend on the event type and on any filters you specify.
![Screenshot showing a run status change automation](/images/automations/run_status_change.png)

Because run status is a property of the entire run, you can create a run status automation only from the the **Automations** page, not from a workspace.

#### Run metrics change
{{% alert %}}
Run metric automations are currently available only in [W&B Multi-tenant Cloud]({{< relref "/guides/hosting/#wb-multi-tenant-cloud" >}}).
Currently available only in [W&B Multi-tenant Cloud]({{< relref "/guides/hosting/#wb-multi-tenant-cloud" >}}).
{{% /alert %}}

#### Threshold
Trigger a workflow based on a logged value for a metric, either a metric in a run's history or a [system metric]({{< relref "/guides/models/app/settings-page/system-metrics.md" >}}) such as `cpu`, which tracks the percentage of CPU utilization. W&B logs system metrics automatically every 15 seconds.

You can create a run metrics automation from the project's **Automations** tab or directly from a line plot panel in a workspace.

To set up a run metric automation, you configure how to compare the metric's value with the threshold you specify. Your choices depend on the event type and on any filters you specify.

Optionally, you can further limit the runs that can trigger an automation by filtering by the user that started a run or the run's name.

##### Threshold
For **Run metrics threshold met** events, you configure:
1. The window of most recently logged values to consider (defaults to 5).
1. Whether to evaluate the **Average**, **Min**, or **Max** value within the window.
Expand All @@ -83,7 +100,9 @@ For **Run metrics threshold met** events, you configure:

For example, trigger an automation when average `accuracy` is above `.6`.

#### Change threshold
![Screenshot showing a run metrics threshold automation](/images/automations/run_metrics_threshold_automation.png)

##### Change threshold
For **Run metrics change threshold met** events, the automation uses two "windows" of values to check whether to start:

- The _current window_ of recently logged values to consider (defaults to 10).
Expand All @@ -99,13 +118,15 @@ To create the automation, you configure:
- Increases by at least
- Decreases by at least
- Increases or decreases by at least

For example, trigger an automation when `loss` decreases by at least `.25`.

For example, trigger an automation when average `loss` decreases by at least `.25`.

![Screenshot showing a run metrics change threshold automation](/images/automations/run_metrics_change_threshold_automation.png)

#### Run filters
This section describes how the automation selects runs to evaluate.

- By default, any run in the project triggers the animation when the event occurs. To consider only specific runs, specify a run filter.
- By default, any run in the project triggers the automation when the event occurs. To consider only specific runs, specify a run filter.
- Each run is considered individually and can potentially trigger the automation.
- Each run's values are put into a separate window and compared to the threshold separately.
- In a 24 hour period, a particular automation can fire at most once per run.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@ A team admin can add a Slack integration to the team.
1. Log in to W&B and go to **Team Settings**.
1. In the **Slack channel integrations** section, click **Connect Slack** to add a new Slack instance. To add a channel for an existing Slack instance, click **New integration**.

If necessary, sign in to Slack in your browser. When prompted, grant W&B permission to post to the Slack channel you select. Read the page, then click **Search for a channel** and begin typing the channel name. Select the channel from the list, then click **Allow**.

![Screenshot showing two Slack integrations in a Team](/images/automations/slack_integrations.png)
1. If necessary, sign in to Slack in your browser. When prompted, grant W&B permission to post to the Slack channel you select. Read the page, then click **Search for a channel** and begin typing the channel name. Select the channel from the list, then click **Allow**.
1. In Slack, go to the channel you selected. If you see a post like `[Your Slack handle] added an integration to this channel: Weights & Biases`, the integration is configured correctly.

Now you can [create an automation]({{< relref "#create-an-automation" >}}) that notifies the Slack channel you configured.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ If the webhook requires a Bearer token or its payload requires a sensitive strin

If the test does not succeed, verify the webhook's configuration and try again. If necessary, refer to [Troubleshoot your webhook]({{< relref "#troubleshoot-your-webhook" >}}).

![Screenshot showing two webhooks in a Team](/images/automations/webhooks.png)

Now you can [create an automation]({{< relref "#create-a-webhook-automation" >}}) that uses the webhook.

## Create an automation
Expand Down Expand Up @@ -84,7 +86,6 @@ A W&B admin can create automations in a project.
{{% /tab %}}
{{< /tabpane >}}


## View and manage automations
{{< tabpane text=true >}}
{{% tab "Registry" %}}
Expand All @@ -97,7 +98,6 @@ From either of these pages, a Registry admin can manage existing automations:
- To edit an automation, click its action `...` menu, then click **Edit automation**.
- To delete an automation, click its action `...` menu, then click **Delete automation**. Confirmation is required.


{{% /tab %}}
{{% tab "Project" %}}
A W&B admin can view and manage a project's automations from the project's **Automations** tab.
Expand Down
Loading
0