Enable Top Clients/Files Metrics in Harvest #3130

rahulguptajss · 2024-09-02T08:54:18Z

rahulguptajss
Sep 2, 2024
Collaborator

Top Clients collection is disabled by default. To enable Top Clients Tracking in Harvest, follow these steps. Note that Top Clients support is available only through the RestPerf Collector in ONTAP version 9.13.1 and later. The collection is enabled in the RestPerf Collector Volume template via a plugin VolumeTopClients, as detailed below.

Note: Available from Harvest 24.11.

Note: Top File Metrics are available in Harvest from 24.11.1 via same plugin. Refer here for more details.

NABox4 and Harvest 25.05

⚠️ Note:
Prerequisite: ssh to NABox via admin
Create relevant directories if they don't exist. You need to run these commands in sudo bash

Navigate to the restperf directory.

cd /etc/nabox/harvest/user/restperf

Create custom.yaml:

vi custom.yaml

Update the file with the following content:

objects:
  Volume: custom_volume.yaml

Create a custom_volume.yaml file in the /etc/nabox/harvest/user/restperf/9.12.0 directory. Update custom_volume.yaml as below.

name:                     Volume
query:                    api/cluster/counter/tables/volume
object:                   volume

counters:
  - ^^uuid
  - ^name                  => volume
  - ^node.name             => node
  - ^parent_aggregate      => aggr
  - ^svm.name              => svm
  - average_latency        => avg_latency
  - bytes_read             => read_data
  - bytes_written          => write_data
  - nfs.access_latency     => nfs_access_latency
  - nfs.access_ops         => nfs_access_ops
  - nfs.getattr_latency    => nfs_getattr_latency
  - nfs.getattr_ops        => nfs_getattr_ops
  - nfs.lookup_latency     => nfs_lookup_latency
  - nfs.lookup_ops         => nfs_lookup_ops
  - nfs.other_latency      => nfs_other_latency
  - nfs.other_ops          => nfs_other_ops
  - nfs.punch_hole_latency => nfs_punch_hole_latency
  - nfs.punch_hole_ops     => nfs_punch_hole_ops
  - nfs.read_latency       => nfs_read_latency
  - nfs.read_ops           => nfs_read_ops
  - nfs.setattr_latency    => nfs_setattr_latency
  - nfs.setattr_ops        => nfs_setattr_ops
  - nfs.total_ops          => nfs_total_ops
  - nfs.write_latency      => nfs_write_latency
  - nfs.write_ops          => nfs_write_ops
  - other_latency
  - read_latency
  - total_ops
  - total_other_ops        => other_ops
  - total_read_ops         => read_ops
  - total_write_ops        => write_ops
  - write_latency

plugins:
  - MetricAgent:
      compute_metric:
        - total_data ADD bytes_read bytes_written
  - Aggregator:
      # plugin will create summary/average for each object
      # any names after the object names will be treated as label names that will be added to instances
      - node
  - Volume:
      include_constituents: false

  - LabelAgent:
      # Ignore transient volumes, e.g. SnapProtect, SnapManager, SnapCenter, CommVault, Clone, and Metadata volumes
      exclude_regex:
        - volume `.+_CVclone`
        - volume `.+(0[1-9]|[12][0-9]|3[01])(0[1-9]|1[012])\d\d[0-9]{6}`
        - volume `cl_.+_(19|20)\d\d(0[1-9]|1[012])( 0[1-9]|[12][0-9]|3[01])[0-9]{6}`
        - volume `sdw_cl_.+`
        - volume `MDV_CRS_.+`
        - volume `MDV_aud_.+`

  - VolumeTopClients:
      # `max_volumes` is the maximum number of volumes to consider for top client metrics. This value is capped at 50, even if a higher number is specified.
      # When enabled, this plugin will collect read/write operations and throughput metrics for the top clients of each volume.
      - schedule:
          - data: 1h # This value should be a multiple of the poll duration. By default, Harvest will check once an hour to see how many volumes have activity_tracking.state set to on.
      - max_volumes: 5
      - objects:
          - client  # collect read/write operations and throughput metrics for the top clients.
          - file    # collect read/write operations and throughput metrics for the top files.

export_options:
  instance_keys:
    - aggr
    - node
    - style
    - svm
    - volume

Restart harvest

rpm/deb/native/container

Note:

Below steps are applicable for rpm/deb/native/container installation. In this example, let's assume that Harvest is installed in /opt/harvest

Navigate to the rest directory:

cd /opt/harvest/conf/restperf

Create custom.yaml:

vi custom.yaml

Update the file with the following content:

objects:
  Volume: custom_volume.yaml

Create a custom_volume.yaml file in the /opt/harvest/conf/restperf/9.12.0/ directory. Update custom_volume.yaml as below.

name:                     Volume
query:                    api/cluster/counter/tables/volume
object:                   volume

counters:
  - ^^uuid
  - ^name                  => volume
  - ^node.name             => node
  - ^parent_aggregate      => aggr
  - ^svm.name              => svm
  - average_latency        => avg_latency
  - bytes_read             => read_data
  - bytes_written          => write_data
  - nfs.access_latency     => nfs_access_latency
  - nfs.access_ops         => nfs_access_ops
  - nfs.getattr_latency    => nfs_getattr_latency
  - nfs.getattr_ops        => nfs_getattr_ops
  - nfs.lookup_latency     => nfs_lookup_latency
  - nfs.lookup_ops         => nfs_lookup_ops
  - nfs.other_latency      => nfs_other_latency
  - nfs.other_ops          => nfs_other_ops
  - nfs.punch_hole_latency => nfs_punch_hole_latency
  - nfs.punch_hole_ops     => nfs_punch_hole_ops
  - nfs.read_latency       => nfs_read_latency
  - nfs.read_ops           => nfs_read_ops
  - nfs.setattr_latency    => nfs_setattr_latency
  - nfs.setattr_ops        => nfs_setattr_ops
  - nfs.total_ops          => nfs_total_ops
  - nfs.write_latency      => nfs_write_latency
  - nfs.write_ops          => nfs_write_ops
  - other_latency
  - read_latency
  - total_ops
  - total_other_ops        => other_ops
  - total_read_ops         => read_ops
  - total_write_ops        => write_ops
  - write_latency

plugins:
  - MetricAgent:
      compute_metric:
        - total_data ADD bytes_read bytes_written
  - Aggregator:
      # plugin will create summary/average for each object
      # any names after the object names will be treated as label names that will be added to instances
      - node
  - Volume:
      include_constituents: false

  - VolumeTopClients:
      # The maximum number of volumes to consider for top client metrics.
      # The actual maximum value is capped at 50, even if a higher number is specified.
      # When enabled, this plugin will collect read/write operations and throughput metrics for the top clients of each volume.
      - schedule:
          - data: 1h  # This value should be a multiple of the poll duration. By default, Harvest will check once an hour to see how many volumes have activity_tracking.state set to on.
      - max_volumes: 5


export_options:
  instance_keys:
    - aggr
    - node
    - style
    - svm
    - volume

Restart harvest

In the volume dashboard under the clients panel, you should see something similar to the following:

mamoep · 2024-09-10T09:37:46Z

mamoep
Sep 10, 2024

How are the volumes chosen? Let's say I have 20 volumes and set a limit of 5. Which volumes will be reported?

0 replies

rahulguptajss · 2024-09-10T10:03:04Z

rahulguptajss
Sep 10, 2024
Collaborator Author

For example, for read IOPS, we will select the top 5 volumes based on descending order of read IOPS in each performance poll. The same approach applies to write IOPS, as well as read and write throughput.

0 replies

WayneShen2 · 2024-11-08T05:00:41Z

WayneShen2
Nov 8, 2024

I have enable it, but I can not see any data. Have any configure must enable from Filer end (ex: Activity Tracking)

5 replies

rahulguptajss Nov 8, 2024
Collaborator Author

I have enable it, but I can not see any data. Have any configure must enable from Filer end (ex: Activity Tracking)

Yes activity tracking must be enabled for relevant volumes.

WayneShen2 Nov 8, 2024

Thanks for your confirmed.

wooyoungAhn Nov 21, 2024

I can not see any data..

time=2024-11-21T05:33:51.153Z level=ERROR source=collector.go:483 msg="" Poller=CVO collector=RestPerf:Volume error="failed to fetchAll href=api/storage/volumes/*/top-metrics/clients..., hrefLength=168 err=error making request StatusCode: 400, Message: Volume wildcard queries are not supported for activity tracking reports., Code: 124518417, API: /api/storage/volumes/*/top-metrics/clients?fields=client_ip%2Ciops.write%2Csvm%2Cvolume.name&max_records=500&return_records=true&svm=svm_CVO1&top_metric=iops.write&volume=vol2" plugin=VolumeTopClients

rahulguptajss Nov 21, 2024
Collaborator Author

Could you share ONTAP version?

rahulguptajss Nov 21, 2024
Collaborator Author

I have updated the instructions to specify that ONTAP version 9.13.1 or higher is required for TopClients to function. This is because the wildcard queries used in TopClients are only supported starting from version 9.13.1 onwards.

kwinkelman-bw · 2025-02-06T13:32:33Z

kwinkelman-bw
Feb 6, 2025

Is there any intent to expose this configuration in the NAbox UI? It seems like a "feature" and not necessarily a "customization" to me.

0 replies

Enable Top Clients/Files Metrics in Harvest #3130

Uh oh!

Uh oh!

rahulguptajss Sep 2, 2024 Collaborator

Replies: 4 comments · 5 replies

Uh oh!

mamoep Sep 10, 2024

Uh oh!

rahulguptajss Sep 10, 2024 Collaborator Author

Uh oh!

WayneShen2 Nov 8, 2024

Uh oh!

rahulguptajss Nov 8, 2024 Collaborator Author

Uh oh!

WayneShen2 Nov 8, 2024

Uh oh!

wooyoungAhn Nov 21, 2024

Uh oh!

rahulguptajss Nov 21, 2024 Collaborator Author

Uh oh!

rahulguptajss Nov 21, 2024 Collaborator Author

Uh oh!

kwinkelman-bw Feb 6, 2025

rahulguptajss
Sep 2, 2024
Collaborator

Replies: 4 comments 5 replies

mamoep
Sep 10, 2024

rahulguptajss
Sep 10, 2024
Collaborator Author

WayneShen2
Nov 8, 2024

rahulguptajss Nov 8, 2024
Collaborator Author

rahulguptajss Nov 21, 2024
Collaborator Author

rahulguptajss Nov 21, 2024
Collaborator Author

kwinkelman-bw
Feb 6, 2025