25.02.0 #3473

cgrinds · 2025-02-13T14:22:56Z

cgrinds
Feb 13, 2025
Maintainer

25.02.0 / 2025-02-13 Release

📌 Highlights of this major release include:

⭐ New Features

⭐ The Volume dashboard was updated to clarify that volume latencies are missing some latencies from NAS protocols. Use the workload volume metrics in the QoS row for a more detailed breakdown. Thanks to MatthiasS for reporting.
All Harvest dashboards default to Datacenter=All instead of the first datacenter in the list. Thanks to @roybatty2019 for reporting.
Harvest provides a FIPS 140-3 compliant container image, available as a separate image at ghcr.io/netapp/harvest:25.02.0-1-fips.
🌾 Harvest bin/grafana import
- Supports nested Grafana folders. Thanks to @IvanZenger for reporting.
- Supports setting variables' default values during import. See #3384 for details. Thanks to @mamoep for reporting.
Harvest collects shelf firmware versions and shows them in the Shelf dashboard, Module row. Thanks to @summertony15 for reporting.
⭐ Several of the existing dashboards include new panels in this release:
- The Disk dashboard includes a Top Disk and Tape Drives Throughput by Host Adapter panel. Thanks to Amir for reporting.
- The Datacenter and Data Protection dashboards were updated with data protection buckets and policy rows.
The volumes templates exclude transient volumes by default. Thanks to Yann for reporting.
Harvest collects rewind context (rwctx) metrics for ONTAP 9.16.0 and later. Thanks to @shawnahall71 for reporting.
📕 Documentation additions
- Document Podman Quadlet as a deployment option. Thanks to ttlexceeded for reporting.
- Describe how to use a Go binary as a credential script for Harvest. Thanks to AdiZ for reporting.

🚀 Performance Improvements

RestPerf collector uses less memory by streaming results.

In case you missed the previous 24.11.1 dot release, here are the features included in it:

Performance Improvements in 24.11.1

Significant memory footprint improvements for the REST collector. More details here. Thanks to @ryan for reporting it.
Reduced memory footprint by using streaming in the REST collector.

New Features in 24.11.1

Harvest supports Top files metrics collection. More details here.
Volume and Cluster tags are supported via Volume and Cluster dashboards.
Field Replaceable Unit (FRU) details have been added to the power dashboard.
Track ONTAP image update progress for a cluster via the Cluster dashboard. Thanks to @knappmi for reporting it.
prom_port is now supported within the poller. More details here.
We've fixed an intermittent latency/operations spike issue in the plugin-generated Harvest performance metrics. Thanks to @wooyoungAhn for reporting it.

Announcements

‼️ IMPORTANT Harvest version 25.02.0 disables the out-of-the-box Qtree templates because of reported ONTAP slowdowns when collecting a large number of qtree objects. If you want to enable the Qtree templates, please see these instructions.

‼️ IMPORTANT Harvest version 25.02.0 removes the WorkloadDetail and WorkloadDetailVolume templates and all dashboard panels that use them. These templates are removed because they are expensive to collect and currently there is no way to collect them from ONTAP without introducing an unacceptable amount of skew in the results. See #3423 for details.

‼️ IMPORTANT If using Docker Compose and you want to keep your historical Prometheus data, please read how to migrate your Prometheus volume

💡 IMPORTANT After upgrade, don't forget to re-import your dashboards, so you get all the new enhancements and fixes. You can import them via the 'bin/harvest grafana import' CLI, from the Grafana UI, or from the 'Maintenance > Reset Harvest Dashboards' button in NAbox3. For NAbox4, this step is not needed.

Known Issues

💡 IMPORTANT FSx ZapiPerf workload collector fails to collect metrics, please use RestPerf instead.

Thanks to all the awesome contributors

🤘 Thanks to all the people who've opened issues, asked questions on Discord, and contributed code or dashboards this release:

@Falcon667, @IvanZenger, @cheese1, @embusalacchi, @mamoep, @roybatty2019, @summertony15, AdiZ, Amir, MatthiasS, Yann, ttlexceeded

🌱 This release includes 18 features, 22 bug fixes, 8 documentation, 1 performance, 1 refactoring, 6 miscellaneous, and 12 ci pull requests.

🚀 Features

Hide Transient Volumes (#3337)
Adding Ifgrp Api To Fetch Ifgrp Labels In Net_port (#3342)
Adding Rwctx Template For Restperf In 9.16.0 (#3349)
Add Disk And Tape Drives Throughput By Host Adapter (#3372)
Adding The Bucket And Policy Rest Template (#3374)
Include Lun And Namespace Templates In Keyperf (#3379)
Dashboard Variables Set Default Value On Import (#3399)
Optimize Workload_detail And Workload_detail_volume Through Delay Center Filter (#3406)
Handled Empty Admin Svm In Plugin Call (#3410)
Support Metric For Module Type In Frus (#3411)
Harvest Grafana Import Should Support Nested Grafana Folders (#3417)
Remove Workload Detail Templates (#3433)
Add Streaming To Keyperf Collector (#3435)
Negative Metrics Spike Handling (#3439)
Disabled Qtree Perf Template And Update Docs (#3445)
Harvest Dashboards Should Default To Datacenter=All (#3448)
Update Qos Row In Volume Dashboard (#3453)
Improve Cp Summary In Disk Dashboard (#3456)

🐛 Bug Fixes

Include Instances Generated By Inbuilt Plugins In Plugininstances Log (#3343)
Handled Duplicate Key In Securityauditdestination (#3348)
Harvest Permissions Should Include Fru (#3354)
No Instances Handling In Rest Collector (#3358)
Rest No Instance Handling (#3360)
Don't Clear Performance Volume Cache When There Is An Error (#3361)
Failed To Find Scanner Instance In Cache Zapiperf (#3366)
Installation Broken On Debian 11 Bullseye (#3368)
Changed Var Label To Ne Null From Empty (#3385)
Update Snapshot Policy Endpoint (#3391)
Update Export Rule Endpoint (#3392)
Upgrade Golang.org/X/Net Due To Dependabot Alert (#3395)
Update Export Rule Endpoint (#3396)
Remove Redundant Label From Node Template (#3404)
Enable Request/Response Logging For Restperf (#3408)
Disable Cache To Avoid Cache Poisoning Attack (#3409)
Duplicate Time Series In Volume Dashboard (#3418)
Update Tr Link In Security Dashboard (#3419)
Typo (#3425)
Handle Only Labels In Zapi Snapshotpolicy (#3444)
"Top Ethernet Ports By Utilization %" Panel Legend Should Not In… (#3451)
Handle Cp Labels In Dashboard (#3455)

📕 Documentation

Fix Release Announcements (#3330)
Keyperf Documentation (#3345)
Updating Doc For Custom.yaml (#3352)
Rest Endpoint Permissions (#3359)
Add Go Binary Steps For Credential Script (#3381)
Fix Alignment Of Template (#3421)
Document Podman Quadlet As A Deployment Option (#3442)
Add Description About Cp In Disk Dashboard (#3454)

⚡ Performance

Restperf Should Stream Results While Parsing (#3356)

Refactoring

Remove Openapi Dependency (#3436)

Miscellaneous

Merge 24.11.1 To Main (#3328)
Update Module Github.com/Shirou/Gopsutil/V4 To V4.24.11 (#3347)
Update All Dependencies (#3365)
Update All Dependencies (#3407)
Update All Dependencies (#3450)
Update All Dependencies (#3457)

🔨 CI

Fix Go Vet Errors (#3331)
Add Misspell, Nakedret, And Predeclared Linter (#3350)
Bump Go (#3351)
Add Role Permissions Validation (#3375)
Enable More Linters (#3413)
Add Workflow Permissions At Codeql Recommendation (#3426)
Add Workflow Permissions At Codeql Recommendation (#3427)
Pin Actions To Sha (#3428)
Cheese1 Has Signed The Ccla (#3429)
Bump Go (#3430)
Ci-Local Requires Passing Admin Argument (#3431)
Bump Go (#3452)

This discussion was created from the release 25.02.0.

WayneShen2 · 2025-02-18T03:18:11Z

WayneShen2
Feb 18, 2025

After upgraded, Volume dashboard highlights no data if query 6 hours

But use 30 mins we can see data

1 reply

rahulguptajss Feb 18, 2025
Collaborator

If you click on these red icons in the corner in the first screenshot, does it show any error messages? From which version did you upgrade to 25.02?

WayneShen2 · 2025-02-18T05:54:00Z

WayneShen2
Feb 18, 2025

I upgrade from 24.11.1
6h error

2 days is normal

0 replies

rahulguptajss · 2025-02-18T07:28:46Z

rahulguptajss
Feb 18, 2025
Collaborator

Thanks. There were 2 labels added on 24.11 for volume_labels: is_space_reporting_logical and is_space_enforcement_logical. Based on the error, the two time series with and without these labels are causing this many-to-many issue. Let's focus on the Volume Average Latency panel. If you update its panel query to the one below, does it fix the error for that panel?

avg(
  topk(
    $TopResources,
    (
      avg by (datacenter, cluster, svm, volume) (
        volume_avg_latency{datacenter=~"$Datacenter", cluster=~"$Cluster", svm=~"$SVM", volume=~"$Volume", style!="flexgroup_constituent"}
      )
    )
    * on (datacenter, cluster, svm, volume) group_left(node)
    (
      max by (datacenter, cluster, svm, volume, node) (
        volume_labels{datacenter=~"$Datacenter", cluster=~"$Cluster", svm=~"$SVM", volume=~"$Volume", tags=~".*$Tag.*"}
      )
    )
  )
)

0 replies

WayneShen2 · 2025-02-18T08:40:00Z

WayneShen2
Feb 18, 2025

That is working for Volume Average Latency.

I could see performance also no data

0 replies

rahulguptajss · 2025-02-18T08:43:30Z

rahulguptajss
Feb 18, 2025
Collaborator

Thanks. This will be fixed for all panels via issue #3480

0 replies

WayneShen2 · 2025-02-18T23:49:35Z

WayneShen2
Feb 18, 2025

If we do not change after 1days later that will normal.

0 replies

rahulguptajss · 2025-02-19T05:54:12Z

rahulguptajss
Feb 19, 2025
Collaborator

Yes, That is expected. We'll share fix for this asap.

0 replies

rahulguptajss · 2025-02-19T08:58:50Z

rahulguptajss
Feb 19, 2025
Collaborator

@WayneShen2 could you please import the JSON file into grafana at: https://raw.githubusercontent.com/NetApp/harvest/185c8f681bff83f7ebff7b0469d54ecb9e462458/grafana/dashboards/cmode/volume.json and verify if it resolves the issue with the volume dashboard?

0 replies

WayneShen2 · 2025-02-20T00:33:36Z

WayneShen2
Feb 20, 2025

Thank you. New dashboard is working.

1 reply

rahulguptajss Feb 20, 2025
Collaborator

Thank you. New dashboard is working.

Great! Fix is also available in nightly release.

WayneShen2 · 2025-03-06T00:49:25Z

WayneShen2
Mar 6, 2025

I have found another issue, after upgraded, i could not see scanner data (ex: latency)

0 replies

Hardikl · 2025-03-06T08:03:10Z

Hardikl
Mar 6, 2025
Collaborator

@WayneShen2

Yes, There is a bug #3500 found in the vscan object last week, which is already fixed now.
The changes are in the latest nightly build if you want to try them out.

2 replies

WayneShen2 Mar 7, 2025

Thanks, @Hardikl
That is fixed, after upgrade to nightly

Hardikl Mar 10, 2025
Collaborator

Thanks for the validation.

25.02.0 #3473

Uh oh!

cgrinds Feb 13, 2025 Maintainer

25.02.0 / 2025-02-13 Release

⭐ New Features

🚀 Performance Improvements

Performance Improvements in 24.11.1

New Features in 24.11.1

Announcements

Known Issues

Thanks to all the awesome contributors

🚀 Features

🐛 Bug Fixes

📕 Documentation

⚡ Performance

Refactoring

Miscellaneous

🔨 CI

Replies: 11 comments · 4 replies

Uh oh!

WayneShen2 Feb 18, 2025

Uh oh!

rahulguptajss Feb 18, 2025 Collaborator

Uh oh!

WayneShen2 Feb 18, 2025

Uh oh!

rahulguptajss Feb 18, 2025 Collaborator

Uh oh!

WayneShen2 Feb 18, 2025

Uh oh!

rahulguptajss Feb 18, 2025 Collaborator

Uh oh!

WayneShen2 Feb 18, 2025

Uh oh!

rahulguptajss Feb 19, 2025 Collaborator

Uh oh!

Uh oh!

rahulguptajss Feb 19, 2025 Collaborator

Uh oh!

WayneShen2 Feb 20, 2025

Uh oh!

rahulguptajss Feb 20, 2025 Collaborator

Uh oh!

WayneShen2 Mar 6, 2025

Uh oh!

Hardikl Mar 6, 2025 Collaborator

Uh oh!

WayneShen2 Mar 7, 2025

Uh oh!

Hardikl Mar 10, 2025 Collaborator

cgrinds
Feb 13, 2025
Maintainer

Replies: 11 comments 4 replies

WayneShen2
Feb 18, 2025

rahulguptajss Feb 18, 2025
Collaborator

WayneShen2
Feb 18, 2025

rahulguptajss
Feb 18, 2025
Collaborator

WayneShen2
Feb 18, 2025

rahulguptajss
Feb 18, 2025
Collaborator

WayneShen2
Feb 18, 2025

rahulguptajss
Feb 19, 2025
Collaborator

rahulguptajss
Feb 19, 2025
Collaborator

WayneShen2
Feb 20, 2025

rahulguptajss Feb 20, 2025
Collaborator

WayneShen2
Mar 6, 2025

Hardikl
Mar 6, 2025
Collaborator

Hardikl Mar 10, 2025
Collaborator