Description
Describe the bug
This asset is running Numaplane 0.22.0.
Unfortunately, it seems if I'm reading the logs correctly that the user was able to delete the ISBServiceRollout and its finalizer got removed prior to the child isbsvc having its Finalizer removed, so what the user saw was the whole thing removed from ArgoCD UI even though the child isbsvc was still there. I'm not sure why this would happen given that we always check whether there are any isbsvc children still there before we remove the ISBServiceRollout finalizer.
Splunk logs:
This shows occurrences of "Deleting ISBServiceRollout" in the Numaplane log for this ISBServiceRollout. It indicates that it retried a bunch of times between 3:19 and 3:24 because the child isbsvc was still there but must have succeeded at 3:24 because there are no lines after that.
(UPDATE: now I'm thinking that maybe at 3:19 and 3:24 those were separate deletion events, since any requeuing should be at a 5 second interval and I don't see any log lines between 3:20 and 3:24 actually)
This Numaflow log shows the occurrences of "Deleting ISB Service" and this log shows the occurrences of "Failed to uninstall", when it couldn't remove the finalizer from isbsvc.
To me, it looks like they tried deleting their isbsvc itself at 3:10 which kept retrying until 3:32 when it succeeded (note the time between retries after errors in the controller framework increases over time due to exponential backoff algorithm). So, it seems from this that they were actually trying to delete their isbsvc first.
Timeline
This is the best I can gather of the timeline:
3:10: user tries to delete their isbsvc but it doesn't delete
3:19: user tries to delete their ISBServiceRollout but it doesn't delete immediately
3:24: ISBServiceRollout finalizer removed by Numaplane and ISBServiceRollout deleted even though child isbsvc is still there
3:32: isbsvc finalizer removed and isbsvc deleted
Message from the maintainers:
Impacted by this bug? Give it a 👍. We often sort issues this way to know what to prioritize.