8000 [Golang] Add restart functionality to go runtime by rokatyy · Pull Request #3447 · nuclio/nuclio · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

[Golang] Add restart functionality to go runtime #3447

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Jan 22, 2025

Conversation

rokatyy
Copy link
Contributor
@rokatyy rokatyy commented Jan 9, 2025

Adds restart functionality to go runtime which is crucial to have for kafka runtime to avoid having busy worker after runtime restart.

Question to address: Should we add metrics calculation after restart? Because gorotine will still alive on the background for now (We might consider adding some timeout in the future like maxTimeOfEventProcessing), so technically we will have metrics there.

jira - https://iguazio.atlassian.net/browse/NUC-338

@rokatyy rokatyy marked this pull request as ready for review January 9, 2025 15:43
@rokatyy rokatyy requested a review from TomerShor January 9, 2025 15:43
Copy link
Contributor
@TomerShor TomerShor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding you general question - what metrics would you want to calculate if the runtime is restarted during event processing?
Perhaps only success/failure which are counted in the worker, so I think that's already handled.

@rokatyy
Copy link
Contributor Author
rokatyy commented Jan 9, 2025

@TomerShor I meant these metrics:

g.Statistics.DurationMilliSecondsSum += uint64(callDuration.Nanoseconds() / 1000000)
g.Statistics.DurationMilliSecondsCount++

@rokatyy rokatyy requested a review from TomerShor January 13, 2025 08:02
Copy link
Contributor
@TomerShor TomerShor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks good.
But we need to be extra sure we don't introduce any delays when changing the event processing flows.

Comment on lines +133 to +135
if currentStatus := g.GetStatus(); currentStatus != status.Ready {
return nil, errors.Errorf("Runtime not ready (current status: %s)", currentStatus)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding condition to a critical flow might cause performance hits.
Can you please run benchmark tests to compare the regular flow of with and without the changes?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I will do benchmarking.

But anyway,

  1. we already have this condition in other runtimes
  2. we will need this condition when we start implementing explicitAck/Termination for go runtime

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TomerShor results:

1000 total/100 parall:
before: 888.37955ms
after: 841.544612ms

10000 total/100 parall:
before: 955.145597ms
after: 894.964659ms

1000 total/1000 parall:
before: 4.743075748s
after: 4.772192623s

10000 total/1000 parall:
before: 8.847363427s
after: 8.617966213s

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably, moving of duration metrics calculation could impact performance, so we can see insignificant improve in some cases

@rokatyy rokatyy requested a review from TomerShor January 16, 2025 08:54
@TomerShor TomerShor requested a review from liranbg January 19, 2025 10:18
@TomerShor TomerShor merged commit 6e64d16 into nuclio:development Jan 22, 2025
14 checks passed
rokatyy added a commit to rokatyy/nuclio that referenced this pull request Jan 22, 2025
Adds restart functionality to go runtime which is crucial to have for
kafka runtime to avoid having busy worker after runtime restart.

Question to address: Should we add metrics calculation after restart?
Because gorotine will still alive on the background for now (We might
consider adding some timeout in the future like
maxTimeOfEventProcessing), so technically we will have metrics there.

jira - https://iguazio.atlassian.net/browse/NUC-338
Copy link
Contributor
@liranbg liranbg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants
0