-
Notifications
You must be signed in to change notification settings - Fork 547
[Golang] Add restart functionality to go runtime #3447
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Regarding you general question - what metrics would you want to calculate if the runtime is restarted during event processing?
Perhaps only success/failure which are counted in the worker, so I think that's already handled.
@TomerShor I meant these metrics:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The code looks good.
But we need to be extra sure we don't introduce any delays when changing the event processing flows.
if currentStatus := g.GetStatus(); currentStatus != status.Ready { | ||
return nil, errors.Errorf("Runtime not ready (current status: %s)", currentStatus) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding condition to a critical flow might cause performance hits.
Can you please run benchmark tests to compare the regular flow of with and without the changes?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, I will do benchmarking.
But anyway,
- we already have this condition in other runtimes
- we will need this condition when we start implementing explicitAck/Termination for go runtime
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@TomerShor results:
1000 total/100 parall:
before: 888.37955ms
after: 841.544612ms
10000 total/100 parall:
before: 955.145597ms
after: 894.964659ms
1000 total/1000 parall:
before: 4.743075748s
after: 4.772192623s
10000 total/1000 parall:
before: 8.847363427s
after: 8.617966213s
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
probably, moving of duration metrics calculation could impact performance, so we can see insignificant improve in some cases
Adds restart functionality to go runtime which is crucial to have for kafka runtime to avoid having busy worker after runtime restart. Question to address: Should we add metrics calculation after restart? Because gorotine will still alive on the background for now (We might consider adding some timeout in the future like maxTimeOfEventProcessing), so technically we will have metrics there. jira - https://iguazio.atlassian.net/browse/NUC-338
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🚀
Adds restart functionality to go runtime which is crucial to have for kafka runtime to avoid having busy worker after runtime restart.
Question to address: Should we add metrics calculation after restart? Because gorotine will still alive on the background for now (We might consider adding some timeout in the future like maxTimeOfEventProcessing), so technically we will have metrics there.
jira - https://iguazio.atlassian.net/browse/NUC-338