daemon: Cleanup after a failed start #32673
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
If the daemon start failed due to e.g. invalid configuration the cleanup was not
performed as it was only done from the stop function that was not invoked since
the daemon start had failed.
This in some situations lead to confusing crashes as hive did stop all the dependencies
of the legacy daemon, for example this crash was reported:
level=fatal msg="failed to start: daemon creation failed: failed to detect devices: ...
... bunch of "stopping" messages omitted ...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1dfa041]
goroutine 460 [running]:
github.com/cilium/cilium/pkg/node.(*LocalNodeStore).Observe(0x3f4cfb0? ...
<autogenerated>:1 +0x21
The crash was caused by the LocalNodeStore stopping and clearing the global variable
in the node package (to make sure for example our tests don't leak data across them
and to make sure we do cleanup correctly).
To fix this, run the cleanup functions also when "newDaemon" or "startDaemon" fails
with an error.
Fixes: f21f303 ("daemon: bubble up error from startDaemon instead of fataling")
Signed-off-by: Jussi Maki <jussi@isovalent.com>
3 tasks
|
/test |
ldelossa
approved these changes
May 23, 2024
|
👋 @joamaki Backporter here. Double-checking whether this PR is still intended to backported to v1.15, or the label should be dropped. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If the daemon start failed due to e.g. invalid configuration the cleanup was not performed as it was only done from the stop function that was not invoked since the daemon start had failed.
This in some situations lead to confusing crashes as hive did stop all the dependencies of the legacy daemon, for example this crash was reported:
The crash was caused by the LocalNodeStore stopping and clearing the global variable in the node package (to make sure for example our tests don't leak data across them and to make sure we do cleanup correctly), while the DNSProxy was still running in the background processing DNS packets and using the globals in the node package.
To fix this, run the cleanup functions also when "newDaemon" or "startDaemon" fails with an error.
Fixes: f21f303 ("daemon: bubble up error from startDaemon instead of fataling")