8000 adjust logstash health check so kubernetes liveness probe doesn't kill it · Issue #630 · cisagov/Malcolm · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

adjust logstash health check so kubernetes liveness probe doesn't kill it #630

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mmguero opened this issue Mar 25, 2025 · 1 comment
Closed

adjust logstash health check so kubernetes liveness probe doesn't kill it #630

mmguero opened this issue Mar 25, 2025 · 1 comment
Assignees
@mmguero
Labels
bug Something isn't working cloud Relating to deployment of Malcolm in the cloud and/or with Kubernetes logstash Relating to Malcolm's use of Logstash
Milestone
v25.04.0

Comments

@mmguero
Copy link
Collaborator
mmguero commented Mar 25, 2025

The liveness check for the logstash container checks for green/yellow status. However, if a pipeline is severely loaded, it could be red, but we really don't want something like Kubernetes to try to force restart the pod, since that would only make it worse. We should probably adjust the container_health script to just return ok as long as it can hit the health report API endpoint at all, regardless of the state of the pipelines.

On the other hand, the logstash pipelines check for the /mapi/ready makes sure none of them are red. In my opinion we should leave this the way it is.
@mmguero mmguero added bug Something isn't working logstash Relating to Malcolm's use of Logstash labels Mar 25, 2025
@mmguero mmguero added this to the v25.04.0 milestone Mar 25, 2025
@mmguero mmguero added this to Malcolm Mar 25, 2025
@mmguero mmguero moved this to Todo (develop) in Malcolm Mar 25, 2025
@mmguero mmguero added the cloud Relating to deployment of Malcolm in the cloud and/or with Kubernetes label Mar 26, 2025
@mmguero mmguero self-assigned this Apr 10, 2025
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 10, 2025
@mmguero
adjust logstash health check so kubernetes liveness probe doesn't kil…
2b26989 
…l it, cisagov#630
@mmguero
Copy link
Collaborator Author
mmguero commented Apr 10, 2025

Okay, I've set it now so that docker-compose.yml runs the health check with a -s argument (for "strict") so it will report unhealthy if things aren't green/yellow. This is desirable because I want it to show as unhealthy, but docker compose doesn't kill/restart unhealthy containers. Kubernetes, on the other hand, doesn't run it with -s so it will just check to make sure the endpoint is live rather than being so strict with the pipeline status.

@mmguero mmguero closed this as completed Apr 10, 2025
@github-project-automation github-project-automation bot moved this from Todo (develop) to Done in Malcolm Apr 10, 2025
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Apr 10, 2025
@mmguero
adjust logstash health check so kubernetes liveness probe doesn't kil…
29273a5 
…l it, cisagov#630
This was referenced Apr 28, 2025
Merged
Merged
@mmguero mmguero moved this from Done to Released in Malcolm Apr 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mmguero mmguero

Labels
bug Something isn't working cloud Relating to deployment of Malcolm in the cloud and/or with Kubernetes logstash Relating to Malcolm's use of Logstash
Projects
Malcolm
Status: Released
Milestone
v25.04.0
Development

No branches or pull requests
1 participant
@mmguero
0