Sometimes, we find pods restarting several times a day without any error. The memory consumption keeps going up, until it reaches the memory limit.
We have a process to diagnose memory leak for Go services. Tools such as pprof and minikube can help us find the root cause.
In this repo, we used the pprof tool to detect the memory leaks
- Gin framework
- Graphviz
- Visit the download location
- Download and run the 32-bit or 64-bit exe file.
- Ignore any security warnings you might get.
- During the installation, make sure you select Add Graphviz top the system PATH for current user.
- When the installation is finished, start CMD as an administrator
- Run command
dot
Note Make sure that during installation checked "Add Graphviz to the system PATH for current user"
- One time install
go install github.com/google/pprof@latest- Dump heap to a file(ensure that application is run!-using new terminal)
curl http://<HOSTNAME>:<PORT>/debug/pprof/heap > heap.out- Use pprof to interact with heap
go tool pprof heap.out- Inside the new command prompt
png💡 Everytime for reporting new result you should repeat levels 2,3,4
💡 After run go run main.go go to localhost on browser then repeat previous level
🔗 http://localhost:8080/append-slice
🔗 http://localhost:8080/hanging
🔗 http://localhost:8080/streams
In the diagram, each box is a function, and each arrow means a function call. The bigger the box, the higher memory usage. From the graph above, the blame goes to runtime function “allocm” (the largest box near the bottom). Once we found which function is causing the problem, we can check how the memory is leaked.