8000 performance: introduce automated performance dashboard with pyroscope by rafaeldtinoco · Pull Request #2968 · aquasecurity/tracee · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

performance: introduce automated performance dashboard with pyroscope #2968

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 7, 2023
Merged

performance: introduce automated performance dashboard with pyroscope #2968

merged 2 commits into from
Apr 7, 2023

Conversation

rafaeldtinoco
Copy link
Contributor
@rafaeldtinoco rafaeldtinoco commented Apr 3, 2023

1. Explain what the PR does

Tracee developers are struggling to find different ways to check performance results when doing the development. This is an attempt to automate performance checks and dashboards in order for an unified experience.

This first feature adds a grafana dash board containing trace pperf

commit bb367ac (HEAD -> new-pyro, rafaeldtinoco/new-pyro)
Author: Rafael David Tinoco rafaeldtinoco@gmail.com
Date: Mon Apr 3 06:09:03 2023

performance: automate pyroscope dashboard view

Developer might execute:

- make -f builder/Makefile.performance pyroscope-start
- make -f builder/Makefile.performance pyroscope-stop

And will get access to a performance dashboard while executing tracee.

commit b3382a5
Author: Rafael David Tinoco rafaeldtinoco@gmail.com
Date: Mon Apr 3 06:08:22 2023

performance: create a pyroscope pperf client feature

2. Explain how to test it

$ make -f builder/Makefile.performance pyroscope-start

<open browser at http://localhost:3000>

image

You can now navigate the dashboard, picking specific timeslices that will contain not only the event and CPU metrics, but also the flamegraph for that time window.

$ make -f builder/Makefile.performance pyroscope-start

3. Other comments

Idea is to make the dashboard better with time:

  • include per application CPU and MEM consumption
  • include 1 graph per eBPF program overhead
  • ...

Developer might execute:

- make -f builder/Makefile.performance pyroscope-start
- make -f builder/Makefile.performance pyroscope-stop

And will get access to a performance dashboard while executing tracee.
Copy link
Collaborator
@NDStrahilevitz NDStrahilevitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work. There's a missing flag declaration and a question I have before I +1 but overall LGTM.

@@ -85,6 +87,20 @@ func (s *Server) EnablePProfEndpoint() {
s.mux.HandleFunc("/debug/pprof/trace", pprof.Trace)
}

// EnablePyroAgent enables pyroscope agent in golang push mode
// TODO: make this configurable
func (s *Server) EnablePyroAgent() error {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC pyroscope should be able to directly scrape pprof and as I can see in the prometheus.yml file you do target the default :3366 port.
And in another file I see you target this port, so i'm not sure what's the difference.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already covered that in private. More performatic and I also couldn't do the "pull mode" (which is what you are referring) to work in an automated way: https://pyroscope.io/docs/golang-pull-mode/.

I'll give another try, but not a blocker as we've spoken.

@NDStrahilevitz
Copy link
Collaborator

@josedonizetti waiting for you before merging or can we go ahead?

@rafaeldtinoco rafaeldtinoco added this to the v0.13.1 milestone Apr 4, 2023
@rafaeldtinoco
Copy link
Contributor Author

Adding to milestone 0.13.1 as this is not a code change and allows us to test Tracee better.

@yanivagman yanivagman modified the milestones: v0.13.1, v0.14.0 Apr 4, 2023
Copy link
Contributor
@josedonizetti josedonizetti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NICE!

@rafaeldtinoco rafaeldtinoco merged commit 23f0a14 into aquasecurity:main Apr 7, 2023
@rafaeldtinoco rafaeldtinoco deleted the new-pyro branch April 7, 2023 16:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants
0