8000 Add embed directive to embed the compiled CORE bpf object into go binary by grantseltzer · Pull Request #818 · aquasecurity/tracee · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Add embed directive to embed the compiled CORE bpf object into go binary #818

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jul 23, 2021
Merged

Add embed directive to embed the compiled CORE bpf object into go binary #818

merged 8 commits into from
Jul 23, 2021

Conversation

grantseltzer
Copy link
Contributor

Handles #783

Signed-off-by: grantseltzer grantseltzer@gmail.com

@grantseltzer grantseltzer added this to the Post CO:RE milestone Jul 13, 2021
@grantseltzer grantseltzer changed the title Add embed directive to embed the compiled bpf object into go binary Add embed directive to embed the compiled CORE bpf object into go binary Jul 13, 2021
Copy link
Collaborator
@itaysk itaysk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also missing updates to the main Makefile

@itaysk
Copy link
Collaborator
itaysk commented Jul 15, 2021

how is this fixing #783?

@grantseltzer
Copy link
Contributor Author

I've pushed a commit that seems to be the only way to conditionally use the embed directive. I use a "core" build flag with two different files that build depending on if the tag is present or not.

Copy link
Collaborator
@itaysk itaysk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just one last todo

@grantseltzer
Copy link
Contributor Author
grantseltzer commented Jul 20, 2021

I've pushed a change which reads the CO:RE binary straight from the embedded go binary into a slice of bytes, and loads it using libbpfgo.NewModuleFromBuffer. I also have changed the regular bpf object reading logic to do the same thing. That is to keep the code clean, we're loading it into memory from file before regardless. This handles cleanup for us as well.

@grantseltzer
Copy link
Contributor Author

After discussing with @yanivagman we came to the agreement that for the sake of optimal portability, the CO:RE should always be built (since BTF kernel is not required, just the vmlinux.h file) and embedded.

At runtime tracee-ebpf should check if it's running on a BTF enabled kernel (see if /sys/kernel/btf exists). If there is BTF, use that embedded CORE binary. If not, look for the compiled bpf object, and if it doesn't exist attempt to build it from the embedded source.

I will push a commit with this soon. @itaysk @rafaeldtinoco any thoughts?

@rafaeldtinoco
Copy link
Contributor
rafaeldtinoco commented Jul 20, 2021

After discussing with @yanivagman we came to the agreement that for the sake of optimal portability, the CO:RE should always be built (since BTF kernel is not required, just the vmlinux.h file) and embedded.

+1. Will start giving us the flexibility we want/need.

At runtime tracee-ebpf should check if it's running on a BTF enabled kernel (see if /sys/kernel/btf exists). If there is BTF, use that embedded CORE binary. If not, look for the compiled bpf object, and if it doesn't exist attempt to build it from the embedded source.

So, wrapping up.. with your change we will have:

  1. check for /sys/kernel/btf
  2. if yes, use CO-RE binary
  3. if no, look for BPF.o for running kernel
  4. if no BPF.o, compile BPF.o for running kernel

And after BTFHUB we will have (@yanivagman pls confirm if this is right):

  • CO-RE phase:
  1. check for /sys/kernel/btf.
  2. if yes, use CO-RE binary
  3. if no sysfs btf, look for BPF support in running kernel.
  4. if BPF support, get BTF for running kernel from BTFHUB
  5. if BTFHUB file exists use CO-RE binary with downloaded/cached BTF file
  • Legacy phase:
  1. if BTFHUB file does not exist look for BPF.o for running kernel
  2. if no, compile BPF.o for running kernel

I will push a commit with this soon. @itaysk @rafaeldtinoco any thoughts?

I like it =)

@grantseltzer
Copy link
Contributor Author

@rafaeldtinoco Just what I had in mind :-)

I will turn your comment into an issue as part of merging this PR

@yanivagman
Copy link
Collaborator
yanivagman commented Jul 21, 2021

@rafaeldtinoco LGTM, just one thing I'm not sure about:

  1. if BTFHUB file does not exist look for BPF.o for running kernel

Do we really want to look for bpf.o only then?
One case I can think about is that for some reason, core bpf will fail at runtime (which can happen if, for example, a field of a struct was moved to some other internal struct in a newer kernel) - then the user (or us, for debugging purposes) might want to use a precompiled bpf object (with the specific running kernel headers). This can be done, for example, by providing an environment variable (which we already support) TRACEE_BPF_FILE.
WDYT?

@itaysk
Copy link
Collaborator
itaysk commented Jul 21, 2021

If there is BTF, use that embedded CORE binary. If not, look for the compiled bpf object, and if it doesn't exist attempt to build it from the embedded source. ... any thoughts?

Yes, exactly. Thanks for clarifying it and implementing!

@itaysk
Copy link
Collaborator
itaysk commented Jul 21, 2021

@rafaeldtinoco

if no sysfs btf, look for BPF support in running kernel.

Isn't it late to look for BPF support in running kernel, at this stage?

Also, I agree with @yanivagman that specific object file, if present, should override the embedded one

Copy link
Collaborator
@itaysk itaysk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LVGTM :)

@@ -132,7 +127,6 @@ $(OUT_DIR)/tracee.bpf.core.%.o: $(BPF_SRC) $(LIBBPF_HEADERS) $(CMD_CLANG)
-DLINUX_VERSION_CODE=$(LINUX_VERSION_CODE) \
-target bpf \
-O2 -c -g -o $@ $<
endif
else
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DOCKER mode seem to be inconsistently supported for the obj files now. We need to be able to build both obj targets with docker and without. currently I think it's not possible? (haven't tried, just by reading the code)
If I'm right, would be easier to implement, and also more readable to just copy the same stanza from the other targets (e.g the OUT_BIN target) where the DOKCER variance is inside the target, not outside.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it resolved? I don't see any change in the code

Copy link
Contributor Author
@grantseltzer grantseltzer Jul 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ugh, I forgot to git push... Sorry about that.

@rafaeldtinoco
Copy link
Contributor

@rafaeldtinoco

if no sysfs btf, look for BPF support in running kernel.

Isn't it late to look for BPF support in running kernel, at this stage?

CO-RE phase:

  • check for /sys/kernel/btf.
  • if yes, use CO-RE binary
  • if no sysfs btf, look for BPF support in running kernel.

If /sys/kernel/btf exists then BPF support is already there (so we are checking it in the first item). This item is to check for old kernels that have BPF support but no BTF support, they would need the BTFHUB (bellow).

  • if BPF support, get BTF for running kernel from BTFHUB
  • if BTFHUB file exists use CO-RE binary with downloaded/cached BTF file

Also, I agree with @yanivagman that specific object file, if present, should override the embedded one

Yes, if environment variable provided, we should use that object no matter what, I'm +1 on that as well.

Signed-off-by: grantseltzer <grantseltzer@gmail.com>
Signed-off-by: grantseltzer <grantseltzer@gmail.com>
Signed-off-by: grantseltzer <grantseltzer@gmail.com>
Signed-off-by: grantseltzer <grantseltzer@gmail.com>
Signed-off-by: grantseltzer <grantseltzer@gmail.com>
… bpf source, add logic for checking if bpf is enabled

Signed-off-by: grantseltzer <grantseltzer@gmail.com>
@grantseltzer
Copy link
Contributor Author

Opened #833

// getBPFObject finds or builds ebpf object file and returns it's path
func getBPFObject() (string, error) {
// getBPFObjectPath finds or builds ebpf object file and returns it's path
func getBPFObjectPath() (string, error) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, we want to have to TRACEE_BPF_FILE to be looked also in core mode. I think that we can call this function an exit before the binary build (or just split it into two functions)

Signed-off-by: grantseltzer <grantseltzer@gmail.com>
Signed-off-by: grantseltzer <grantseltzer@gmail.com>
Copy link
Collaborator
@yanivagman yanivagman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Collaborator
@itaysk itaysk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approving but I'm still not sure why the docker vs non-docker bpf targets are asymmetrical? in docker it's $(OUT_BPF) and in non-docker it's $(OUT_DIR)/tracee.bpf.%.o. Not the end of the world but possibly future bugs potential

@grantseltzer
Copy link
Contributor Author

@itaysk I don't fully understand it either, I don't have the strongest make knowledge. Doesn't having OUT_BPF allow for calling the $(OUT_DIR)/tracee.bpf.%.o target so arguments can be passed in via $@?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants
0