feat: add support for OSV schema #217
Conversation
Signed-off-by: juan131 <jariza@vmware.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: juan131 <jariza@vmware.com>
f2b5170 to
3f96831
Compare
Signed-off-by: juan131 <jariza@vmware.com>
|
@wagoodman sorry for the long delay on resuming this work, I've been involved in other projects these last weeks and I didn't find the time for this. |
|
Hi @wagoodman, is there anything we can help with at this time? |
|
Hi @wagoodman, I hope you’re doing well! We wanted to check in on the status of our recent contributions. We have some open PRs, and we’re unsure about the current progress or the next steps. Could you kindly provide an update? It would help us plan our work and understand how we can continue contributing effectively. Here are a couple of the open PRs for reference: We appreciate any details you can share. Thanks in advance for your help! |
|
Hi @carrodher! I wanted to give an overview of what still needs to happen to get grype supporting the bitnami vulnerability provider. The list is at least mostly in the order things need to happen. The items don't say who's doing what or when; I'm trying to just get the scope of work remaining written down, and then we can figure that out. I am happy to help with the work but I want to clarify the remaining work first. Step 1: Syft needs to find the right packagesSyft needs to find packages that can be compared to the data bitnami publishes. The best way to do this is probably to write a Syft cataloger that parses the SPDX SBOMs that bitnami puts at paths like This work is tracked at anchore/syft#3065. This issue is largely ready to pick up. It's important that we do this first for two reasons:
Step 2: Make a test imageIn order to test the new data provider end to end, there needs to be an image that grype can scan to find known vulnerabilities from the new provider. Add a test image built from an old Bitnami (so it will have some vulns) image to our test images repo. Step 3: Local end to end workflow workingThe way to do that is to make a 3 repo local workspace with Vunnel, Grype, and Grype-DB all incorporating any necessary changes to scan a bitnami image. To attempt and end to end run, you can go into the make dev provider=bitnami # creates a subshell
make update-db
grype "$image_from_ste_2" This will fail, because there are a few other things to do, but when it starts finding the known vulnerabilities that are present in the old Bitnami image, things are working and we can set up an end to end test and merge. There are a few changes that I know about, and we may discover more. PR changes that I know aboutThis PR (#217) needs to have a transformer implemented. The transformer is a function that is responsible for mapping the OSV data that vunnel downloaded from Bitnami to the vulnerability and vulnerability metadata structures that grype uses in its SQLite database. You can see a transformer here: Additionally, the vunnel pr will need to be changed so that it emits records in dictionaries shaped like OSV JSON, in other words this comment needs to be addressed. It's also possible that Grype needs to have a new matcher added for the bitnami package type, but the stock matcher might work (it tries to compare semver for packages that don't have a more specific matcher type), but some package types need a specific match. You can see all the matchers here: After these changes, I think that we will be ready to add the end to end tests. There are also some linting fixes and missing unit tests, but I think it makes sense to get the end to end flow working. If you prefer to add unit tests as you build the blocks of the system, that's fine too. Adding labels and test casesVunnel, Grype, and Grype-DB are guarded by end-to-end test cases that basically have 3 inputs:
The test gate than asserts that for each test image, the candidate configuration is better, or at least no worse within some tolerance, than the reference configuration. One grype in your workflow can match the test image, you can label matches grype finds against your test images. Some details are at That ended up being a bit longer post than I thought, but I wanted to get the remaining steps written down concretely to help get your hard work over the line. |
|
@carrodher and @juan131 I am pretty determined to see these changes land and will start picking up this work. I'd also be happy to sync up with you a 8000 ll if you'd like to provide any additional assistance - just let me know! As an aside, we are striving to make this process easier - I realize from outside the team the process outlined in my last comment is probably a little daunting and hard to discover. I'd also be glad to sync with you on what might have made that easier. |
|
@willmurphyscode thanks so much for writing down the required steps and your determination to get this done. If that's fine to you, we could set up a sync meeting (@carrodher and myself from Bitnami side) so we can plan the action items and assign who'll be leading the development |
|
@juan131 a sync up meeting sounds great. Probably the easiest way to hash out the details is to DM me in our discourse (not discord) instance: https://anchorecommunity.discourse.group/ ( We also have regularly scheduled community meetings. The next one is at Thursday, 10 October at noon Eastern. |
|
@juan131 here's the branch I mentioned during the meeting today: https://github.com/anchore/syft/compare/spike-bitnami-cataloger It should help in creating a PR to resolve anchore/syft#3065. Let us know if you have any questions, or when you want to set up the next meeting. Thanks! |
|
@willmurphyscode now that we completed the 1st step from #217 (comment), we can continue with the plan and add the test image at https://github.com/anchore/test-images, right? |
|
indeed -- or if you have an image that is public then we can use that directly without needing to make an explicitly image in the test_images repo |
|
@wagoodman I think we can use
There several "high" vulnerabilities affecting Airflow listed in Bitnami vulndb: There are also several "high" vulnerabilities affecting python, setuptools and virtualenv listed in Bitnami vulndb:
What do you think? |
|
Hi @juan131, that's a reasonable image to start with. We always can change it later. I think this PR can be done in either order relative to anchore/vunnel#512, since vunnel#512 will make vunnel emit OSV compliant JSON, and this PR will transform OSV compliant JSON, so as long as both sides read the same spec the work can be concurrent. Would it be helpful if I picked up anchore/vunnel#512, at least long enough to get it going again? |
|
@willmurphyscode I just updated vunnel PR but I'd appreciate help with the CI errors |
|
Hi @juan131! The vunnel PR is merged - it was almost there, just some Python specific nonsense and a couple tests. Let me know if you need a hand with this PR - there has been a whole db schema released since it was opened, so the codebase has diverged quite a bit from where this branch branched off. Thanks! |
Signed-off-by: juan131 <juan.ariza@broadcom.com>
|
Working on transformers for v5 and v6, I'll update the PR soon |
|
You only need to do the v6 transformers, since we don't even have a way to match bitnami packages for v5. |
Signed-off-by: juan131 <juan.ariza@broadcom.com>
|
I just added the transformer at 0a3cf9d but I'm missing more unit tests yet @willmurphyscode |
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
Some changes still needed to version range handling
Signed-off-by: juan131 <juan.ariza@broadcom.com>
Signed-off-by: Will Murphy <willmurphyscode@users.noreply.github.com>
Summary
As described at anchore/grype#1609, Bitnami is providing vulnerability matching data for their containers via the Bitnami Vulnerability Database repository.
This database uses the Open Source Vulnerability format (AKA OSV) and it's currently part of the aggregated OSV vulnerability database.
This PR follows up anchore/vunnel#447, adding support to grype-db to aggregate data from CVE feeds using the OSV schema.