8000 grype db import fails since v0.88 and above · Issue #2542 · anchore/grype · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

grype db import fails since v0.88 and above #2542

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ep4sh opened this issue Mar 19, 2025 · 3 comments · Fixed by #2546
Closed

grype db import fails since v0.88 and above #2542

ep4sh opened this issue Mar 19, 2025 · 3 comments · Fixed by < 8000 a href="https://github.com/anchore/grype/pull/2546" data-hydro-click="{"event_type":"issue_cross_references.click","payload":{"reference_location":"ISSUE_HEADER","user_id":null,"issue_id":2930789451,"pull_request_id":2404544545,"originating_url":"https://github.com/anchore/grype/issues/2542"}}" data-hydro-click-hmac="b50eeda64c0756917d24536f620824cf708c4dcce644579d4aafb788ffc43e12" data-hovercard-type="pull_request" data-hovercard-url="/anchore/grype/pull/2546/hovercard">#2546
Assignees
@kzantow
Labels
bug Something isn't working

Comments

@ep4sh
Copy link
ep4sh commented Mar 19, 2025 •
edited
Loading

What happened:
grype db import fails since v0.88 and above

What you expected to happen:
import works successfully

How to reproduce it (as minimally and precisely as possible):

root@f208c080bd13:/# curl --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" "${CI_API_V4_URL}/projects/<REDACTED>/packages/generic/db/${GRYPE_DB_LATEST_VERSION}/grype-db.tar.gz" > "${CI_PROJECT_DIR}/grype-db.tar.gz"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  206M  100  206M    0     0  14.2M      0  0:00:14  0:00:14 --:--:-- 14.8M
root@f208c080bd13:/# grype db import grype-db.tar.gz
Vulnerability database imported
root@f208c080bd13:/# grype --version
grype 0.87.0
root@f208c080bd13:/# wget https://github.com/anchore/grype/releases/download/v0.90.0/grype_0.90.0_linux_amd64.tar.gz > /dev/null
.....
.....
.....

Saving to: 'grype_0.90.0_linux_amd64.tar.gz.1'

grype_0.90.0_linux_amd64.tar.gz.1                    100%[=====================================================================================================================>]  22.52M  16.8MB/s    in 1.3s    

2025-03-19 07:44:52 (16.8 MB/s) - 'grype_0.90.0_linux_amd64.tar.gz.1' saved [23618869/23618869]

root@f208c080bd13:/# ./grype --version
grype 0.90.0
root@f208c080bd13:/# ./grype db import grype-db.tar.gz
 ⠹ Vulnerability DB                ━━━━━━━━━━━━━━━━━━━━  [hydrating]  
panic: runtime error: invalid memory address or nil pointer dereference
                                                                       [signal SIGSEGV: segmentation violation code=0x1 addr=0x59 pc=0x1f28d12]

                                                                                                                                               goroutine 27 [running]:
                                                                                                                                                                      github.com/anchore/grype/grype/db/v6.(*store).Close(0x27a77a8?)
                  	/home/runner/work/grype/grype/grype/db/v6/store.go:101 +0x12
                                                                                    github.com/anchore/grype/internal/log.CloseAndLogError({0x2f84880?, 0x0?}, {0xc000a9a030, 0x2d})
                                                                                                                                                                                    	/home/runner/work/grype/grype/internal/log/errors.go:10 +0x38
                                  github.com/anchore/grype/cmd/grype/cli/commands.runDBImport.NewCurator.Hydrater.func1({0xc000a9a030, 0x2d})
                                                                                                                                             	/home/runner/work/grype/grype/grype/db/v6/db.go:105 +0x49
                                                                                                                                                                                                         github.com/anchore/grype/grype/db/v6/installation.curator.hydrate({{0x2fc3600, 0x47e9520}, {0x2fb0c58, 0xc0009fc000}, {{0xc0009a9998, 0x15}, 0x0, 0x1, 0x1, 0x188e6d68b0000, ...}, ...}, ...)
                                                                                                                                                                                   	/home/runner/work/grype/grype/grype/db/v6/installation/curator.go:492 +0xb3
                                                github.com/anchore/grype/grype/db/v6/installation.curator.activate({{0x2fc3600, 0x47e9520}, {0x2fb0c58, 0xc0009fc000}, {{0xc0009a9998, 0x15}, 0x0, 0x1, 0x1, 0x188e6d68b0000, ...}, ...}, ...)
                           	/home/runner/work/grype/grype/grype/db/v6/installation/curator.go:480 +0xfe
                                                                                                           github.com/anchore/grype/grype/db/v6/installation.curator.Import({{0x2fc3600, 0x47e9520}, {0x2fb0c58, 0xc0009fc000}, {{0xc0009a9998, 0x15}, 0x0, 0x1, 0x1, 0x188e6d68b0000, ...}, ...}, ...)
                                                                                    	/home/runner/work/grype/grype/grype/db/v6/installation/curator.go:460 +0x71f
                                                                                                                                                                    github.com/anchore/grype/cmd/grype/cli/commands.runDBImport({{{{0x2788d0d, 0x5}, {0x2f79a40, 0x6}, {0x2fabd00, 0x28}, {0x2f7a938, 0x7}, {0x2f82a20, 0x14}}, ...}, ...}, ...)
                                                                                                                             	/home/runner/work/grype/grype/cmd/grype/cli/commands/db_import.go:48 +0x3ff
                                                                                                                                                                                                           github.com/anchore/grype/cmd/grype/cli/commands.DBImport.func1(0x0?, {0xc00091e0f0?, 0x0?, 0x0?})
                                                                                         	/home/runner/work/grype/grype/cmd/grype/cli/commands/db_import.go:24 +0x98
                                                                                                                                                                          github.com/anchore/clio.(*application).setupCommand.(*application).WrapRunE.func2.1(0x0?, {0xc00091e0f0?, 0x0?, 0x0?})
                                                                             	/home/runner/go/pkg/mod/github.com/anchore/clio@v0.0.0-20241115144204-29e89f9fa837/application.go:150 +0x8c
                                                                                                                                                                                           github.com/anchore/clio.async.func1()
             	/home/runner/go/pkg/mod/github.com/anchore/clio@v0.0.0-20241115144204-29e89f9fa837/application.go:364 +0x63
                                                                                                                           created by github.com/anchore/clio.async in goroutine 1
                                                                                                                                                                                  	/home/runner/go/pkg/mod/github.com/anchore/clio@v0.0.0-20241115144204-29e89f9fa837/application.go:362 +0xc5

Environment:

  • Output of grype version: 0.88, 0.90
  • OS (e.g: cat /etc/os-release or similar): VERSION="20.04.6 LTS (Focal Fossa)"
@ep4sh ep4sh added the bug Something isn't working label Mar 19, 2025
@ep4sh ep4sh changed the title grype db import fails grype db import fails since v0.88 and above Mar 19, 2025
@popey
Copy link
Contributor
popey commented Mar 19, 2025

Hi @ep4sh
In grype 0.88 we moved to grype db schema v6. Is the database you're trying to import a v5 schema?

I was able to download one of our v6 databases and import it.

Download v6 schema vuln database

$ wget -U 'grype 0.90.0' https://grype.anchore.io/databases/v6/latest.json
--2025-03-19 13:06:18--  https://grype.anchore.io/databases/v6/latest.json
Resolving grype.anchore.io (grype.anchore.io)... 104.22.75.215, 104.22.74.215, 172.67.15.216, ...
Connecting to grype.anchore.io (grype.anchore.io)|104.22.75.215|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 249 [application/json]
Saving to: ‘latest.json’

latest.json 100%[============================================================>]     249  --.-KB/s    in 0s

2025-03-19 13:06:18 (23.7 MB/s) - ‘latest.json’ saved [249/249]
$ jq -r '.path' latest.json
vulnerability-db_v6.0.2_2025-03-19T01:31:00Z_1742357230.tar.zst
$ wget https://grype.anchore.io/databases/v6/vulnerability-db_v6.0.2_2025-03-19T01:31:00Z_1742357230.tar.zst
--2025-03-19 13:06:54--  https://grype.anchore.io/databases/v6/vulnerability-db_v6.0.2_2025-03-19T01:31:00Z_1742357230.tar.zst
Resolving grype.anchore.io (grype.anchore.io)... 104.22.75.215, 104.22.74.215, 172.67.15.216, ...
Connecting to grype.anchore.io (grype.anchore.io)|104.22.75.215|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 66656009 (64M) [application/zstd]
Saving to: ‘vulnerability-db_v6.0.2_2025-03-19T01:31:00Z_1742357230.tar.zst’

vulnerability-db_v6.0.2_2025-03-19T01:31:00Z_1742357230.tar. 100%[============================================================>] 63.57M  23.8MB/s    in 2.7s

2025-03-19 13:06:57 (23.8 MB/s) - ‘vulnerability-db_v6.0.2_2025-03-19T01:31:00Z_1742357230.tar.zst’ saved [66656009/66656009]

Import the v6 schema database

$ grype db import ./vulnerability-db_v6.0.2_2025-03-19T01:31:00Z_1742357230.tar.zst
 ✔ Vulnerability DB                [imported]

Download v5 schema database

$ wget -U 'grype 0.87.0' https://toolbox-data.anchore.io/grype/databases/listing.json
--2025-03-19 13:15:05--  https://toolbox-data.anchore.io/grype/databases/listing.json
Resolving toolbox-data.anchore.io (toolbox-data.anchore.io)... 172.67.15.216, 104.22.75.215, 104.22.74.215, ...
Connecting to toolbox-data.anchore.io (toolbox-data.anchore.io)|172.67.15.216|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3890 (3.8K) [binary/octet-stream]
Saving to: ‘listing.json.2’

listing.json.2                                               100%[============================================================>]    3.80K  --.-KB/s    in 0s

2025-03-19 13:15:05 (22.0 MB/s) - ‘listing.json.2’ saved [3890/3890]
$ jq -r '.available."5"[0].url'  listing.json.2
https://grype.anchore.io/databases/vulnerability-db_v5_2025-03-19T01:31:00Z_1742358120.tar.gz
$ wget -U 'grype 0.88.0' https://grype.anchore.io/databases/vulnerability-db_v5_2025-03-19T01:31:00Z_1742358120.tar.gz
--2025-03-19 13:24:19--  https://grype.anchore.io/databases/vulnerability-db_v5_2025-03-19T01:31:00Z_1742358120.tar.gz
Resolving grype.anchore.io (grype.anchore.io)... 104.22.75.215, 172.67.15.216, 104.22.74.215, ...
Connecting to grype.anchore.io (grype.anchore.io)|104.22.75.215|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 217089442 (207M) [application/gzip]
Saving to: ‘vulnerability-db_v5_2025-03-19T01:31:00Z_1742358120.tar.gz’

vulnerability-db_v5_2025-03-19T01:31:00Z_1742358120.tar.gz   100%[============================================================>] 207.03M  24.2MB/s    in 8.2s

2025-03-19 13:24:27 (25.2 MB/s) - ‘vulnerability-db_v5_2025-03-19T01:31:00Z_1742358120.tar.gz’ saved [217089442/217089442]

Import v5 database into grype 0.90.0

grype db import ./vulnerability-db_v5_2025-03-19T01:31:00Z_1742358120.tar.gz
 ⠙ Vulnerability DB                ━━━━━━━━━━━━━━━━━━━━  [hydrating]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x59 pc=0x10223aa08]

   goroutine 14 [running]:
github.com/anchore/grype/grype/db/v6.(*store).Close(0x10235e0e6?)
/home/runner/work/grype/grype/grype/db/v6/store.go:101 +0x18
github.com/anchore/grype/internal/log.CloseAndLogError({0x1032af680?, 0x0?}, {0x14000a42040, 0x3b})
/home/runner/work/grype/grype/internal/log/errors.go:10 +0x38
github.com/anchore/grype/cmd/grype/cli/commands.runDBImport.NewCurator.Hydrater.func1({0x14000a42040, 0x3b})
/home/runner/work/grype/grype/grype/db/v6/db.go:105 +0x50
github.com/anchore/grype/grype/db/v6/installation.curator.hydrate({{0x1032ee060, 0x104a94280}, {0x1032dbcb8, 0x14000a2e000}, {{0x1400086cb70, 0x23}, 0x0, 0x1, 0x1, 0x188e6d68b0000, ...}, ...}, ...)
/home/runner/work/grype/grype/grype/db/v6/installation/curator.go:492 +0xa8
github.com/anchore/grype/grype/db/v6/installation.curator.activate({{0x1032ee060, 0x104a94280}, {0x1032dbcb8, 0x14000a2e000}, {{0x1400086cb70, 0x23}, 0x0, 0x1, 0x1, 0x188e6d68b0000, ...}, ...}, ...)
/home/runner/work/grype/grype/grype/db/v6/installation/curator.go:480 +0xe8
github.com/anchore/grype/grype/db/v6/installation.curator.Import({{0x1032ee060, 0x104a94280}, {0x1032dbcb8, 0x14000a2e000}, {{0x1400086cb70, 0x23}, 0x0, 0x1, 0x1, 0x188e6d68b0000, ...}, ...}, ...)
/home/runner/work/grype/grype/grype/db/v6/installation/curator.go:460 +0x5ec
github.com/anchore/grype/cmd/grype/cli/commands.runDBImport({{{{0x10233f847, 0x5}, {0x102b3a4f8, 0x6}, {0x102b3b5e0, 0x28}, {0x102b3b1a0, 0x7}, {0x102b3b280, 0x14}}, ...}, ...}, ...)
/home/runner/work/grype/grype/cmd/grype/cli/commands/db_import.go:48 +0x2dc
github.com/anchore/grype/cmd/grype/cli/commands.DBImport.func1(0x0?, {0x14000b19c40?, 0x0?, 0x0?})
/home/runner/work/grype/grype/cmd/grype/cli/commands/db_import.go:24 +0x8c
github.com/anchore/clio.(*application).setupCommand.(*application).WrapRunE.func2.1(0x0?, {0x14000b19c40?, 0x0?, 0x0?})
/home/runner/go/pkg/mod/github.com/anchore/clio@v0.0.0-20241115144204-29e89f9fa837/application.go:150 +0x90
github.com/anchore/clio.async.func1()
/home/runner/go/pkg/mod/github.com/anchore/clio@v0.0.0-20241115144204-29e89f9fa837/application.go:364 +0x68
created by github.com/anchore/clio.async in goroutine 1
/home/runner/go/pkg/mod/github.com/anchore/clio@v0.0.0-20241115144204-29e89f9fa837/application.go:362 +0xc0

@kzantow kzantow self-assigned this Mar 19, 2025
@kzantow kzantow moved this to In Progress in OSS Mar 19, 2025
@kzantow kzantow moved this from In Progress to In Review in OSS Mar 19, 2025
@kzantow kzantow mentioned this issue Mar 19, 2025
Merged
@github-project-automation github-project-automation bot moved this from In Review to Done in OSS Mar 19, 2025
@kzantow
Copy link
Contributor
kzantow commented Mar 19, 2025

Hi @ep4sh, I'm going to close this issue since we don't plan on reimplementing V5 DB support in Grype at this time. However, I have fixed the panic and made error messaging better to help understand the issue for anyone who runs into this in the future.

But I didn't want to close this without understanding if there are things we could do to help prevent issues like this in the future if we need to implement new, incompatible database revisions.

Aside from continuing to support older technology with new releases, is there anything we could have done to make this more clear or seamless for you? (We did add information to release notes and a blog about this release, but I assume many people won't see these.) It seems you had a custom step to manage downloading and using the database. Is there some way this could have been made to work more cleanly -- maybe some grype db save function or similar which downloads the latest db at the right version which could have started using the right endpoint and the right db archive automatically?

@ep4sh
Copy link
Author
ep4sh commented Mar 20, 2025

Thank you so much, the suggested changes helped to resolve!

@BrewTestBot BrewTestBot mentioned this issue Apr 1, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kzantow kzantow

Labels
bug Something isn't working
Projects
OSS
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: panic and improve non-db-v6 error messaging kzantow-anchore/grype
3 participants
@popey @kzantow @ep4sh
0