8000 Ensure deps.dev data is fresh for each run of Criticality Score by calebbrown · Pull Request #267 · ossf/criticality_score · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Ensure deps.dev data is fresh for each run of Criticality Score #267

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Dec 1, 2022
Merged

Ensure deps.dev data is fresh for each run of Criticality Score #267

merged 6 commits into from
Dec 1, 2022

Conversation

calebbrown
Copy link
Contributor

This PR ensures the deps.dev data is always fresh.

  1. Introduce "IF NOT EXISTS" to the table generation so that if more than one worker attempts to create the table none of them fail (one will create the table, the others will exit without an error)
  2. Add support for dataset table expiration. This allows old/stale tables to be automatically cleaned up by GCP avoiding unlimited growth of tables.
  3. Introduce a "jobID" to collection to allow for caching across sources during runs.
  4. Use "jobID" in the depsdev source
    • table creation is now done during Count
    • stores a last usage cache to avoid reattempting to check if the table exists, etc

@calebbrown
Add "IF NOT EXISTS" to the caching query for deps.dev.
3b336e2 
This will *hopefully* avoid failures if multiple workers attempt to
create the table at the same time.

Signed-off-by: Caleb Brown <calebbrown@google.com>
@calebbrown
Set default TTL on dataset for deps.dev tables.
875962b 
This ensures tables only hang around for a short period of time.

Signed-off-by: Caleb Brown <calebbrown@google.com>
@calebbrown
Add support for deps.dev expiration to the criticality_score tool.
eea9ce0 
Signed-off-by: Caleb Brown <calebbrown@google.com>
@calebbrown
Add a jobID to collection to allow for per-job deps.dev tables.
3a6f732 
Signed-off-by: Caleb Brown <calebbrown@google.com>
@calebbrown
Fix lint warning
ae3dad1 
Signed-off-by: Caleb Brown <calebbrown@google.com>
@calebbrown calebbrown merged commit b1a355a into main Dec 1, 2022
@calebbrown calebbrown deleted the depsdev_improve branch December 1, 2022 02:11
nathannaveen pushed a commit to nathannaveen/criticality_score that referenced this pull request Feb 9, 2023
@calebbrown @nathannaveen
Ensure deps.dev data is fresh for each run of Criticality Score (ossf…
602d382 
…#267)

* Add "IF NOT EXISTS" to the caching query for deps.dev.

This will *hopefully* avoid failures if multiple workers attempt to
create the table at the same time.

Signed-off-by: Caleb Brown <calebbrown@google.com>

* Set default TTL on dataset for deps.dev tables.

This ensures tables only hang around for a short period of time.

Signed-off-by: Caleb Brown <calebbrown@google.com>

* Add support for deps.dev expiration to the criticality_score tool.

Signed-off-by: Caleb Brown <calebbrown@google.com>

* Add a jobID to collection to allow for per-job deps.dev tables.

Signed-off-by: Caleb Brown <calebbrown@google.com>

* Fix lint warning

Signed-off-by: Caleb Brown <calebbrown@google.com>

Signed-off-by: Caleb Brown <calebbrown@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@oliverchang oliverchang oliverchang approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@calebbrown @oliverchang
0