8000 Track GitHub release date by coverbeck · Pull Request #5950 · dockstore/dockstore · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Track GitHub release date #5950

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 15 commits into from
Aug 6, 2024
Merged

Track GitHub release date #5950

merged 15 commits into from
Aug 6, 2024

Conversation

coverbeck
Copy link
Collaborator
@coverbeck coverbeck commented Jul 26, 2024 •
edited
Loading

Description

Stores the release date of the latest GitHub release on the workflow. This will be used when searching for DOIs generated by the Zenodo-GitHub integration.

We do this by processing the GitHub release event -- this requires updating our GitHub app to subscribe to the release event; it does not require updated permissions.

Background

Zenodo issues a DOI at some point after a GitHub release has been published. I've seen it take 8 minutes, presumably it can take even longer. So when a GitHub release happens, store the timestamp of the release. Then other code, not in this PR, will check with Zenodo for x amount of time after the release to see if a DOI has been issued.

This took a while to do because I was initially going to track this at the version level. I ended up ditching that approach because:

  1. The logic gets complicated because we might get the release event before the version has been created.
  2. It's just as easy to search for workflow DOI as it is for a particular version's DOI in the Zenodo API; there's no performance benefit to searching for the version only.

Todo

  • Update the lambda to pass on release events.
  • Update GitHub delivery code in dockstore-support
  • Write code that polls for new Zenodo DOIs.

Review Instructions
Note: These are overall review instructions for all the PRs associated with SEAB-6466:

  1. Have a dockstore.yml enabled workflow
  2. In the GitHub UI, for the GitHub repository for the workflow from step 1, create and publish a GitHub release. It should not be a pre-release.
  3. Wait a couple of minutes for the release to be processed by Dockstore.
  4. Check the GitHub app logs for the org containing the workflow from step 1. You should see a release event in the logs.
  5. Fetch the workflow via the proprietary API (it's not in the GA4GH API), and verify that the field latestReleaseDate is set to the time of the release.

Variants:

  1. Do a pre-release on GitHub. After the 2 minute wait, it should not show up in the logs nor update latestReleaseDate.
  2. Then publish the pre-release. It should be represented in Dockstore.

Issue
SEAB-6466

Security and Privacy

If there are any concerns that require extra attention from the security team, highlight them here and check the box when complete.

  • Security and Privacy assessed

e.g. Does this change...

  • Any user data we collect, or data location?
  • Access control, authentication or authorization?
  • Encryption features?

Please make sure that you've checked the following before submitting your pull request. Thanks!

  • Check that you pass the basic style checks and unit tests by running mvn clean install
  • Ensure that the PR targets the correct branch. Check the milestone or fix version of the ticket.
  • Follow the existing JPA patterns for queries, using named parameters, to avoid SQL injection
  • If you are changing dependencies, check the Snyk status check or the dashboard to ensure you are not introducing new high/critical vulnerabilities
  • Assume that inputs to the API can be malicious, and sanitize and/or check for Denial of Service type values, e.g., massive sizes
  • Do not serve user-uploaded binary images through the Dockstore API
  • Ensure that endpoints that only allow privileged access enforce that with the @RolesAllowed annotation
  • Do not create cookies, although this may change in the future
  • If this PR is for a user-facing feature, create and link a documentation ticket for this feature (usually in the same milestone as the linked issue). Style points if you create a documentation PR directly and link that instead.

@coverbeck coverbeck self-assigned this Jul 26, 2024
@coverbeck coverbeck force-pushed the feature/seab-6466/trackreleases branch from 4a20923 to 0fb9bdc Compare July 26, 2024 00:28
@codecov Codecov
Copy link
codecov bot commented Jul 26, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 67.18750% with 21 lines in your changes missing coverage. Please review.

Project coverage is 73.02%. Comparing base (70ec770) to head (d4e8a8b).
Report is 1 commits behind head on develop.

Files Patch % Lines
...kstore/webservice/core/webhook/ReleasePayload.java 30.43% 16 Missing ⚠️
...ckstore/webservice/resources/WorkflowResource.java 89.65% 1 Missing and 2 partials ⚠️
...java/io/dockstore/webservice/core/LambdaEvent.java 0.00% 2 Missing ⚠️
Additional details and impacted files 
@@              Coverage Diff              @@
##             develop    #5950      +/-   ##
=============================================
- Coverage      74.23%   73.02%   -1.22%     
+ Complexity      5384     5308      -76     
=============================================
  Files            376      378       +2     
  Lines          19501    19564      +63     
  Branches        2035     2036       +1     
=============================================
- Hits           14477    14287     -190     
- Misses          4050     4278     +228     
- Partials         974      999      +25
Flag Coverage Δ
bitbuckettests 26.84% <1.56%> (-0.09%) ⬇️
hoverflytests 27.46% <1.56%> (-0.09%) ⬇️
integrationtests 56.85% <67.18%> (+0.05%) ⬆️
languageparsingtests 11.06% <1.56%> (-0.04%) ⬇️
localstacktests 21.54% <1.56%> (-0.07%) ⬇️
toolintegrationtests 30.22% <1.56%> (-0.10%) ⬇️
unit-tests_and_non-confidential-tests 25.86% <1.56%> (-0.08%) ⬇️
workflowintegrationtests 34.41% <1.56%> (-3.81%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@coverbeck coverbeck force-pushed the feature/seab-6466/trackreleases branch from 2649dc3 to 47342d5 Compare July 31, 2024 00:18
coverbeck
8000 coverbeck commented Jul 31, 2024
View reviewed changes
dockstore-webservice/src/main/java/io/dockstore/webservice/core/Workflow.java
@@ -161,6 +162,10 @@ public abstract class Workflow extends Entry<Workflow, WorkflowVersion> {
@JoinColumn(name = "actualDefaultVersion", referencedColumnName = "id", unique = true)
private WorkflowVersion actualDefaultVersion;

@Column(nullable = true)
@Schema(type = "integer", format = "int64", description = "The timestamp of the most recent version control release, such as a GitHub release")
private Timestamp latestReleaseDate;
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I put this in Workflow instead of Entry because the only tools we support in the .dockstore.yml are AppTools. But I could see an argument to put it in Entry.

denis-yuen
denis-yuen approved these changes Jul 31, 2024
View reviewed changes
Copy link
Member
@denis-yuen denis-yuen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presumably will need note in 1.16 deploy instructions to update app

@coverbeck coverbeck mentioned this pull request Aug 3, 2024
Merged
1 task
@coverbeck
Copy link
Collaborator Author

Re-requesting review.

  1. I wasn't recording a lambda event.
  2. Format for date string was wrong, discovered in manual integration testing.

@coverbeck
Copy link
Collaborator Author

Presumably will need note in 1.16 deploy instructions to update app

Added

@coverbeck coverbeck requested a review from denis-yuen August 3, 2024 00:24
@coverbeck coverbeck requested review from svonworl and kathy-t August 3, 2024 00:24
kathy-t
kathy-t approved these changes Aug 6, 2024
View reviewed changes
dockstore-webservice/src/main/java/io/dockstore/webservice/resources/WorkflowResource.java
}
return Response.status(HttpStatus.SC_NO_CONTENT).build();
}

private LambdaEvent createReleaseLambdaEvent(String deliveryId, ReleasePayload payload) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May need to create a UI PR at some point to add a friendly value for the RELEASE lambda event

https://github.com/dockstore/dockstore-ui2/blob/c112718f83b8596bbbd695f7e0e39b3add9c8d38/src/app/search/map-friendly-values.pipe.ts#L119-L123

@sonarqubecloud SonarQubeCloud
Copy link
sonarqubecloud bot commented Aug 6, 2024

Quality Gate Failed Quality Gate failed

Failed conditions
66.7% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud

@coverbeck coverbeck merged commit ee8503a into develop Aug 6, 2024
16 of 19 checks passed
@coverbeck coverbeck deleted the feature/seab-6466/trackreleases branch August 6, 2024 23:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@denis-yuen denis-yuen denis-yuen approved these changes

@kathy-t kathy-t kathy-t approved these changes

@svonworl svonworl svonworl approved these changes

@david4096 david4096 Awaiting requested review from david4096

Assignees

@coverbeck coverbeck

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@coverbeck @denis-yuen @kathy-t @svonworl
0