Migrate GitHub importer to aboutcode pipeline #1584
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
- Refrence Use the SCIO Pipeline Mechanism for Importers/Improvers #1509
36 tasks
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
6d4cc76 to
bcd01d0
Compare
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
|
Import log from a clean run (only for NuGet ecosystem). ❯ python3 manage.py import github_importer 2>&1 | tee -a github-importer.log
INFO 2024-09-27 16:29:40.954 Pipeline [GitHubAPIImporterPipeline] starting
INFO 2024-09-27 16:29:40.954 Step [collect_and_store_advisories] starting
INFO 2024-09-27 16:29:41.347 Collecting 2,281 advisories
INFO 2024-09-27 16:29:53.042 Progress: 10% (229/2281) ETA: 105 seconds (1.8 minutes)
INFO 2024-09-27 16:30:06.163 Progress: 20% (457/2281) ETA: 99 seconds (1.6 minutes)
INFO 2024-09-27 16:30:12.833 Progress: 30% (685/2281) ETA: 73 seconds (1.2 minutes)
INFO 2024-09-27 16:30:23.018 Progress: 40% (913/2281) ETA: 63 seconds (1.1 minutes)
INFO 2024-09-27 16:30:38.787 Progress: 50% (1141/2281) ETA: 57 seconds
INFO 2024-09-27 16:30:52.844 Progress: 60% (1369/2281) ETA: 48 seconds
INFO 2024-09-27 16:30:58.406 Progress: 70% (1597/2281) ETA: 33 seconds
INFO 2024-09-27 16:31:06.659 Progress: 80% (1825/2281) ETA: 21 seconds
INFO 2024-09-27 16:31:19.738 Progress: 90% (2053/2281) ETA: 11 seconds
INFO 2024-09-27 16:31:34.287 Progress: 100% (2281/2281)
INFO 2024-09-27 16:31:34.291 Successfully collected 2,281 advisories
INFO 2024-09-27 16:31:34.291 Step [collect_and_store_advisories] completed in 113 seconds (1.9 minutes)
INFO 2024-09-27 16:31:34.291 Step [import_new_advisories] starting
INFO 2024-09-27 16:31:34.292 Importing 2,281 new advisories
INFO 2024-09-27 16:31:37.507 Progress: 10% (229/2281) ETA: 29 seconds
INFO 2024-09-27 16:31:41.146 Progress: 20% (457/2281) ETA: 27 seconds
INFO 2024-09-27 16:31:44.656 Progress: 30% (685/2281) ETA: 24 seconds
INFO 2024-09-27 16:31:48.454 Progress: 40% (913/2281) ETA: 21 seconds
INFO 2024-09-27 16:31:52.471 Progress: 50% (1141/2281) ETA: 18 seconds
INFO 2024-09-27 16:31:56.287 Progress: 60% (1369/2281) ETA: 15 seconds
INFO 2024-09-27 16:32:01.159 Progress: 70% (1597/2281) ETA: 12 seconds
INFO 2024-09-27 16:32:05.411 Progress: 80% (1825/2281) ETA: 8 seconds
INFO 2024-09-27 16:32:09.978 Progress: 90% (2053/2281) ETA: 4 seconds
INFO 2024-09-27 16:32:15.003 Progress: 100% (2281/2281)
INFO 2024-09-27 16:32:15.022 Successfully imported 2,281 new advisories
INFO 2024-09-27 16:32:15.022 Step [import_new_advisories] completed in 41 seconds
INFO 2024-09-27 16:32:15.022 Pipeline completed in 154 seconds (2.6 minutes) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.