Closed
Description
We are revamping the scan queue. The new process will have scancode.io workers get jobs from the purldb by visiting an API endpoint that returns the next available scan queue entry to be worked on. (api/scan_queue/get_next_download_url). When the worker is done with the scan, it will post the results back to api/scan_queue/submit_scan_results.
TODO:
- Guard indexing endpoints with special API key
- Update scan queue to create new scan queue entries whenever we reindex a package, rather than reusing existing entry
- Create scancode.io management command that gets jobs from
api/scan_queue/get_next_download_urland sends results back- Get download url
- Create project, get input, run pipeline
- Return results
Consider:
- POST scancode io worker info before getting download url so we can tell which worker took what job
Metadata
Metadata
Assignees
Labels
No labels