8000 Fix slow cache restore on Windows by antontroshin · Pull Request #515 · actions/setup-go · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Fix slow cache restore on Windows #515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

Fix slow cache restore on Windows #515

wants to merge 16 commits into from

Conversation

antontroshin
Copy link
@antontroshin antontroshin commented Nov 19, 2024 •
edited
Loading

Description:
The issue with running setup-go and restoring relatively big cache files, causes the cache restore process to be very slow on Github Action Windows runners.
Similar to the change that was made in version v5.0.0 (PR #411), I've made a change to override the default GOCACHE and GOMODCACHE variables on Windows runners to be using drive D.
On average, restoring ~1.7Gb cache on Windows pipelines took about 13-15 minutes to extract.
With this change, the extraction time was reduced to ~1m.

Restore cache logs before the change:

Wed, 20 Nov 2024 02:24:29 GMT Cache Size: ~1784 MB (1870225672 B)
Wed, 20 Nov 2024 02:24:30 GMT Received 1870225672 of 1870225672 (100.0%), 295.9 MBs/sec
Wed, 20 Nov 2024 02:37:21 GMT Cache restored successfully
Restore elapsed: 12 min 51 sec

Wed, 20 Nov 2024 01:38:54 GMT Cache Size: ~1784 MB (1870225672 B)
Wed, 20 Nov 2024 01:38:55 GMT Received 1870225672 of 1870225672 (100.0%), 118.7 MBs/sec
Wed, 20 Nov 2024 01:54:19 GMT Cache restored successfully
Restore elapsed: 15 min 24 sec

Restore cache logs after the change:

Wed, 20 Nov 2024 03:38:03 GMT Cache Size: ~1780 MB (1866088667 B)
Wed, 20 Nov 2024 03:38:04 GMT Received 1866088667 of 1866088667 (100.0%), 197.3 MBs/sec 
Wed, 20 Nov 2024 03:38:59 GMT Cache restored successfully
Restore elapsed: 55 sec

Wed, 20 Nov 2024 03:38:10 GMT Cache Size: ~1780 MB (1866088667 B)
Wed, 20 Nov 2024 03:38:11 GMT Received 1866088667 of 1866088667 (100.0%), 197.6 MBs/sec
Wed, 20 Nov 2024 03:39:06 GMT Cache restored successfully
Restore elapsed: 55 sec

Wed, 20 Nov 2024 04:04:07 GMT Cache Size: ~1780 MB (1866088667 B)
Wed, 20 Nov 2024 04:04:07 GMT Received 1866088667 of 1866088667 (100.0%), 88.8 MBs/sec
Wed, 20 Nov 2024 04:05:05 GMT Cache restored successfully
Restore elapsed: 1 min 2 sec

Solution:

At first, I tried to implement a solution similar to #411 by using symlink, however, during testing, it was clear that the GOCACHE and GOMODCACHE directories were created early by the tooling, which did not present an easy way to create symlinks without deleting the already in-use directories and introducing bigger code changes.

The solution to override the path for env vars as early as possible only on Windows runners is easier.

This PR adds the code to make this change by default for Windows runners, this "workaround" is possible to use via the Github Action code itself, by manually overriding the env vars.
Preferably, this change would be accepted and merged (after review of course), so users can enjoy this improvement by default without manually modifying each and every Github action.

Related issue:
Numerous related issues were raised regarding this issue, some improvements were made to address it.

Check list:

  • Mark if documentation changes are required.
  • Mark if tests were added or updated to cover the changes.

@antontroshin
Add GOCACHE AND GOMODCACHE symlink on Windows
28b19b8 
Use D drive for faster cache restore

Signed-off-by: Anton Troshin <anton@diagrid.io>
@antontroshin
add logs
0859a18 
Signed-off-by: Anton Troshin <anton@diagrid.io>
@antontroshin
build
97e00a5 
Signed-off-by: Anton Troshin <anton@diagrid.io>
@antontroshin
Fix path replace
434fb17 
Signed-off-by: Anton Troshin <anton@diagrid.io>
@antontroshin
add check for existing symlink
0dc6404 
Signed-off-by: Anton Troshin <anton@diagrid.io>
@antontroshin
lint and build
6cd0eaa 
Signed-off-by: Anton Troshin <anton@diagrid.io>
@antontroshin
move symlink check
fd3fcb3 
Signed-off-by: Anton Troshin <anton@diagrid.io>
@antontroshin
add logs
896e2cb 
Signed-off-by: Anton Troshin <anton@diagrid.io>
@antontroshin
wrap with try, debugging errors
c8eefa5 
Signed-off-by: Anton Troshin <anton@diagrid.io>
@antontroshin
lint and build
f8cf508 
Signed-off-by: Anton Troshin <anton@diagrid.io>
@antontroshin
more logs and logic change
e3c077d 
Signed-off-by: Anton Troshin <anton@diagrid.io>
@antontroshin
fix
e91efc5 
Signed-off-by: Anton Troshin <anton@diagrid.io>
@antontroshin
remove empty cache folders to be able to create symlinks
5b1dffc 
Signed-off-by: Anton Troshin <anton@diagrid.io>
@antontroshin
try different approach by overriding go ENV GOCACHE and GOMODCACHE va…
be77556 
…rs for Windows

Signed-off-by: Anton Troshin <anton@diagrid.io>
@antontroshin
remove previous attempt logic
bda02de 
Signed-off-by: Anton Troshin <anton@diagrid.io>
@antontroshin antontroshin changed the title Add GOCACHE AND GOMODCACHE symlink on Windows Fix slow cache restore on Windows Nov 20, 2024
@antontroshin antontroshin marked this pull request as ready for review November 20, 2024 04:33
@antontroshin antontroshin requested a review from a team as a code owner November 20, 2024 04:33
@antontroshin antontroshin mentioned this pull request Nov 20, 2024
Merged
7 tasks
@y-bruin y-bruin mentioned this pull request Nov 26, 2024
Merged
@y-bruin
Copy link
y-bruin commented Dec 31, 2024

@antontroshin When do you plan to merge it?

@antontroshin
Copy link
Author

@antontroshin When do you plan to merge it?

@y-bruin Unfortunately, I cannot merge, as this is up to repo maintainers, once the PR is reviewed and approved.

@MatthewRyanRead MatthewRyanRead mentioned this pull request Jan 15, 2025
Closed
5 tasks
@lmb
Copy link
lmb commented Jan 21, 2025

For anyone else also waiting for this to get some attention:

    env:
      # Fix slow Go compile and cache restore
      # See https://github.com/actions/setup-go/pull/515
      GOCACHE: D:\gocache
      GOMODCACHE: D:\gomodcache
      GOTMPDIR: D:\gotmp

    steps:
      # Go requires gotmp to be present
      - run: mkdir D:\gotmp
        shell: pwsh

Setting GOTMPDIR to D: was necessary to cut down the time it took to go install from cache.

@kruskall kruskall mentioned this pull request Feb 10, 2025
Merged
2 tasks
@manugarg manugarg mentioned this pull request Feb 11, 2025
Merged
@bbockelm bbockelm mentioned this pull request Feb 18, 2025
Closed
@antontroshin
add GOTMPDIR override for Windows workflow
fdd3859 
add validation for GOCACHE, GOMODCACHE, and GOTMPDIR on Windows

Signed-off-by: Anton Troshin <anton@diagrid.io>
@douglascamata douglascamata mentioned this pull request Mar 12, 2025
Merged
bertschneider added a commit to steadybit/extension-host-windows that referenced this pull request Apr 23, 2025
@bertschneider
feat: improve go caching in CI pipeline
edbd88a 
See actions/setup-go#515
@bertschneider bertschneider mentioned this pull request Apr 23, 2025
Merged
bertschneider added a commit to steadybit/extension-host-windows that referenced this pull request Apr 23, 2025
@bertschneider
feat: improve go caching in CI pipeline
4fd42a9 
See actions/setup-go#515
@antontroshin
Copy link
Author

Hi @HarithaVattikuti, @priya-kinthali, @aparnajyothi-y, @priyagupta108
I'm very sorry for tagging you directly. Is it possible to request your review of this PR?
I, and many others, would benefit greatly from having this feature merged and enabled by default.
Gladly would address any comments or requests.
Thanks!

@aparnajyothi-y
Copy link
Contributor

Hello @antontroshin, Thank you for submitting this pull request. We will review and get back to you once we have some feedback on this :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@antontroshin @y-bruin @lmb @aparnajyothi-y
0