8000 chore: Fix `no space left on device` error in `predictor-runtime-build` by ckadner · Pull Request #3049 · kserve/kserve · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

chore: Fix no space left on device error in predictor-runtime-build #3049

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Jul 25, 2023
Merged

chore: Fix no space left on device error in predictor-runtime-build #3049

merged 3 commits into from
Jul 25, 2023

Conversation

ckadner
Copy link
Member
@ckadner ckadner commented Jul 25, 2023 •
edited
Loading

What this PR does / why we need it:

A few months back the E2E tests exhausted the available disk space on the GitHub action runner nodes (#2950).

We added an additional action step to the minikube-setup composite action to remove unnecessary tools, libraries and caches before installing Minikube (#2972). This provided about 20 GB of additional disk space to build kserve images or pull large third-party images.

More recently other actions are running into similar disk space limits, like the predictor-runtime-build:

Run ishworkh/docker-image-artifact-upload@v1
Error: Error: Command failed: docker save kserve/custom-model-grpc:abf300d60f48813116af504c9357921d6199a754 -o /tmp/kserve_custom-model-grpc_abf300d60f48813116af504c9357921d6199a754
write /tmp/.docker_temp_2888533438: no space left on device

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Type of changes
Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Feature/Issue validation/testing:

  • predictor-runtime-build

Checklist:

  • Have you added unit/e2e tests that prove your fix is effective or that this feature works?
  • Has code been commented, particularly in hard-to-understand areas?
  • Have you made corresponding changes to the documentation?

Release note:

NONE

@ckadner
Separate action to free-up disk space on GH action runner
338cf72 
Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
@ckadner ckadner requested a review from yuzisun July 25, 2023 19:23
@ckadner ckadner marked this pull request as draft July 25, 2023 19:24
@ckadner ckadner self-assigned this Jul 25, 2023
rachitchauhan43
rachitchauhan43 approved these changes Jul 25, 2023
View reviewed changes
Copy link
Contributor
@rachitchauhan43 rachitchauhan43 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I just faced this issue on my last PR test run.
Thanks for the fix @ckadner

8000
@kserve-oss-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ckadner, rachitchauhan43
To complete the pull request process, please assign yuzisun after the PR has been reviewed.
You can assign the PR to them by writing /assign @yuzisun in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@rachitchauhan43
Copy link
Contributor

@ckadner : Is there anything pending on this for which you have kept it in draft mode ?

@ckadner
Move 'free-up-disk-space' step outside 'minikube-setup' action
5ac1e8b 
Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
@ckadner
Copy link
Member Author
ckadner commented Jul 25, 2023

@ckadner : Is there anything pending on this for which you have kept it in draft mode ?

Yup, I am using incremental commits, to break one thing at a time :-)

@ckadner
Copy link
Member Author
ckadner commented Jul 25, 2023

One more commit to come to plug the new action into other workflows that need more disk space.

@ckadner
Free-up disk space for the 'predictor-runtime-build'
d9114cf 
Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
@ckadner
Copy link
Member Author
ckadner commented Jul 25, 2023

The predictor-runtime-build completes without errors now.

https://github.com/kserve/kserve/actions/runs/5661529450/job/15339631407?pr=3049#step:12:88

@ckadner ckadner marked this pull request as ready for review July 25, 2023 21:11
@ckadner ckadner changed the title chore: Separate action to free-up disk space on GH action runner chore: Fix "no space left on device" error in predictor-runtime-build Jul 25, 2023
@ckadner ckadner changed the title chore: Fix "no space left on device" error in predictor-runtime-build chore: Fix no space left on device error in predictor-runtime-build Jul 25, 2023
@rachitchauhan43
Copy link
Contributor

/lgtm

@ckadner ckadner merged commit a8b9046 into kserve:master Jul 25, 2023
@rachitchauhan43
Copy link
Contributor

@ckadner : I saw this Python package test failed for your PR too https://github.com/kserve/kserve/actions/workflows/python-test.yml?query=branch%3Afree-up-disk-space-action.
How did you fix this ?

rhuss pushed a commit to rhuss/kserve that referenced this pull request Jul 28, 2023
@ckadner @rhuss
chore: Fix no space left on device error in `predictor-runtime-buil…
702dbde 
…d` (kserve#3049)

* Create a separate action to free-up disk space on GH action runner
* Move 'free-up-disk-space' step outside 'minikube-setup' action
* Free-up disk space before running 'predictor-runtime-build'

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
greenmoon55 pushed a commit to greenmoon55/kserve that referenced this pull request Aug 2, 2023
@ckadner @greenmoon55
chore: Fix no space left on device error in `predictor-runtime-buil…
757d6bd 
…d` (kserve#3049)

* Create a separate action to free-up disk space on GH action runner
* Move 'free-up-disk-space' step outside 'minikube-setup' action
* Free-up disk space before running 'predictor-runtime-build'

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
Signed-off-by: Jin Dong <jdong183@bloomberg.net>
yuzisun added a commit that referenced this pull request Aug 7, 2023
@yuzisun @sivanantha321
@ddelange @ckadner @rafvasq @tessapham @andyi2it
Cherry-pick: Release 0.11 branch (#3067)
1529b71 
* Fix configmap docs removed by kustomize build (#3024)

Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>

* Loosen ray version range (#2993)

* Loosen ray range

Signed-off-by: ddelange <14880945+ddelange@users.noreply.github.com>

* Fix import of RayServeHandle

ref ray-project/ray#34714

Signed-off-by: ddelange <14880945+ddelange@users.noreply.github.com>

* Allow pandas 2.0

Signed-off-by: ddelange <14880945+ddelange@users.noreply.github.com>

* Remove exclusion markers

Signed-off-by: ddelange <14880945+ddelange@users.noreply.github.com>

* Run poetry lock --no-update

Signed-off-by: ddelange <14880945+ddelange@users.noreply.github.com>

---------

Signed-off-by: ddelange <14880945+ddelange@users.noreply.github.com>

* Retry on resource conflict error (#3035)

Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>

* chore: Fix `no space left on device` error in `predictor-runtime-build` (#3049)

* Create a separate action to free-up disk space on GH action runner
* Move 'free-up-disk-space' step outside 'minikube-setup' action
* Free-up disk space before running 'predictor-runtime-build'

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>

* Update MM version to v0.11.0-rc1 (#3047)

Signed-off-by: Rafael Vasquez <raf.vasquez@ibm.com>

* Bump scikit-learn version to 1.3.0 (#3040)

* Bump scikit-learn version to 1.3.0

Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>

* Update test models

Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>

* Downgrade ray serve to 2.4

Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>

---------

Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>

* template out kube-rbac-proxy image (#3048)

* template out kube-rbac-proxy image

Signed-off-by: Tessa Pham <hpham111@bloomberg.net>

* add kserve-addressable-resolver to clusterroles

Signed-off-by: Tessa Pham <hpham111@bloomberg.net>

---------

Signed-off-by: Tessa Pham <hpham111@bloomberg.net>

* Add ARM64 arch support for serving runtimes (#3038)

Except for Paddle, which does not support arm64/aarch64

Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>

* Support secret name overrides with annotation (#3034)

* Add annotation for secret name overrides

Signed-off-by: Dan Sun <dsun20@bloomberg.net>

* Remove the field

Signed-off-by: Dan Sun <dsun20@bloomberg.net>

* Fix secret name annotation

Signed-off-by: Dan Sun <dsun20@bloomberg.net>

* Update kserve url storage example docs

Signed-off-by: Dan Sun <dsun20@bloomberg.net>

* Address review comments

Signed-off-by: Dan Sun <dsun20@bloomberg.net>

* Update config docs for storage secret

Signed-off-by: Dan Sun <dsun20@bloomberg.net>

* Fix storage secret name

Signed-off-by: Dan Sun <dsun20@bloomberg.net>

* Add test for secret name annotation override

Signed-off-by: Dan Sun <dsun20@bloomberg.net>

* Fix flake8 lint

Signed-off-by: Dan Sun <dsun20@bloomberg.net>

---------

Signed-off-by: Dan Sun <dsun20@bloomberg.net>

* release: Update defaultModelMeshVersion to v0.11.0 (#3059)

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>

* chore: Rename ModelMesh validating webhook (#3064)

Resolves #3063

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>

* Release v0.11.0 (#3061)

Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>

* Update poetry lock files (#3066)

Signed-off-by: Dan Sun <dsun20@bloomberg.net>

---------

Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
Signed-off-by: ddelange <14880945+ddelange@users.noreply.github.com>
Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
Signed-off-by: Rafael Vasquez <raf.vasquez@ibm.com>
Signed-off-by: Tessa Pham <hpham111@bloomberg.net>
Signed-off-by: Dan Sun <dsun20@bloomberg.net>
Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>
Co-authored-by: Sivanantham <90966311+sivanantha321@users.noreply.github.com>
Co-authored-by: ddelange <14880945+ddelange@users.noreply.github.com>
Co-authored-by: Christian Kadner <ckadner@us.ibm.com>
Co-authored-by: Rafael Vasquez <raf.vasquez@ibm.com>
Co-authored-by: Tessa Pham <hpham111@bloomberg.net>
Co-authored-by: Andrews Arokiam <87992092+andyi2it@users.noreply.github.com>
greenmoon55 pushed a commit to greenmoon55/kserve that referenced this pull request Aug 10, 2023
@ckadner @greenmoon55
chore: Fix no space left on device error in `predictor-runtime-buil…
c552b71 
…d` (kserve#3049)

* Create a separate action to free-up disk space on GH action runner
* Move 'free-up-disk-space' step outside 'minikube-setup' action
* Free-up disk space before running 'predictor-runtime-build'

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
Signed-off-by: Jin Dong <jdong183@bloomberg.net>
Iamlovingit pushed a commit to Iamlovingit/kserve that referenced this pull request Oct 1, 2023
@ckadner @Iamlovingit
chore: Fix no space left on device error in `predictor-runtime-buil…
7dea17c 
…d` (kserve#3049)

* Create a separate action to free-up disk space on GH action runner
* Move 'free-up-disk-space' step outside 'minikube-setup' action
* Free-up disk space before running 'predictor-runtime-build'

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
Signed-off-by: iamlovingit <freecode666@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rachitchauhan43 rachitchauhan43 rachitchauhan43 approved these changes

@yuzisun yuzisun Awaiting requested review from yuzisun

@sukumargaonkar sukumargaonkar Awaiting requested review from sukumargaonkar

Assignees

@rachitchauhan43 rachitchauhan43

@ckadner ckadner

Labels
area/testing lgtm
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ckadner @kserve-oss-bot @rachitchauhan43
0