8000 Intermittent ` text file busy` Error When Running Ginkgo Labelled test pack subsets in Parallel on Jenkins · Issue #1558 · onsi/ginkgo · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Intermittent text file busy Error When Running Ginkgo Labelled test pack subsets in Parallel on Jenkins #1558

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
NalisRI opened this issue May 16, 2025 · 2 comments
Closed

Intermittent text file busy Error When Running Ginkgo Labelled test pack subsets in Parallel on Jenkins #1558

NalisRI opened this issue May 16, 2025 · 2 comments

Comments

@NalisRI
Copy link
NalisRI commented May 16, 2025 •
edited
Loading

Hi @onsi

Thank you for Ginkgo, it's been an incredibly helpful tool for testing needs!

Question
Is it a known limitation that multiple Ginkgo processes cannot run concurrently from the same package directory, even when using mutually exclusive label filters?

Problem Summary
We’re encountering the following error when running multiple Ginkgo commands in parallel using Jenkins parallel stages:

fork/exec /workspace/integration/tests.test: text file busy
This happens intermittently

Setup

I am splitting around 600 tests into two subsets with labels as two make commands like this

Makefile targets

run-integration-tests:
	ginkgo --procs=8 --compilers=8 --label-filter="integration_test" ./integration/tests

run-non-integration-tests:
	ginkgo --procs=10 --compilers=10 --label-filter="!integration_test" ./integration/tests

calling these commands as parallel stages, as part of high-level test stage

Jenkinsfile stages

parallel([
                  "integration Tests": {
                      stage('integration Tests') {
                      try {
                           sh 'make run-integration-tests'
                         } catch (e) {
                            currentBuild.result = 'FAILED'
                            error("integration label tests have failed")
                            }
                      }
                  },
                  "non-integration Tests": {
                    stage('non-integration Tests') {
                    try {
                         sh 'make run-non-integration-tests'
                       } catch (e) {
                          currentBuild.result = 'FAILED'
                          error("non-integration Tests have failed")
                          }
                    }
       }])

I believe the issue arises because both commands are accessing the temporary test binary (e.g., tests.test).

NOTE: This happens intermittently
@NalisRI NalisRI changed the title text file busy Error When Running Ginkgo Test Binaries in Parallel on Jenkins Intermittent text file busy Error When Running Ginkgo Labelled test pack subsets in Parallel on Jenkins May 16, 2025
@onsi
Copy link
Owner
onsi commented May 17, 2025

hey there - hadn’t seen this before. i suppose one question might be - why split the run this way? why not just run

ginkgo -procs=18 ./integration/tests

and let Ginkgo parallelize things for you?

In any event - I suspect things are failing because the two processes are racing around compiling and deleting the temporary test binary. You can around this by first compiling the binary ginkgo build ./integration/tests (in series) and then running ginkgo -procs=8 -label-filter=… /path/to/binary etc. in parallel. not sure how hard/easy it is to do that in jenkins but it should work for you? if you’re saving off reports you’ll want to point them to different output directories.

@NalisRI
Copy link
Author
NalisRI commented May 19, 2025

Hi @onsi ,

Thanks a lot for the prompt response, really appreciate the insight!

Yes, I’ve now reverted to the single ginkgo -procs=18 ./integration/tests approach as you suggested. I initially explored splitting the suite using label filters across Jenkins parallel stages to see if it could reduce overall test wall time — but the gains were negligible, and it introduced complexity.

I also tried your suggestion of pre-compiling the test binary. Since ginkgo build doesn’t offer an -o flag to give a custom name to the output binary, I used go test -c -o to generate two differently named binaries for the separate label subsets.

That worked to some extent, but then ran into issues where test data files (which use relative paths) couldn't be located — presumably because the binary was being executed from outside its original package context. I would’ve needed to refactor the paths or move the data files, which wasn’t ideal.

So finally, I consolidated back to the single Ginkgo run with increased --procs, and switched to a larger Jenkins agent with more vCPUs. That approach seems to be working well for now.

Thanks again for the help — and Ginkgo continues to be a great tool for our large-scale test suites!

@NalisRI NalisRI closed this as completed May 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@onsi @NalisRI
0