8000 Running tests with `ginkgo` significantly slower than `go test` · Issue #644 · onsi/ginkgo · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Running tests with ginkgo significantly slower than go test #644

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
charleskorn opened this issue Feb 10, 2020 · 15 comments
Closed

Running tests with ginkgo significantly slower than go test #644

charleskorn opened this issue Feb 10, 2020 · 15 comments

Comments

@charleskorn
Copy link
charleskorn commented Feb 10, 2020 •
edited
Loading

Thanks for the awesome project!

I'm seeing an odd issue: running my test suites with ginkgo takes significantly longer than running them with go test:

$ time go test -tags unitTests ./server/...
ok  	github.com/batect/abacus/server/api	0.496s
ok  	github.com/batect/abacus/server/cmd	0.335s
ok  	github.com/batect/abacus/server/middleware	0.195s
?   	github.com/batect/abacus/server/middleware/testutils	[no test files]
ok  	github.com/batect/abacus/server/storage	0.173s
        2.69 real         5.60 user         1.93 sys

$ time ~/go/bin/ginkgo -tags unitTests server/...
[1581318849] API Suite - 46/46 specs •••••••••••••••••••••••••••••••••••••••••••••• SUCCESS! 2.535598ms PASS
[1581318849] Application Suite - 0/0 specs  SUCCESS! 241.13µs PASS
[1581318849] Middleware Suite - 9/9 specs ••••••••• SUCCESS! 778.471µs PASS
[1581318849] Storage Suite - 3/3 specs ••• SUCCESS! 206.337µs PASS

Ginkgo ran 4 suites in 4.485361489s
Test Suite Passed
        4.49 real         9.35 user         5.09 sys

I've tried setting -compilers to a low number as suggested in #258, but this doesn't have a significant impact.

batect/abacus@f8d4c57 is the commit I'm testing this on.
@charleskorn charleskorn changed the title Running tests with ginkgo twice as slow as go test Running tests with ginkgo significantly slower than go test Feb 10, 2020
@toddkazakov
Copy link

At Tidepool we're also seeing roughly 2 times slower test execution times. We have a very large test suite, which takes more than 8 minutes to complete with ginkgo vs 3.5 minutes with go test. Unfortunately, test randomization is only supported by ginkgo. Please advice how to profile this.

@elamre
Copy link
elamre commented Apr 17, 2020

Also I notice a very slow performance of running ginkgo. Has any of you found a workaround?

@onsi
Copy link
Owner
onsi commented Apr 17, 2020

sorry for the radio silence, y'all. i'd like to get too the bottom of this. ginkgo uses vanilla go test but it first compiles all the test suites it discovers before running them. my instinct is that the slowdown happens during the compilation phase and likely has something to do with the evolution of go in recent versions towards module support.

specifically ginkgo compiles the test suites it discovers in parallel. so, a few requests:

  • can anyone here point me at a public repo that reproduces this problem (please include go version, etc.)
  • have you tried explicitly setting -compilers=1 to ensure there is no concurrency at all during compilation?

we'll get to the bottom of this. ginkgo's just using go test after all so it should be about as fast. 2x slower makes no sense.

@onsi
Copy link
Owner
onsi commented Apr 17, 2020

actually i see that @charleskorn is pointing to a repo that reproduces the issue. i'll dig in and report back

@onsi
Copy link
Owner
onsi commented Apr 21, 2020

I've made some progress outlined here:
https://github.com/onsi/ginkgo_perf

tl;dr: Ginkgo compiles test suites into a temp directory and then runs them. I'm observing a latency of O(500ms) when launching compiled test suites stored in temp directories. Changing Ginkgo's behavior to store compiled assets (temporarily) in the package path itself reduces this latency substantially.

Without this change, Ginkgo is O(500ms) slower than go test -c -o out.test && out.test. With this change it is about as fast.

Note that Ginkgo is still O(100ms) slower than go test - but this is a limitation that will be hard to address as Ginkgo's approach to parallelism requires creating a compiled test artifact.

With the new behavior I see the ginkgo unit tests go from O(16.2s) down to O(12.7s).

I'm pushing a commit with the new behavior. I don't expect it to address all the issues folks are experiencing, but it's a start. Please give it a whirl.

@toddkazakov I'd love more detail on your usecase. Any chance there's a repo I can clone that reproduces the issue?

onsi added a commit that referenced this issue Apr 21, 2020
@onsi
improve ginkgo performance - makes progress on #644
a14f98e
@onsi
Copy link
Owner
onsi commented Apr 21, 2020

I'll note that I don't understand why this change makes such a significant difference. And it may well be OS/hardware dependent (I'm running MacOS on a macbook pro with an SSD drive).

@toddkazakov
Copy link

Thanks for your help @onsi. The issue can be reproduced in https://github.com/tidepool-org/platform. The tests can be executed with make test, however they require mongo to be running at the default port on localhost.

The issue is more apparent in Travis due to the resource constraints. I tried removing all extra options (coverage, race condition detection, randomization, etc) and set the compilers to 1. There was no significant speed up. However, this still doesn't explain why plain go test is 4x faster.

Locally, the test compilation is parallelized and they run ok on a 6-core MBP with SSD.

@onsi
Copy link
Owner
onsi commented Apr 21, 2020

thanks @toddkazakov i’ll take a look. I assume you see the same discrepancy on your MPB between ginkgo and go test? And I see that you don’t have -p or -nodes set in your makefile so I assume you are running the tests without ginkgo’s parallelism, correct?

@onsi
Copy link
Owner
onsi commented Apr 21, 2020

hey @toddkazakov quick update:

  • i'm seeing mongod crap out with Abort trap: 6 when I run the test suite (and this, in turn, causes tests to hang/fail). any thoughts on what i'm doing wrong?
  • when i skip the mongo tests by passing -skipPackage=mongo to ginkgo I get the following run time:
    • With the previous version of the ginkgo CL (that uses tmp directories): 2m10s
    • With the new version (that doesn't use tmp directories): 1m46s

a difference of almost 30s! Can you give the latest commit a try and see if you're also seeing improvements?

@toddkazakov
Copy link
toddkazakov commented Apr 22, 2020 •
edited
Loading

The mongo error you're seeing is most likely caused by running a too recent version - we're using 3.6.

I ran a series of test today. Using the latest commit does provide a speed up locally, albeit small. I believe this will have a much bigger effect in Travis.

CI —compilers=1 / Latest commit
Ginkgo ran 155 suites in 5m3.807436627s
Test Suite Passed

real	5m3.867s
user	5m5.360s
sys	2m11.561s

vs the version we were using previously:

Ginkgo ran 155 suites in 5m17.166737852s
Test Suite Passed

real	5m17.236s
user	6m26.646s
sys	2m33.479s

I was able to make go test execution slower with go test -p 1 and adding the extra options that we use currently (-race being the one that slows down the process the most).

In my case I was comparing apples to oranges, but it was not obvious. Thank you for looking into this - I still think the result is very valuable.

@onsi
Copy link
Owner
onsi commented Apr 22, 2020

ah ok makes sense. yes the race detector will slow things down substantially.

@onsi
Copy link
Owner
onsi commented Apr 22, 2020

@elamre can you point me at a repo or give me a bit more detail so i can understand what sort of performance issues you are seeing?

@onsi
Copy link
Owner
onsi commented Apr 5, 2021 •
edited
Loading

Closing due to inactivity

@onsi onsi closed this as completed Apr 5, 2021
@lahabana lahabana mentioned this issue Jun 30, 2022
Open
@stroiman stroiman mentioned this issue Nov 7, 2024
Open
@jerbob92
Copy link
jerbob92 commented Feb 24, 2025 •
edited
Loading

Would it be an idea to limit this somehow?
If you have a computer with a lot of CPU cores, this can have dramatic effects on performance due to the amount of build processes it will start concurrently, this may also consume a lot of memory, which may cause the build process to become even slower because of swapping.

I'm also wondering if it's able to utilize caching efficiently like this if it start a lot of build processes at the same time of tests that share a lot of packages.

@onsi
Copy link
Owner
onsi commented Mar 19, 2025

Thanks to @myzhan over on #1481 some progress has been made here. It turns out that MacOS's XProtect substantially slows down Ginkgo's performance. I've added details and directions for how to disabled it to the documentation in case folks want to give it a try and see if it improves things:

https://onsi.github.io/ginkgo/#if-you-are-running-on-macos

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@onsi @jerbob92 @elamre @charleskorn @toddkazakov
0