TLDR is a suspiciously-delightful testing framework for Ruby.
As a test library, TLDR is largely API-compatible with Minitest. As a test runner, TLDR boasts a few features RSpec's CLI still doesn't have.
The library, command line interface, and every decision in-between was prioritized to maximize productivity by promote fast feedback loops. Some highlights:
- Numerous ways to run specific tests: by path (
foo_test.rb), line number (foo_test.rb:13), name (--name test_foo), or regex pattern (-n /_foo_/) - Parallel test execution by default, as well as controls for serial execution of thread-unsafe tests
- Continuously run after every file change with
--watch - An optional timer to enforce your tests never get slow with
--timeout - A
--fail-fastflag that aborts the run as soon as a failure is encountered - Running your most-recently-edited test before all the others (see
--prepend) - Delightful diffs when assertions fail, care of super_diff
We hope you'll give it a try!
You can either read the rest of this README and learn about TLDR passively, or you can just clone this TLDR demo repo and work through its README as you play with its tests and run them with various options.
JUST IN CASE YOU'RE ALREADY SKIMMING THIS, I said stop reading and clone this interactive repo if you're a hands-on learner.
You know the drill. gem install tldr or add it to your Gemfile:
gem "tldr"
By default, TLDR expects your tests to be in test/ with filenames that match test_*.rb or *_test.rb and will require a test/helper.rb if you define one.
$ tldr --help
Usage: tldr [options] some_tests/**/*.rb some/path.rb:13 ...
-t, --[no-]timeout [TIMEOUT] Timeout (in seconds) before timer aborts the run (Default: 1.8)
--watch Run your tests continuously on file save (requires 'fswatch' to be installed)
--fail-fast Stop running tests as soon as one fails
--[no-]parallel Parallelize tests (Default: true)
-s, --seed SEED Random seed for test order (setting --seed disables parallelization by default)
-n, --name PATTERN One or more names or /patterns/ of tests to run (like: foo_test, /test_foo.*/, Foo#foo_test)
--exclude-name PATTERN One or more names or /patterns/ NOT to run
--exclude-path PATH One or more paths NOT to run (like: foo.rb, "test/bar/**", baz.rb:3)
--helper PATH One or more paths to a helper that is required before any tests (Default: "test/helper.rb")
--no-helper Don't require any test helpers
--prepend PATH Prepend one or more paths to run before the rest (Default: most recently modified test)
--no-prepend Don't prepend any tests before the rest of the suite
-l, --load-path PATH Add one or more paths to the $LOAD_PATH (Default: ["lib", "test"])
--base-path PATH Change the working directory for all relative paths (Default: current working directory)
-c, --[no-]config PATH The YAML configuration file to load (Default: '.tldr.yml')
-r, --reporter REPORTER Set a custom reporter class (Default: "TLDR::Reporters::Default")
--[no-]emoji Enable emoji output for the default reporter (Default: false)
--[no-]warnings Print Ruby warnings (Default: true)
-v, --verbose Print stack traces for errors
--yes-i-know Suppress TLDR report when suite runs beyond any configured --timeout
--print-interrupted-test-backtraces
Print stack traces of tests interrupted after a timeout
The tldr CLI will look for a .tldr.yml that can set the same set of options supported by the CLI file in the root of your project. You can specify a custom YAML location with --config some/path.yml if you need to.
Any options found in the dotfile will override TLDR's defaults, but can still be overridden by the tldr CLI or a TLDR::Config object passed to TLDR::Run.at_exit!.
Here's an example project that specifies a .tldr.yml file as well as some internal tests demonstrating its behavior.
If you've ever seen a Minitest test, then you already know how to write TLDR tests. Rather than document how to write tests, this section just highlights the ways TLDR tests differ from Minitest tests.
First, instead of inheriting from Minitest::Test, your test classes must subclass TLDR instead:
class MyTest < TLDR
def test_looks_familiar
assert true
end
endSecond, if your tests depend on a test helper, so long as you name it test/helper.rb, it will automatically be required by TLDR, so you don't need to add require "helper" at the top of each test. If you want to name the helper something else, you can do so with the --helper option:
tldr --helper test/test_helper.rb
Third, TLDR offers fewer features:
- No built-in mock library (use mocktail, maybe!)
- No "spec" API
- No benchmark tool
- No test bisect script
And that's it! You now know how to write TLDR tests.
Because TLDR ships with a CLI, it offers a lot of ways to run your tests.
Once installed, running all your tests is just five keystrokes away:
tldr
This assumes your tests are stored in test/. It will also add lib/ to Ruby's load paths and require test/helper.rb before your tests, if it exists.
TLDR ships with a minimal rake task that simply shells out to the tldr CLI by default. If you want to run TLDR with Rake, you can configure the task by setting flags on an env var named TLDR_OPTS or in a .tldr.yml file.
All your Rakefile needs is require "tldr/rake and you can run the task individually like this:
$ rake tldr
# Or, with options in TLDR_OPTS
$ TLDR_OPTS="--no-parallel" rake tldr
Here's an example Rakefile that runs both TLDR and Standard Ruby as the default task:
require "standard/rake"
require "tldr/rake"
task default: ["tldr", "standard:fix"]One reason you'd want to invoke TLDR with Rake is because you have multiple test suites that you want to be able to conveniently run separately (this talk discussed a few reasons why this can be useful).
To create a custom TLDR Rake task, you can instantiate TLDR::Task like this, which allows you to configure TLDR::Config in code:
require "tldr/rake"
TLDR::Task.new(name: :safe_tests, config: TLDR::Config.new(
paths: FileList["safe/**/*_test.rb"],
helper_paths: ["safe/helper.rb"],
load_paths: ["lib", "safe"]
))The above will create a second Rake task named safe_tests running a different set of tests than the default tldr task. Here's an example from TLDR's test suite.
The tldr CLI includes a --watch option that will watch for changes in any of the configured load paths (["test", "lib"] by default) and then execute your tests each time a file is changed. To keep the output up-to-date and easy
to scan, it will also clear your console before each run.
Note that this feature requires you have fswatch installed and on your PATH
Here's what that might look like with the --emoji flag enabled:
If you'd rather use TLDR by running Ruby files instead of the tldr CLI, you can simply require "tldr/autorun" (just like require "minitest/autorun").
Given a file test/some_test.rb:
require "tldr/autorun"
class SomeTest < TLDR
def test_truth
assert true
end
endYou can then run the test by passing ruby the file:
$ ruby test/some_test.rb
Any CLI options you add will still be parsed, as well (e.g. ruby my_test.rb --emoji will work).
If you want to be explicit about setting the Kernel.at_exit hook, or if you want to configure TLDR with code, you can invoke TLDR.at_exit! directly:
require "tldr"
TLDR::Run.at_exit! TLDR::Config.new(emoji: true)If we just want to know if the build passes, we want to know as fast as possible. Ever see a test fail and then sit around waiting for the whole suite to finish running anyway? Why wait? Turn on --fail-fast and abort the test run the instant a failure is encountered:
tldr --fail-fast
Additionally, you might notice the top of each run will show you a command you can use to execute the same run, like this:
Command: bundle exec tldr --fail-fast --prepend "test/calculator_test.rb"
That's because TLDR will look at the file system and move your most-recently-edited test file to the front of the queue with --prepend. When used in conjunction with --fail-fast, you'll fail extra fast, because the most likely test to fail is the one you're actively working on.
We initially developed TLDR because we wanted a test runner that supported suite-wide time limits as a first-class feature. When test suites become slow, people run them (much) less often. And once a developer gets in the habit of only running tests occasionally, it's not long before they only run them before push, and then only the ones they're immediately working on, and then they just wait for CI to run them in a pull request. And if you don't run all your tests very often, you don't feel any pain when you make your tests (or the code its testing) slower.
Each time you write code and don't run your tests, you're making an assumption that whatever code you just wrote works flawlessly. If that assumption is correct, you saved however much time it takes to run your tests. But every time that assumption is incorrect, you've just extended the amount of time before discovering that you broke something. If that's a few minutes later, that might only cost a few minutes of rework. If you only run tests once or twice a day, you might have to undo hours of work to fix it.
We originally came up with the idea of TLDR on a livestream, joking that only an unconfigurable 1.8 second time limit would prevent test suites from ballooning in duration over time. As of 1.0, we've made the timeout configurable, but we still think it's a good idea to enable it with the --timeout option:
tldr --timeout
Not only does a timeout keep us running the whole suite frequently (it'll never take more than 1.8 seconds, after all), but even if the suite begins to exceed our self-imposed timeout, TLDR's random test order and parallel execution means that—so long as you keep running that partial suite frequently—you'll still be running ALL your tests many more times than if they waited until some arbitrary checkpoint to run them.
When enabled, --timeout will set the timer to 1.8 seconds. But you can set whatever time limit you like. The right value is depends on your individual capacity for paying attention. Basically, "however long you're willing to wait without caving and running your tests less often."
Examples:
# A badass 200ms timeout
tldr --timeout 0.2
# A miserable-sounding 20 second timeout
tldr --timeout 20
If we've won you over towards this way of working, we suggest creating a .tldr.yml file in the root of your project and specifying your desired timeout.
For TLDR Classicâ„¢ and a 1.8s timeout:
# .tldr.yml
timeout: trueOr any number of seconds you like:
# .tldr.yml
timeout: 0.01And if you're running with the timeout enabled this way, you can still disable it for any given test run by adding the --no-timeout flag.
TLDR is very similar to Minitest in API, but different in enough ways that you probably have some questions.
The tldr gem was initially developed and released after we did a lighthearted pairing session and imagined a Ruby test runner with a CLI and an unorthodox, unconfigurable rule: a hard-and-fast 1.8 second time limit on every test suite.
In the 18 months since, and to our utter surprise, TLDR did not immediately rise to the top of the charts and dominate the Ruby testing world.
While we still contend the mandatory time limit was a Very Good Idea (even if your tests are necessarily slower than 1.8 seconds), we believe there is a very remote, almost certainly wrong possibility that it was slowing adoption of this otherwise very capable test runner. As a result, **as of 1.0.0, the 1.8s timeout is disabled by default, and can be re-enabled (and even set to a specific value) with the --timeout option.
Tests you write with tldr are designed to be mostly-compatible with Minitest tests.
Details:
- We implemented all of Minitest's built-in assertions (e.g.
assert,assert_equals) setupandteardownhook methods should work as you expect. (We even threw in our ownaroundhook, free of charge!)- If you need anything else from Minitest as you port tests to TLDR, try
include TLDR::MinitestCompatibility, and if that doesn't do the trick add whatever you need in a pull request
Read this before you add --no-parallel because some tests are failing when you run tldr.
The vast majority of test suites in the wild are run sequentially even though they'd work perfectly fine in parallel. Why? Because test runners tend not to enable it by default. Moreover, when they do, they usually rely on forking processes, which is slower and more resource-intensive than using threads or Ractors. For this reason, TLDR is optimistic by default and will multi-thread your test suite.
So, what do you do when you run into a situation where a test has good reason not to run in parallel? Either because of resource contention or because the order of its test cases actually matters? Here's what we'd do:
- Start by challenging the assumption that the test can't be run in a thread-safe way (it may be as simple as changing a globally-edited instance variable to a thread-local)
- If the problem is that a subset of your tests depend on the same resource, try using TLDR.run_these_together! class to group the tests together. This will ensure that those tests run in the same thread in sequence (here's a simple example)
- For tests that affect process-wide resources like setting the system clock or changing the process's working directory (i.e.
Dir.chdir), you can sequester them to run sequentially after all parallel tests in your suite have run with TLDR.dont_run_these_in_parallel!, which takes the same arguments asrun_these_together!(example) - Give up and make the whole suite
--no-parallel. If you find that you need to resort to this, you might save some keystrokes by addingparallel: falsein a .tldr.yml file
If you're currently using Minitest, you can take a stab at dropping your dependency on the minitest gem and replace references to Minitest::Test dynamically in a test helper, like I did for my todo_or_die gem:
require "tldr"
if defined?(Minitest::Test)
TLDR::MinitestTestBackup = Minitest::Test
Minitest.send(:remove_const, "Test")
end
module Minitest
class Test < TLDR
include TLDR::MinitestCompatibility
end
end
This probably won't work for complex projects, but it might for simple ones!
There is not.
Currently, the only pluggable aspect of TLDR are reporters, which can be set
with the --reporter command line option. It can be set to any fully-qualified
class name that responds to the same methods defined in
TLDR::Reporters::Base.
If you define a custom reporter, be sure to require it from your test helper, so TLDR can instantiate it!
There's a command-line utility named tldr that might conflict with this gem's executable in your PATH. If that's the case, you could either change your path, invoke bundle exec tldr, run with Rake, or use the tldt ("too long; didn't test") executable alias that ships with this gem.
If you want to submit PRs on this repo, please know that the code style is Kirkland-style Ruby, where method definitions have parentheses omitted but parentheses are generally expected for method invocations.
Thanks to George Sheppard for freeing up the tldr gem name!