There are plenty of unit/regression testing tools for Emacs, and even some for functional testing. What Emacs is missing though is a really good testing framework for integration testing. This is where Ecukes comes in.
Cucumber is a great integration testing tool, used mostly for testing web applications. Ecukes is Cucumber for Emacs. No, it's not a major mode to edit feature files. It is a package that makes it possible to write Cucumber like tests for your Emacs packages.
Ecukes is not a complete clone of Cucumber and is not intended to be. If however Ecukes is missing some feature that you feel really should be included, please make a bug report.
If you don't know anything about Cucumber I suggest you read up a bit about it before continuing with Ecukes.
Debugging
Example
Contribution
This method uses Cask and installs Ecukes from MELPA.
Add ecukes to your Cask file:
(source melpa)
(package "super-duper" "0.0.1" "Super Duper.")
(development
(depends-on "ecukes"))Then run cask to install.
Create an Elisp file in the root directory called super-duper.el and
(provide 'super-duper). To setup Ecukes for the project, run:
$ cask exec ecukes new
create features
create step-definition
create super-duper-steps.el
create support
create env.el
create super-duper.feature
The project will now look like this:
$ ls super-duper
super-duper/
|-- README
|-- features
| |-- step-definitions
| | |-- super-duper-steps.el
| |-- super-duper.feature
| |-- support
| |-- env.el
|-- super-duper.el
đź’ˇ Eask is very similar to Cask; anything that applies to Cask will apply to Eask
This method uses Eask and installs Ecukes from MELPA.
- Add
(depends-on "ecukes")toEaskfile of your project. - Run
eask install-deps. eask exec ecukes [COMMAND] [OPTIONS]to start the test.
To use the development version of Ecukes, you can clone the repository
and use the eask link feature to use the code from the clone.
git clone https://github.com/ecukes/ecukes.gitsomewhere to your computer.- Add
(depends-on "ecukes")toEaskfile of your project. - Run
eask link add ecukes <path-to-ecukes-repo>. eask exec ecukes [COMMAND] [OPTIONS]to start the test.
First off, install cucumber.el to get proper syntax highlighting and indenting.
To run Ecukes, use:
$ cask exec ecukes [COMMAND] [OPTIONS]with Eask:
$ eask exec ecukes [COMMAND] [OPTIONS]If you add a file called .ecukes in the project root, options are
automatically used for each run. For example:
--tags ~@exclude
--no-win
--reporter progress
Ecukes can run in a few different modes: Script (--script), no win
(--no-win) and win (--win).
If no mode is specified, the features will run in script mode. This
will run the features as a batch job (emacs --script or emacs --batch).
This mode has a few quirks and it's recommended as a way to run
features in local development, because it's fast and the output is
printed directly (there's no way in Emacs to print to stdout or
stderr when not running as a batch job).
It is for example not possible to test font faces in script mode. There's also no concept of lines so it's not possible to make assertions on lines.
If you run features on travis, use --no-win mode.
Example:
$ cask exec ecukes
$ cask exec ecukes --script # same as aboveIn this mode, Ecukes will run Emacs with the -nw option.
Example:
$ cask exec ecukes --no-winThis will start up an Emacs window and run the features. To use this
mode, use the --win option.
Example:
$ cask exec ecukes --winFeatures and scenarios can be tagged using syntax @tag, for example:
@core
Feature: Foo
@wip @io
Scenario: Bar
@io
Scenario: Baz
All tags on the feature will be copied down to the scenarios.
You can choose to run scenarios tagged with a tag using:
$ cask exec ecukes --tags @ioor run scenarios not tagged with a tag using:
$ cask exec ecukes --tags ~@ioor combine the two:
$ cask exec ecukes --tags @io,~@wipEcukes has support for reporters so that you can get the output in a few different ways depending on your philosophical bent. To list all available reporters, run:
$ cask exec ecukes list-reportersTo use a specific reporter, run:
$ cask exec ecukes --reporter progressDefault reporter is dot.
You can run scenarios matching or not matching a pattern using
--pattern (-p) and --anti-pattern (-a) respectively. For example:
$ cask exec ecukes --pattern awesome
$ cask exec ecukes -p awesome almost-awesome$ cask exec ecukes --anti-pattern awesome
$ cask exec ecukes -a awesome almost-awesomeIf you have a failing scenario and you only want to run that, you can
use a tag, but you can also use the --only-failing (-f)
option. For example:
$ cask exec ecukes --only-failingThere are three different kind of steps: Regular, table and py string. Check out Espuds for a comprehensive collection of steps.
A regular step always only consumes one line. The most basic step looks like this:
Given a known state
The corresponding step definition is:
(Given "a known state"
(lambda ()
;; ...
))The second argument could also have been the symbol of a function name:
(Given "a known state" 'do-something)Steps accept arbitrary arguments. The arguments sent to the step definition function are all regular expression match groupings from the matching of the step name and the match string.
Single argument:
Given I am in buffer "buffer-name"
The corresponding step definition is:
(Given "I am in buffer \"\\(.+\\)\""
(lambda (buffer-name)
;; ...
))
Multiple arguments:
Given I am in buffer "buffer-name" with text "Foo"
The corresponding step definition is:
(Given "I am in buffer \"\\(.+\\)\" with text \"\\(.+\\)\""
(lambda (buffer text)
;; ...
))A table step looks like this:
Given these meals:
| meal | price |
| Hamburger | $4.50 |
| Pizza | $5.30 |
The corresponding step definition is:
(Given "these meals:"
(lambda (meals)
;; ...
))
The argument meals is a simple list where the car of the list is
the header and the cdr or the list are the rows.
The header would in the above case be:
("meal" "price")And the rows would be:
(("Hamburger" "$4.50")
("Pizza" "$5.30"))To pick out the header and rows attribute from a table you do:
(let* ((table ...)
(header (car table))
(rows (cdr table)))
;; ...
)
It is also possible to send arguments to a table step. The table list will be the last argument.
Given these meals at "fast food":
| meal | price |
| Hamburger | $4.50 |
| Pizza | $5.30 |
The corresponding step definition is:
(Given "these meals at \"\\(.+\\)\":"
(lambda (restaurant meals)
;; ...
))A Py string step looks like this:
Given the following text:
"""
some text
"""
The corresponding step definition is:
(Given "the following text:"
(lambda (text)
;; ...
))It is also possible to send arguments to a Py string step. The Py string will be the last argument.
Given the following text in buffer "buffer-name":
"""
some text
"""
The corresponding step definition is:
(Given "the following text in buffer \"\\(.+\\)\":"
(lambda (buffer text)
;; ...
))In order to keep your steps DRY, you can call steps from other steps like this:
(Given "I go to a"
(lambda ()
;; Go to a
))
(Given "I go to b"
(lambda ()
;; Go to b
))
(Given "I go from a to b"
(lambda ()
(Given "I go to a")
(Given "I go to b")))You can also pass arguments when calling other steps, like this:
(Given "I am in buffer \"%s\"" buffer-name-variable)Or like this:
(Given "I am in buffer \"buffer-name\"")If you pass one extra parameter to a step definition, that is used as a callback. Once called, the step is considered done.
(When "^run command \\(.+\\)$"
(lambda (command callback)
(run-command command callback)))If callback is not called within
8000
10 seconds, it will fail
automatically. The timeout can be changed using the --timeout
(-t), for example:
$ cask exec ecukes --timeout 30Checkout the features for Async for examples how to use: https://github.com/rejeep/ecukes.el/blob/master/features/async.feature
You can list all defined steps with list-steps, for example:
$ cask exec ecukes list-stepsIf you want to get the step description, use the --with-doc option:
$ cask exec ecukes list-steps --with-docAnd if you want to know where it is defined, use the --with-file option:
$ cask exec ecukes list-steps --with-fileEcukes provides a few different hooks. They are useful if you test your program and change the state in some feature. Since all scenarios execute in the same environment, the state change will affect all scenarios after. You can solve that by resetting the state in a before or after hook.
Hooks should be placed in any file in the project features/support
directory.
Runs once before anything runs.
(Setup
;; Run code
)Runs once before each scenario runs.
(Before
;; Run code
)Runs once after each scenario runs.
(After
;; Run code
)Runs once when everything is done.
(Teardown
;; Run code
)Runs when a scenario fails.
(Fail
;; Run code
)If you have a failing scenario and you cant figure out how why, first
off, try adding the --debug flag. For example:
$ cask exec ecukes --debugNext step if that doesn't help, use the --error-log (-l)
argument. If an error occurs, the backtrace will be logged to that
file.
$ cask exec ecukes --error-log ecukes.errTo get an idea of how Ecukes can be used, here is an example from a feature in drag-stuff.
~/Code/drag-stuff $ cask exec ecukes features/line.feature --reporter spec
Feature: Drag line
In order to move a line up and down
As an Emacs user
I want to drag it
Background:
Given I am in the buffer "*drag-stuff*"
And the buffer is empty
And I insert
"""
line 1
line 2
"""
Scenario: Drag line up
When I go to line "2"
And I press "<M-up>"
Then I should see:
"""
line 2
line 1
"""
Scenario: Drag line down
When I go to line "1"
And I press "<M-down>"
Then I should see:
"""
line 2
line 1
"""
2 scenarios (0 failed, 2 passed)
11 steps (0 failed, 11 passed)
All contributions are much welcome and appreciated!
Before submitting a patch, make sure to write a test for it (if possible). Ecukes is unit tested with a testing framework called Ert (Emacs Lisp Regression Testing). But most notably, it's tested using Ecukes! :)
To run the tests, you have to install Eask if you haven't already. Once
installed, run the eask command to install all dependencies.
To run all tests, simply run the make command.
You can always report issues on Github if you're not up to fixing it yourself.