qCumber is a unit testing library for q, which offers assertion testing, benchmark testing, and property-basedtesting. This library is provided both as a q API and as a q command line script. Additionally, there is integrationwithin Developer which provides Right-click > Test capabilities on repositories and test files.
Tests are stored in files with the .quke
extension. An example of a simple test file (count.quke
) is shown below.
feature Count should return the count of its input expect count on an atom to return 1 1 ~ count `a expect count of 1 drop list to return n - 1 n:1 + first 1?10; l:1 _ n?`8; count[l] ~ n - 1
The above feature test (feature
) contains one described behavior (should
) which contains two assertions (expect
)to check the the validity of the behavior. The feature
, should
, and expect
blocks contain natural language,allowing simple organization and structure of tests, as well as clear specification of intent in human-readable terms.
expect
assertions are always contained in a should
, and should
declarations are always contained in a feature
. Each .quke
test file can have one or more feature
s.
expect
assertions contain q code. The final statement in the code block should be true (1b
) to result in apassing test. For all other return values, it fails. Local assignments made within an expect
block do notimpact other expect
blocks.
Running tests
There are three ways of executing tests in qCumber: command line, API, or within the Developer IDE.
Command line
qCumber tests can be run at the command line by using the qcumber.q_
library packaged in ax-libraries
.This is useful for both q developers and for automated q builds. Test results can be printed to the console,or written to serialized q, JSON, or JUnit XML formats. See the Libraries section for moreinformation.
$ q $AXLIBRARIES_HOME/ws/qcumber.q_ -helpqCumber: q test runnerUsage: q qcumber.q_ <flags>Flags: -help - display this help -src <dir> - path to a q script to load before running tests -test <dir/file> - path to a quke file or directory of quke files to test [-out] <dir> - path to a file to store results files supported: .json, .xml (junit), .dat (q: get `:file.dat) [-insert] <key=val> - add columns to the output: col1=val1,col2=val2,... [-times] <number> - number of times to run each property block [-color] - colorize the output [-showAll] - print passed and skipped tests as well as failures to stdout [-quiet] - do not print any results to stdout [-reporter] <file.q> - custom report from a q file defining a function 'write[file; results]' [-errorsAsFailures] - convert all errors to failures in final output useful for systems which only look at 'failure' [-exitFailuresWith] <number> - exit code to use when script terminates and there are failing tests defaults to '1' [-breakOnErrors] - propagate errors in unit tests to facilitate debugging [-version] - print the library version
API
The qcumber.q_
library, if loaded into a q process from another script or from a running process provides.qu.runTestFile
and .qu.runTestFolder
functions, returning a table of test results.
IDE
Tests can also be executed within the IDE by right-clicking on an artifact in the artifact tree, and clickingCode > Run Tests.
Using Code > Run Tests on:
- a repository or module will run all the
.quke
files contained - a
.quke
file will run that file - functions or data will run associated test files
Tests can be associated with functions, data or modules (See Conventions, namespaces, and contexts
below).Multiple files, modules, and repositories may be tested simultaneously by multi-selectingartifacts. It is also possible to test an individual file by right clicking an open editor window and clickingTest.
Output
The output for qCumber tests reports the tests that failed, as well as any malformed test filesand any skipped tests.
If running from the command line, test results can be exported to a number of on-disk formats, or printed to theconsole, as shown below:
If running within the Developer IDE, the failed tests, malformed files, and skipped tests are nested in the output as expandable lists that can be opened by clicking the arrow.
The failed tests section will display the test blocks that failed, the file name and line numberwhere the failed test occurred, and, depending on the type of test, details as to the reason thatthe test failed. These could be failed test conditions or errors in the q code. Clicking the linenumber where the error occurred will open the failed test file directly to the test that failed.
The malformed files section displays a list of formatting errors as well as the file name andline number where the error occurred. These can also be clicked to open the location of the error.
Skipped tests show only the identity and location of the skipped tests (see below).
Test setup and teardown
qCumber provides several optional blocks which can be used to initialize state prior to running tests, and to clean up the environment after tests have run. Each are defined under a feature
.
before
will execute arbitrary code before running the firstshould
,bench
, orproperty
block for afeature
after
will execute arbitrary code after running the firstshould
,bench
, orproperty
block for afeature
before each
will execute arbitrary code before running eachshould
,bench
, orproperty
block for afeature
after each
will execute arbitrary code after running eachshould
,bench
, orproperty
block for afeature
As with expect
blocks, assignments made in a before
or before each
block will not impact other blocks. Thus,to set state visible to the expect
blocks, globally assign to identifiers with ::
, as the examples below demonstrate.
As an example, the below test sets initial state and deletes the state after running the should
checks.
feature demoBeforeAndAfter before randomNumbers :: 50?50i; after delete randomNumbers from `. should use data created in our before block expect its count to be 50 50 ~ count randomNumbers expect it to be integers `ints ~ typeOf randomNumbers
The example below runs the before each
block before each should:
feature demoBeforeEach before each $[`randomList in key system "d"; randomList ,: 1?`8; randomList :: 1?`8]; after each delete randomList from (system "d") should showcase that our list is growing expect the count to be 1 1 ~ count randomList should showcase that our list is still growing expect the count to be 2 2 ~ count randomList should showcase that our list is still growing again expect the count to be 3 3 ~ count randomList
Comparing values
To show the expected and actual values in the test output, the .qu.compare
function can be used.The function accepts two arguments, the expected result and the actual result. If these two results match,then the function returns boolean true (1b
), and the test passes.If the results do not match, then a dictionary will be passed to the expect
block containing the expectedand actual results.
feature flip should test flip operator expect flip to produce a table l : til 6; .qu.compare[ ([] a: 0 1 2 3 4 5; b: 5 4 3 2 1 0); flip `a`b!(l; reverse l)]
Skipping tests
qCumber supports ignoring tests by prepending an x
to the front of a given test block. qCumbersupports skipping feature
, should
, expect
, bench
, and property
blocks.
For example, in the following test the first expect
block is skipped. In the Developer IDE,the skipped section is grayed out.
feature count should return the count of its input xexpect to skip this test 1 ~ count `a expect correct count of list of length 2 2 ~ count `a`b
The output from this test would report the skipped test. To view the skipped tests in the output,click the arrow to the right of the Skipped Tests header.
Skipping tests conditionally
A skip if
blocks allow a feature to be skipped if a condition is true.If the skip if
block returns boolean false (0b
), then the feature
is executed.If a non-boolean value is returned or if an error is thrown the feature
aborts.
feature loadFile skip if not .z.o ~ `w64 ...feature loadFile skip if not .z.o like "m*" ...
Benchmark blocks
qCumber provides a facility for benchmarking and comparing code run time. A simple example isthe following:
feature benchmark 1 bench baseline til 100000000 behaviour til 1
Benchmark blocks appear at the same level as should
blocks. The simple case above states that the code til 1
should be at least as fast as til 100000000
.
Setup and teardown of a benchmark is allowed as well, as in:
feature benchmark 2 bench setup .ab.num : 1 baseline til 100000000 behaviour til .ab.num teardown delete num from `.ab
For more information about benchmark blocks, refer to qu
in the Function Reference available from theHelp menu.