Skip to content

Running tests

What a test is, and what a test may write, is in the specification’s Tests chapter (spec §17). This page is what nomi test does with them.

nomi test accepts an optional file or directory path. A file path runs tests in that .nomi file regardless of its name; a directory path searches recursively for every *_test.nomi file plus ordinary .nomi files that contain test declarations.

nomi test <file> --line N runs the test whose declaration starts on line N (or N + 1): a test, an attached //! test (any of its //! lines), or a group, whose header line runs every test in the group. In a group’s subtests block (spec §17 Subtests in a group), the block’s line runs the whole group too, and a test’s line runs that test once for every row, each row a test of its own in the report, todo api [sqlite] / creates a todo. When nothing starts there, nomi test runs nothing and exits non-zero with no test at <file>:<N> and the nearest test’s first line; with --format json that error is a "type":"file" failure record.

nomi test --run <regexp> runs only the tests whose names the regular expression matches (Go’s RE2 syntax, as go test -run takes it). The name is the one the report prints after <file> :: : parses a date, dates / parses a date for a test in a group, double //! test line 4 for an attached test, and todo api [sqlite] / creates a todo for a test in a group’s subtests block. The pattern is matched anywhere in the name, so --run 'creates' runs that test for every row, --run '\[sqlite\]' runs the sqlite row’s tests, and --run '^dates / ' runs one group. A row whose tests all miss is never booted. The rows of a subtests statement in a test’s body are computed as the test runs, so they are not selected on their own: a test that matches runs all its rows. The / between a group and its test is part of the name, with its spaces, and a pattern does not split on it.

--run and --line both narrow: --line 12 --run sqlite runs the test on line 12 for the sqlite row only. A file in which the pattern selects no test is not loaded. The count, the summary line and the exit status cover only the tests that ran. A pattern that selects nothing is an error, no test matches --run "<regexp>", not an empty pass, and a "type":"file" failure record with --format json.

nomi test --watch [path], with any of the other flags, runs the tests, then runs them again whenever a .nomi file under the path, or a file the tests import, changes, as nomi run --watch does (--watch): tests still running are stopped, <file> changed; running again is printed on standard error, and a file that fails to compile is reported as nomi test reports it. A new test file under the path is picked up by the next run.

Each test may run for 10 seconds of wall-clock time, counting its group’s clock, boot and setup with its body; a group with clock Clock.Virtual is held to real time too. --timeout <duration> changes the limit, written as 30s, 2m or 500ms, and --timeout 0 removes it. A test still running when its limit passes is stopped, tasks it started included, and fails; the tests after it run as usual. Its failure is at the line it was running in its own file, outside the standard library, and lists where it was running, innermost call first, each as <path>:<line> in <function> (a test body is named by its test):

FAIL coins_test.nomi :: flips forever
  line 7: did not finish within 10s, so it was stopped
  it was running:
    coins_test.nomi:7 in flips
    coins_test.nomi:12 in flips forever
  help: `nomi test --timeout <duration>` sets how long a case may run (10s unless given), and `--timeout 0` removes the limit

A subtests table is one test, so its rows share the limit. nomi run has no limit, and neither has a test run under the debugger (nomi dap), where a paused test must not time out. Work inside a single call into Go (a Go binding, say) cannot be stopped until the call returns.

What a test writes while it runs (io.print, io.write, io.inspect and dbg) is reported under the test, not where it was written. Each test’s output is collected while it runs: from its group’s boot and setup, its body, the tasks it starts, and a once initializer it is the first to force. The text report prints it under the test’s ok, FAIL or BLOCKED line, indented two spaces, for passing tests too; a failing test’s output comes before its failure block, and a test stopped early keeps what it wrote before stopping. dbg lines keep their colour when the report is coloured. Tests run one at a time, so outputs never interleave. Output inside an io.capture belongs to the capture, as anywhere, except dbg, which still goes to the test. Output a task writes after its test has finished goes to stdout in text mode and to stderr in json mode. In a test case outside any io.capture, stderr is part of the case’s output, which the report shows under the case.

ok a_test.nomi :: passes
  dbg line 4: 5 = 5
  printed
FAIL a_test.nomi :: fails
  dbg line 10: 6 = 6
  line 11: assertion failed
    assert False

When assert a == b fails, both operands are Strings and at least one holds a line break, the report shows a line diff in place of their two rows. The left operand is the actual value and the right the expected, so the value under test goes on the left. Each row is a marker, │, then the line as it is: - marks a line only the expected side has, + a line only the actual side has, and a blank marker a line both have. When the expected side is a """ literal in the file, each of its lines carries its line number in the file; a line only the actual side has carries no number. The assertion line shows that literal collapsed, """ lines 8-10 """, or """ ... """ when its lines are not numbered (an interpolated value with a line break changed their count). A changed empty line reads (empty line). More than three unchanged lines away from a change collapse to ... N lines match .... A changed line that ends in whitespace ends in $ (a trailing tab reads \t), a carriage return reads \r, and when only one side ends in a newline the other’s last line is followed by \ no newline at end. A report with a line diff has no values: row for a String equal to either side, line for line, with trailing whitespace and final newlines set aside; when no row is left, the values: heading goes too.

  line 7: assertion failed
    assert output == """ lines 8-10 """
    expected """ ... """ (-) and actual output (+) differ

         8   │ balance?
         9 - │ Your balance: 500
        10 - │ bye
             │ \ no newline at end
           + │ Your balance is 500
           + │ bye  $
             │ ($ ends a line that ends in whitespace)

A failed refute a == b or assert a != b means the two were equal, so it has no diff. An Assertable whose AssertionDetails sets actual and expected to different values gets a line diff of the two in place of its actual: row.

nomi test --format json reports as JSON Lines on stdout instead of text (--format text is the default). Each test produces one record, and a summary record comes last:

{"type":"test","file":"/abs/path/a_test.nomi","line":14,"end_line":17,"name":"a_test.nomi :: fails","status":"failed","message":"  line 16: assertion failed\n    assert x == 4","error_line":16,"output":"dbg line 15: x = 3\n","report":"FAIL a_test.nomi :: fails\n  dbg line 15: x = 3\n  line 16: assertion failed\n    assert x == 4\n"}
{"type":"file","file":"/abs/path/b_test.nomi","name":"b_test.nomi","status":"failed","message":"...","report":"..."}
{"type":"test","file":"/abs/path/a_test.nomi","line":20,"end_line":22,"name":"a_test.nomi :: counts","status":"blocked","message":"a_test.nomi:9:20: this call to `skip_odd` is not supported yet, so `fn evens` cannot run","error_line":9}
{"type":"summary","passed":3,"failed":2,"blocked":1}
  • file is absolute; line is the declaration’s first line, the one --line matches, and end_line its last. Tools should locate a result by file and line: name is the label the text report prints, and an attached test’s label embeds display wording and line numbers. A test in a group’s subtests block has one record per row, all at the test’s line, each name carrying its row (todo api [sqlite] / creates a todo).
  • status is passed, failed or blocked. blocked marks a test the VM could not run (see When code cannot run yet); its message is what the text report’s BLOCKED lines give: for code the compiler could not lower, the error nomi check reports, as <path>:<line>:<col>: <message> with a <path>:<line>:<col>: help: ... line per hint, the path relative to the working directory when it is under it. Its error_line is that code’s line when it is in the test’s own file. The summary’s blocked counts those tests.
  • message is the block the text report prints under a failing test, without colour; it is empty for a passing test. error_line is the line the failure points at and is omitted when there is none.
  • output is what the test wrote while it ran, without colour, and is omitted when it wrote nothing.
  • report is what the text report prints for that record, trailing newline included: ok <name> for a passed test, the FAIL line and its block for a failure, the BLOCKED lines for a blocked test, each with the test’s output indented under its line. It is coloured and its source lines highlighted when text mode would colour stdout, which NOMI_COLOR=always forces even though stdout is a pipe; otherwise it is plain. message never carries colour. An editor shows report in an output panel and uses message for diagnostics.
  • A "type":"file" record is a failure that stopped a whole file from running, such as a type error: the text report’s FAIL <file>: <error> line. It has no line.
  • A test that runs subtests counts one case per row it ran (spec §17 Subtests). Its record carries its cases count, and a failed one its failed_cases; the summary counts cases.
  • A test that cannot run counts one blocked test, whatever its rows.

In json mode stdout carries only records: a test’s output is in its record. The exit status is the same as in text mode: nonzero when a test file fails to load, a case fails, or a case is blocked.

--line, an editor’s test lens and the debugger treat a test that runs subtests as one test. A breakpoint in a subtests body stops once per row, in a frame of its own.

nomi test sets the environment variable NOMI_ENV to test for the run, unless the environment already sets it. Nothing else reads it: a program the tests start reads it from startup.env as it reads any variable. A boot that a test group’s boot line calls gets the Startup that line builds, not the real environment, so a test passes the variable itself: boot boot(Startup{env: {"NOMI_ENV" => "test"}}) (App Fields, Defer & Context). nomi run does not set it.

Under nomi test a test file’s working directory is its module root: the nearest directory at or above the file holding a nomi.toml, or the file’s own directory when there is none. So a test that reads its module’s files by a relative path (fs.read_text("data/rates.csv")) reads the same files wherever nomi test runs.