Running and isolating tests

test.only runs that case and skips the rest; describe.only does it for a group. What did not run is counted and said out loud:

  only: 27 other tests did not run
  1 passed, 0 failed, 27 skipped

That line is the point. A .only left in a commit otherwise looks exactly like a suite that got faster, and a skipped case missing from the report is the same failure as a case that quietly never ran.

Filtering by name

-t (--test-name-pattern) runs only the tests whose full name matches a regular expression. The full name includes the describe groups, joined with >. --test-skip-pattern leaves out the tests that match:

esdev test -t=auth                        # every test under "auth", in every file
esdev test "-t=auth > logs (in|out)"
esdev test -t=auth --test-skip-pattern=slow

The value is attached with =, as for every esdev flag. Tests left out are counted as skipped and named on a line of their own, like .only:

  filter: 12 other tests did not match
  3 passed, 0 failed, 12 skipped

A group with no selected tests does not run its beforeAll. Path filters (esdev test auth) choose files; -t chooses tests within them, and the two combine. To choose tests by category rather than name, see Tags and --tags-filter.

Listing tests

--list names the tests each file registers, by full name, and runs none of them — no test and no hook. It combines with path filters and -t:

esdev test --list
esdev test --list -t=auth

A file's own top-level code does run, since that is where its tests are registered.

Stopping early

--bail stops after the first failed test; --bail=5 after five. Failures add up across files. The tests left in a file are counted as skipped, and files that were not started are named on a line of their own:

  bail: 2 tests did not run after the failure limit
  0 passed, 1 failed, 2 skipped

bail: 3 files did not run after the failure limit

Files already running when the limit is reached finish on their own count, which is why --jobs=1 stops the soonest.

Random order

--randomize runs files, and tests within their groups, in a shuffled order. A test that passes only after another one ran shows up as a failure in some orders. The run prints its seed; --seed runs that order again:

esdev test --randomize
# randomize: seed=1327238386 (run this order again with --seed=1327238386)
esdev test --seed=1327238386

Tests are shuffled among their siblings inside each describe, and groups among theirs, so a group's beforeAll and afterAll still wrap exactly its own tests.

Only what changed

--changed runs the test files that reach a file git says changed, through their imports:

esdev test --changed               # uncommitted changes, and untracked files
esdev test --changed=origin/main   # everything since this branch left main

--related does the same for files you name, so a pre-commit hook can pass the staged files:

esdev test --related src/cart.ts src/price.ts

A test file reaches a file when it imports it, or imports something that does, at any depth. Static imports, re-exports and import() of a string count, each resolved as a run resolves it — through the project's aliases too. Type-only imports do not, since they are gone at run time. A change to esdev.json, package.json, a tsconfig.*.json or jsconfig.*.json, a lockfile, or a setup module runs every test file. A change to a configured plugin, or any module it imports, also runs every test file, since the plugin can change how test imports resolve and compile.

The run says what it chose on stderr:

changed: 3 files since origin/main; 5 of 41 test files reach them

When nothing is reached, nothing runs and the run passes.

Splitting across machines

--shard=<index>/<count> runs one part of the test files, so a suite can be split across CI jobs. --shard=1/3 is the first of three:

# GitHub Actions
strategy:
  matrix:
    shard: [1, 2, 3]
steps:
  - run: esdev test --shard=${{ matrix.shard }}/3 --reporter=junit --reporter-outfile=junit-${{ matrix.shard }}.xml

Every file runs in exactly one shard, and the shards differ in size by at most one file. Files are assigned by a hash of their path, as Jest and Vitest do, so a shard mixes files from across the project. Sharding splits files, not the tests inside them. The run names its shard on stderr:

shard 1/3: 14 of 41 files

A shard with no files, because there are more shards than files, passes and still writes its report. --shard cannot be combined with --file or --watch.

Concurrent tests

Tests in a file run one at a time by default. Tests that only wait, on the network or a timer, can run alongside each other instead:

describe.concurrent("fetches", () => {
  test("users", async () => { … });
  test("orders", async () => { … });
  test("invoices", async () => { … });
});

test.concurrent("also alongside", async () => { … });

Concurrent tests next to each other run together, at most five at once. --max-concurrency=<n>, or maxConcurrency under test in esdev.json, sets the limit. test.sequential, describe.sequential and { concurrent: false } opt out.

Each test keeps its own beforeEach, fixtures and afterEach, and a group's beforeAll runs once before any of them. The global expect knows which test it is in, even after an await, so assertion counts and snapshots stay with the right test. Tests that share state, such as a module variable or a database table, should stay sequential. Browser runs run concurrent tests one at a time.

Flaky tests

--repeats=<n> runs every test n more times after the first; a test that fails on any run fails, and the report names the run. A single test can ask for the same with { repeats: n }, which takes precedence over the flag:

  FAIL uploads the file
    failed on run 7 of 21

One process per file

A test suite is where isolation matters most. A file that wedges, exhausts its heap or calls exit() must not decide the fate of the others, and a global left behind by one must not be visible to the next. Each file gets its own process — the prelude snapshot makes that cheap.

--file=<path> is what a child is invoked with, and is equally a supported way to run one file by hand.

Files run in parallel — the machine's parallelism, at most 8, since every job holds a V8 heap. Each file's output is held and printed whole when it finishes, so two suites never interleave line by line.

esdev test                  # all of them, in parallel
esdev test db --watch       # the files whose path contains "db", again on every save
esdev test --jobs=1         # one at a time, writing straight to the terminal

--jobs=1 is the run to reach for when a test is hanging and you want to watch it happen. --watch re-discovers files every pass, so a test you are about to write is one the watcher will find.

Watch mode

esdev test --watch runs the tests again whenever a source file changes. On a terminal, keys change what it runs:

p and t ask for the filter and show what it would select as you type. Enter applies it, an empty filter clears it, and Escape leaves it as it was. The filters in force are shown above the line the watch waits under, and a change to a file reruns with them.

A key that starts a run, pressed while one is running, cancels that run and starts its own; q quits, ending the files being run. With --isolation=none the run is not cancelled, and the key waits for it to end.

The keys need a terminal on stdin, and are read on Linux and macOS. Elsewhere, and when stdin is not a terminal, the watch reruns on changes and ^C stops it.

A change to esdev.json (or the --config file) restarts the watch and reruns global setup. Changes to resolve.alias, jsx or test settings apply on the next run. Changes to tsconfig.json also apply on the next run, without restarting the watcher.

Sharing one runtime

For a suite whose startup and repeated imports cost more than its tests, opt out of the process boundary:

esdev test --isolation=none

All selected files run serially in one process. They share a V8 heap, globals and ES-module cache, so shared dependencies and --setup modules load once.

This can speed up a suite, but files are no longer isolated: globals and mocks can leak between them, and exit() or a crash ends the whole run. A hanging file cannot be stopped independently. --jobs, --timeout and per-file JSON reports are unavailable with this mode.

The default, --isolation=process, retains the process-per-file behavior.

Rehearse the production grant

By default each file runs with every capability, the way esdev runs anything. That is the loop you develop in — and it is wider than what production grants, so a path the suite covers can pass here and die there with NotAllowedError.

The permission flags narrow each file's run to a production-shaped grant, so a covered path meets its deployment's capabilities before deployment:

esdev test --deny-all --allow-read=./data --allow-env=DATABASE_URL

--deny-all takes everything; --allow-* grants back what the suite needs — the same flags, resolved the same way, as a run. --deny-<name> subtracts a single capability instead: esdev test --deny-net fails a suite the moment anything in it touches the network. A combination that cannot exist (--allow-env=X without --deny-all) is refused before anything runs, not once per file.

Grant what the harness needs too, not just the program: snapshot writes need --allow-write, helper imports --allow-imports. The flags travel to every child, and --isolation=none resolves them for its single process. They are flags only, never esdev.json keys — a rehearsal decides a single run. Run with none of them and nothing changes: full grant, full fidelity, exactly as before.

A grant per file

A test file can declare the grant its module is deployed with, in a /** … */ comment:

/**
 * @permissions --allow-imports --allow-read=./config --allow-env=PORT
 */
import { expect, test } from "runtime:test";
import { loadConfig } from "./config.ts";

The flags are esrun's, and they start from what esrun starts from: nothing. Copy them from the command line the module is deployed with, and the file's tests run under exactly that grant. Paths are relative to the project root, as they are when esrun runs there. Several @permissions lines add up.

A file's own grant is the one it runs under. The command-line flags apply to the files that declare none, so esdev test --deny-net narrows the rest of the suite without widening or narrowing a file that states its grant.

A flag that is not a permission flag, or a combination esrun would refuse, fails that file with the reason. --isolation=none is refused when any file declares a grant, since one process has one grant, and so is a browser run of such a file, since a page has no grant to rehearse.

The file is the entry, unaltered

It keeps its own path, its module resolution and its relative imports, and is compiled as any module the project runs is: types erased, JSX and the project's plugins applied. Nothing wraps it, and line numbers are kept, so a failing assertion names the line you wrote. A --setup module is imported on the file's first line, without moving it.

A test file is an ordinary module, so running one directly works and reports the same way:

esdev app.test.ts

An unhandled rejection

Fails the test it surfaced in. A test that expects one claims it:

addEventListener("unhandledrejection", (event) => event.preventDefault());

A listener that does not call preventDefault() leaves it unhandled, as in a browser.

A test that never finishes

It is reported as a failure, not a hang:

  FAIL never finishes
    the test never finished — it is waiting on something that never happened
  3 passed, 1 failed

The cases queued behind it are reported as never started, so the report points at the one that is stuck.

A file that does not end

A file whose tests all passed can still not finish. Its process ends when nothing is left to run, so a timer, a server or a socket its tests left open keeps it waiting until --timeout stops it. --detect-async-leaks names what was left instead:

timers.test.ts
  FAIL async leak: a setInterval
    a setInterval was still pending after the file's tests finished
        at poll (src/poller.ts:4:3)
        at timers.test.ts:7:3

Each leak fails the file and points at the line that started it, and the file then ends instead of waiting. A server or socket started inside the runtime has no line of your code on its stack; the report says so. Close it in afterAll. A timer meant to run in the background, such as a heartbeat, is not a leak once it is let go with unrefTimer. It is not available in browser runs, or with --isolation=none, where the files share one event loop.