Benchmarks

A benchmark is a test that measures. It lives in a *.bench.* or *.benchmark.* file and gets bench from its context:

// src/parse.bench.ts
import { expect, test } from "runtime:test";
import { parse, parseFast } from "./parse.ts";

test("parseFast is faster", async ({ bench }) => {
  const results = await bench.compare(
    bench("parse", () => parse(input)),
    bench("parseFast", () => parseFast(input)),
  );
  expect(results.get("parseFast")).toBeFasterThan(results.get("parse"), { delta: 0.1 });
});

esdev bench
esdev bench parse

esdev bench takes the same flags and filters as esdev test, and runs one file at a time so that files do not slow each other down. esdev test does not run benchmark files.

The results are printed as a table:

  name           ops/sec    mean     p75     p99     rme  samples
  parse          901,228  1.11µs  1.13µs  1.44µs  ±0.75%      299  2.97× slower
  parseFast    2,675,276   374ns   377ns   546ns  ±1.00%      299       fastest

Measuring

bench(name, options?, fn) registers a benchmark. fn may be async; each call is awaited.

• .run(options?) measures one benchmark and resolves to its result.

• bench.compare(...benchmarks, options?) measures several, a sample of each in turn, and resolves to a Map of results by name.

A benchmark's own options may also have beforeEach and afterEach, which run around each sample and are not timed.

A function too fast for the clock is timed in loops of calls, and each call's time is the loop's time divided by its count. clock.freeze() does not affect measurement.

Timing part of a call

The function is called with b. When it calls b.start() and b.end(), only the code between them is timed, so each call can prepare its own input:

bench("sort", (b) => {
  const data = shuffled(10_000);
  b.start();
  data.sort();
  b.end();
});

The section is timed on its own each call, so it should take many times the clock's step: a microsecond in esdev, and as much as a millisecond in some browsers. A function must mark a section on every call or on none.

Warnings

A result's warnings lists what makes it untrustworthy, and the table prints them beneath it:

Comparing with an earlier run

writeResult writes a benchmark's result to a file each time it is measured, and bench.from(name, path) reads it back as a benchmark that is not measured again. Committing the file makes the comparison repeatable:

import { env } from "runtime:process";

test("parse has not slowed down", async ({ bench }) => {
  const parse = () => parseConfig(input);
  if (env.WRITE_BENCH) {
    await bench("parse", { writeResult: "./bench/parse.json" }, parse).run();
    return;
  }
  const results = await bench.compare(
    bench("current", parse),
    bench.from("stored", "./bench/parse.json"),
  );
  expect(results.get("current")).not.toBeSlowerThan(results.get("stored"), { delta: 0.1 });
});

WRITE_BENCH=1 esdev bench   # store the result
esdev bench                 # compare against it

Paths are relative to the project root. Files are read and written with the test's own permissions, so --deny-write stops writeResult, and a test in a browser page can use neither. bench.from also takes a function that returns a result, for one stored elsewhere.

A stored result was measured on whatever machine wrote it. Compare against it on the same machine.

Results

warnings and stored (true for a result bench.from read) complete it.

With --reporter=json, each measured result is a line of its own: { "type": "bench", "file", "test", "result" }.

expect(a).toBeFasterThan(b, { delta }) and toBeSlowerThan compare two results' throughput. delta is the margin required, 0.1 being 10%, so that noise alone cannot pass or fail the test.