Internals: Durable workers

How runtime:workers behaves underneath: what "durable" is actually promising, where the state lives, who is allowed to open it, and what each decision costs.

For signatures see the API reference. The reasoning and the rejected alternatives are DECISIONS D80.

The gate is the design

A durable worker's method can write and return. The write is committed to SQLite asynchronously; the return value is not handed to the caller until that commit has happened.

async add(item) {
  const items = this.state.get("items") ?? [];
  items.push(item);
  this.state.set("items", items);   // not awaited
  return items.length;              // the caller waits for the commit anyway
}

That ordering is the whole promise. A process that dies mid-call is a call that never returned — not one that returned something the disk never heard about. It is also what makes coalescing safe: several sets in one call become one transaction, because nothing has left the process yet to be contradicted.

What the gate does not cover is a side effect issued in the middle of a call — a fetch, a message on a port, a row written to another database. Those leave before the method returns, so the gate has not run yet.

await this.state.sync();        // …then the effect
await fetch(webhook, { method: "POST", body });

This is the same shape Cloudflare's output gate has, and the same caveat.

Measured. A test kills a real esrun with SIGKILL after five acknowledged appends and finds five on restart. It is in crates/runtime-cli/tests/durable_workers.rs, and it fails if the gate is removed.

Why the state is resident, and therefore capped

state.get(k) is synchronous — a Map lookup, not an await. That is possible because the whole key/value page is read when the worker is opened and kept in its heap.

Resident state has a memory cost that somebody has to bound, so it is bounded out loud: 1 MiB per worker, 128 KiB per value, refused at the set with ERR_DURABLE_STATE_TOO_LARGE rather than nudged with a warning. A soft cap on a resident cache is an unbounded cache with a comment.

The consequence is a boundary you can state: durable-worker state is the small, hot thing a request needs immediately — a cart, a session, a counter, a cursor. Anything that accumulates belongs in runtime:db, which is in the same runtime and needs the same grants.

The value format

Values are stored as structured-clone bytes — the same serialization postMessage and structuredClone already use — so a Date comes back a Date, a Map a Map, a BigInt a BigInt, and a cyclic object cyclic.

Those MessagePack rows are measured, not assumed, which is why the choice was made once rather than deferred: changing the format later means rewriting every worker's file, and a data migration is a far worse thing to owe than a decision.

A codec tag is stored beside every value. State written by a newer runtime, in a format this build does not know, is refused by name (ERR_DURABLE_STATE_FORMAT) rather than handed to a deserializer that will misread it.

A set that stores what is already stored writes nothing: the encoded bytes are compared with what is on disk first. That comparison is worth its cost because the storing is the expensive half — a commit that changes a page costs milliseconds against one that changes none — and "read it, put it back" is what a handler written over resident state does all day.

One process owns a directory

Each worker's state is its own SQLite database, under <dir>/<class>/<xx>/<hash>.db, with a _registry.db beside them holding the catalog of what exists.

The file name is a hash of the id, never the id. An id is a string a program chose: it may hold slashes, it may be 400 characters, and on macOS and Windows Cart and cart are the same file. The id itself is stored inside the file and checked when it is opened, so a hash collision is an error rather than two workers quietly sharing a state.

A directory belongs to one process, and nothing in this module arranges that. The embedded engine takes an exclusive lock on a database file for as long as it is open, and the operating system drops it when the process ends. So the guarantee needs no heartbeat, cannot be lost while it is held, and leaves nothing stale behind when a process is killed — the three ways an advisory lock file goes wrong. What the module adds is the sentence: an engine Locking error about a file the caller never named becomes ERR_DURABLE_LOCKED, naming the directory.

The cost is stated rather than hidden: two processes cannot share a durable directory at all, not even to read. A second one is refused until the first exits — which is the correct behaviour for an overlapping redeploy, and a real constraint for anything that wanted two readers.

One connection is one conversation

Each worker's database has exactly one writer — its own flush — but the catalog is shared by every worker in the process, and twelve materializing at once would put twelve statements on one connection. The embedded engine does not refuse that; it panics from its WAL, which panic containment turns into a JavaScript exception a caller cannot act on. So every catalog statement goes through one queue. It is a small thing that took a flaky test to find, and it is an argument for the file-per-worker layout that was not the reason for it.

The mailbox

Calls to one worker queue and run one at a time, in the order they were made. There is no lock to take, because there is nothing to take it against: the state has exactly one reader and one writer, and they are the same call.

The queue is bounded (mailbox, 1024 by default). Past it a call is refused with ERR_DURABLE_BUSY rather than waited on — a queue that grows without limit is a failure that has been hidden rather than reported.

Two workers of the same class are two mailboxes. Work on one is not work on the other, which is the reason to address state by worker rather than by row.

A cycle deadlocks. A worker calling back into one that is calling it waits forever, because a mailbox is strictly one at a time and nothing yet detects the cycle. It is written down here rather than discovered in production.

Eviction has no timer

A worker that has been idle past evictAfter is closed: its writes are flushed, stop("idle") runs, and its database handle is released. It is opened again, with its state, the next time it is addressed.

That sweep runs when work arrives, never on a timer. This runtime's timers keep the process alive, so a repeating sweep would be a reason a program could never exit — a script that used one durable worker would sit there for the whole idle window with nothing to do. The cost is that a worker alone in a quiet process stays open until something else happens, which costs one file handle.

Collections: a blob the database cannot read, beside columns it can

A collection is a table per name: id, the document as structured-clone bytes, and one real column for each field the class declared, with an index on it.

That is the whole trade, and it is deliberate. The blob is what keeps a Date a Date — the property the keys have and JSON does not — and it is exactly what makes the document opaque to SQL. So what you want to query, you declare, and it is copied out into a column where the database can order it. What you do not declare is still stored, still returned, and simply not indexed.

{ scan: true } is the escape hatch and says what it costs: the rows are read, the documents decoded, and the filter applied here. On a small collection that is honest work; on a large one it is a full read, which is why it cannot happen by accident.

Declaring a field later

The column is added on the first wake after the deploy — and then filled in from the documents already stored. Without that backfill the column would be null for everything written before the field existed, and a query over it would silently return a subset. That read-and-update pass is the cost of declaring a field late, and it is paid once, on the worker's first wake, where it is visible as a slow first call rather than as wrong answers later.

Nothing is ever dropped. A field or a collection removed from the schema keeps its column and its table: a class that stopped asking for something is not the same as data nobody wants, and a schema edit that deletes rows is one nobody can undo.

One connection, two chains

A worker's connection has two users that know nothing about each other: the flush behind a set, and whatever a collection is doing. Both go through one serial chain, for the reason the catalog does — the engine panics rather than refuses when two statements overlap.

A transaction is the exception, and it needs a second chain rather than a bypass. It holds the outer chain for its whole body, so nothing else can reach the connection; statements inside it join an inner chain instead, which keeps them one at a time without waiting for the holder. A set inside a transaction therefore stops scheduling a flush of its own and is written by the transaction, which is what makes state.transaction cover both halves of the storage.

A query is read in one turn rather than streamed while the caller iterates. A cursor would hold the connection across the loop body, and the obvious loop body — await state.set(…) per document — would then be waiting on the connection its own iteration is holding. .limit() bounds the read instead.

Alarms: an index that may be early, never late

A worker's alarm time is stored twice — in its own file, where it is the truth, and in the catalog, where it is what makes the alarm findable without opening every worker to ask. Two files means no transaction spans them, so the order of the two writes is the design:

1. the catalog, moved earlier only (MIN of what is there and the new time),

2. the worker's own file,

3. the catalog again, exactly.

A crash anywhere in that sequence therefore leaves the index early, and an early index costs one wake-up that opens a worker, finds nothing due, and puts the index right. A late one would be an alarm that never fires. Opening a worker reconciles the two in the same way, so a directory repairs itself.

The scheduler asks the catalog for the minimum future time among the classes it runs and sleeps until then — no polling loop, no fixed interval. Setting an earlier alarm wakes it immediately; alarmPoll (60s) is only a ceiling, so a clock that jumps cannot leave it asleep for ever.

Why the class list is required

startAlarms({ classes }) will not guess. A class is a JavaScript value in one process's module graph; whether this deployment is the one meant to service a given class is not something the runtime can read off it. If the scheduler had fallen back to "classes something has addressed so far", an alarm would fire on a busy process and not on an idle one — the failure mode that is hardest to see and worst to have. Rows for classes not listed are excluded by the query itself, so they are neither woken nor stepped over.

Firing

An alarm goes through the worker's mailbox, so it cannot interleave with a call, and it is cleared before the handler runs: a handler that sets the next time repeats, one that sets nothing is done. A failure puts one back — 1s, 2s, 4s, to a five-minute cap — with the attempt count stored beside the alarm, so a restart does not reset it, and the last failure is reported rather than dropped.

The process stays alive while the scheduler runs, and this is the one place this module holds a timer at all. Which is why it is not started for you: eviction deliberately has no timer so that a script exits, and an alarm scheduler that started itself would undo that for anyone who merely set a time.

What this is not

Not an actor model. The non-goal — no process model, scheduler, preemption, mailboxes or supervisors — is about the runtime, and it stands: nothing in the Rust crates gained a scheduler, nothing preempts, and no agent gained a mailbox. A durable worker is a value in guest JavaScript with a queue in front of it. The runtime still does not decide when your code runs.

The module is written in JavaScript over runtime:db, runtime:fs and runtime:hashing, and adds no capability and no Rust. That is the same rule runtime:db's driver kit set for database backends, applied to the layer above them.

Not yet

Shards. Today a worker runs on the agent that addressed it. The next phase puts it on a Worker of its own — which brings a real memory ceiling, a watchdog that can terminate() a worker stuck in a loop, and a failure domain smaller than the process. Placement is a shard problem rather than a thread-per-worker one for a reason that is arithmetic rather than taste: an isolate costs megabytes of heap and an OS thread, so a thousand live workers would be gigabytes and a thousand threads.

Durable execution — retries, a step journal, workflows — is deliberately not planned as a second subsystem. With alarms and workers in place it is a class on top of this primitive.