Internals: archives

What runtime:archive does below the API: what it refuses, what it holds in memory, and how a file source is read without buffering the archive.

The codecs

Tar and zip are read and written by the tar and zip crates. Both are safe Rust, and the zip build enables only the deflate decoder, not the encoders a program could reach. Gzip is flate2, which the runtime already carried for CompressionStream.

The layers above the codecs are ours: the format sniff, path validation, layout checks, the decompression budget, and the deterministic writers. That is where an archive stops being bytes and becomes a set of names the filesystem may receive.

Format detection

The format is read from the leading bytes: gzip's 1f 8b, zip's local, empty or spanned signature, and everything else is tar. A name or extension is never consulted. An input that claims to be a zip but starts like a tar is handled as a tar, and if it is not one it fails as a tar.

Validation before use

Every entry is validated as it is read, and the eager APIs finish a full layout check before returning. Each path must be:

relative, slash-separated, valid UTF-8;
no empty component, no . and no ..;
no backslash and no drive letter;
free of duplicates and file/directory collisions, tracked as entries arrive;
on Windows and macOS, not differing only by case from an earlier path (Unicode lowercase, not just ASCII), because the filesystem would merge entries the archive treats as distinct;
on Windows, not a device name (NUL, COM1.txt), not containing : or a character Windows forbids, and not ending in a dot or space. A device name opens the device instead of a file under the destination, : names an alternate data stream of another file, and Win32 drops a trailing dot or space so a. would land on a. Go's filepath.IsLocal refuses the same names.

Tar links and special types, zip symlinks, encrypted zip entries, and zip compression other than Stored and Deflated are refused. A directory entry that carries data is malformed and refused.

Refusal is the design: an archive is never rewritten into a safe-looking one. Dropping a .. component or skipping a link would silently change what the caller unpacks, and code that relies on the difference would be wrong in a way nothing reports.

Bounds

Two limits apply, maxEntries and maxBytes (the uncompressed total). They are checked twice:

from the declared sizes in tar headers and the zip central directory, before any entry data is allocated — and, for zip, from the end record before the central directory is parsed, because that parse allocates its index;
from the bytes a decompressor actually yields, so a header that understates its size cannot walk past the budget.

A gzip stream has no declared total to trust, so its decompression is bounded by maxBytes plus a per-entry allowance for tar headers. Reaching that bound is an error, not an early end of input: an early end could stop the tar reader cleanly at a block boundary and list a truncated archive as if it were whole. Over either limit is a RangeError with ERR_ARCHIVE_TOO_LARGE.

The tar reader stops at its end-of-archive marker, but gzip's CRC-32 and length are in a trailer after it, and are checked only when the decoder reaches the end of its input. Every pass that reads the whole archive — listing, the async iterator, files(), an entry's bytes() or stream(), and extract() — therefore reads on to the end, so a corrupted or truncated trailer is ERR_ARCHIVE_DECODE_FAILED rather than accepted. Concatenated gzip members are one stream, as gzip reads them; bytes after the last member are refused.

Listing and single-entry reads do not materialize entries they do not return: zip metadata comes from the central directory and tar headers are read in order. entries() therefore costs a pass over headers, not a pass over contents, and the async iterator yields those headers one at a time instead of collecting an array. A tar.gz entry is reached by decompressing through its predecessors, but those bytes are discarded as they are skipped rather than retained. Its stream() then decompresses the rest of the archive before it closes, which is what checks the trailer above; a stream cancelled early skips that work.

File sources are streamed

file() is opened through the jailed SyncFileSystem, which can seek and read on demand. The fd is opened once and shared by every reader; each reader keeps its own position and seeks and reads under a lock. The handle remains attached to the same file if its path is renamed or removed, but it is not an immutable snapshot if another process writes to that file.

The source is never read into a single buffer: for await yields metadata one entry at a time, files(filter) reads only the entries the filter selected, ArchiveEntry.stream() pulls decompressed chunks through a bounded channel. Extraction writes chunks into a temporary directory, so memory is bounded by the channel's few chunks and the paths needed for collision checks, not by the archive's size. Staging uses temporary disk space up to the uncompressed output size. A zip parser still builds its central-directory index, which maxEntries bounds.

A provider with no synchronous filesystem falls back to reading the file once through FileSystem; that is the best its whole-file seam can do. bytes() also allocates the whole archive, because returning it is the operation.

Why extraction stages before it writes

extract() writes through a pinned tree (D180): the destination directory is admitted by the jail, opened by descriptor without following a link at its name, and every path beneath it is opened from that descriptor with O_NOFOLLOW. A link anywhere beneath the destination — there before, or swapped in during extraction — is refused rather than written through. That is a stronger promise than the jail's: the jail keeps a path inside the project, and a link that stays inside the project is allowed by it.

It first scans metadata and checks each destination path for an existing link, so a refused archive writes nothing. It then decodes the archive once into a temporary directory beneath the destination, checking entry paths, limits, data lengths, zip CRCs and the gzip trailer while it streams. It checks the paths again from the entries actually decoded, then renames each staged file into place. A rename within one directory tree moves a name, not bytes, so the commit does not write the archive a second time, and a file already at a path is replaced atomically. The metadata scan and extraction use the same open file handle, but in-place changes made during extraction are visible to that read.

If decoding or validation fails, nothing is moved into the destination. The temporary directory costs disk space proportional to uncompressed output. An I/O error while moving staged files into place can still leave partial output; the staging guarantee covers archive decoding and validation, not failures during the final renames. On Windows the tree checks each component and then uses it by path, so the window between the two stays open there, as it does for the jail.

Determinism

A build sorts entries by path, writes 0:0 ownership, 0644 files and 0755 directories, and uses timestamp 0 unless an entry supplied lastModified. Zip's default options depend on whether its time feature is enabled in the dependency graph — self_update enables it — so the writer sets the timestamp and the system explicitly rather than inheriting a default that could change between builds. The same entries therefore produce the same bytes, which is what lets an archive be hashed or cached.

Gzip's header mtime is likewise not the clock: flate2 writes a zero mtime.

Where the work runs

Metadata scans run on the embedder's TaskSpawner (tokio's blocking pool in esrun and esdev), inline when there is none. File reads and gzip decompression run on a dedicated archive worker thread; the writer side awaits the bounded channel, and extraction writes through the FileSystem provider one chunk at a time.

Not here

7z, rar, brotli and zstd; passwords; selective extraction (read an entry and write it yourself); preserving ownership, modes or arbitrary timestamps from an input archive; and appending.