Internals: serialization

What happens between a parse() call and the object you get back, why the formats disagree about what survives a round trip, and which conversions are deliberate rather than incidental.

This page explains behaviour rather than listing signatures — for those, see the runtime:serialization reference.

Everything crosses as JSON, except where it cannot

A parser running in Rust has to get its result into the isolate. The obvious route is to build the value tree directly through the FFI boundary, one allocation per node. The route actually taken is to serialize to a JSON string and let the guest's JSON.parse build the graph: that turns thousands of small crossings into one string, and hands the tree-building to a path V8 has already tuned rather than to ours.

That pivot decides most of what follows, because JSON is narrower than the formats being parsed. Two places pay for it explicitly:

YAML and TOML build values directly instead. JSON has no Infinity and no NaN, so a transcode would quietly turn .inf and .nan into null — a document that said "unbounded" arriving as "absent". Both parsers therefore construct the runtime's values themselves and non-finite floats survive as real numbers. TOML also gains from this: its datetimes become RFC 3339 strings rather than leaking the toml crate's internal round-trip sentinel.

A TOML integer past 2⁵³ is a BigInt. TOML 1.0 requires a reader to handle 64-bit integers losslessly or refuse them, and a number would round 9007199254740993 to …992 without a word. CBOR made the same choice for the same reason (D170). The writer is the mirror image: a number past the safe range is written as a float, so it comes back a number rather than turning into a BigInt on the round trip.

Every builder converts the same awkward values first. A Date, a Map and a Set reach the host as empty objects, since they carry no enumerable properties. MessagePack and CBOR already converted them; the text builders did not, so a Date became an empty TOML table and {} in YAML. All builders now share the conversion, and the text formats refuse bytes instead of writing them as null.

MessagePack cannot use the pivot at all for binary. JSON has no byte string, so the bin family — the reason to reach for a binary format — has nowhere to go. Rather than give up the fast path for every document, the encoded form is walked first by an exact structural scan that allocates nothing and reads no payload; only a document that actually carries bin or ext pays for a value tree. The scan is structural rather than a byte search on purpose: a 0xc4 byte inside a string payload is not a bin marker, and treating it as one would push every document containing arbitrary text onto the slow path.

A non-string map key takes the value tree too. The pivot spells one its own way — the float 1 as "1.0", and a nil or array key not at all (it throws) — while the tree spells it as a property access would, the rule CBOR uses. Two paths with two spellings would make a key's name depend on whether a bin sat elsewhere in the document, so the scan sends every such key down the one path that gets it right. An integer key keeps its exact wire digits (uint64 max is "18446744073709551615", not the f64 it rounds to), as a CBOR integer key past 2⁵³ does.

What each format keeps

Numbers are IEEE-754 doubles once they are in the isolate, so an integer past 2⁵³ loses precision on the way in regardless of the format that carried it. That is JavaScript's limit, not the parser's.

MessagePack

The type mapping is the part worth knowing, because two entries are conversions rather than round trips:

Map, Set and ArrayBuffer are converted because they reach the host as empty objects — they carry no own enumerable properties, so without the conversion every entry is silently dropped and {} goes out.

Unrepresentable values throw rather than encoding as nil. A function or a BigInt has no MessagePack form, and writing nil for it is total data loss in a format chosen for fidelity, reported nowhere. The refusal names the value at the call site instead.

An ext value decodes to its payload bytes. No JavaScript type corresponds to a MessagePack extension, and inventing a wrapper object would make it indistinguishable from real data the document contained.

Nesting is bounded. The input is guest-supplied and both the scan and the decoder recurse, so a document deeper than 256 levels is refused rather than exhausting the stack. The same bound applies on the way out, which is also what stops a cyclic object from encoding forever.

CBOR

CBOR (RFC 8949) is the second binary format, and it mostly shares MessagePack's type mapping: a byte string decodes to a Uint8Array, a Map/Set/ArrayBuffer is converted rather than flattened to {}, and a value with no representation throws. It decodes straight to the value tree rather than through the JSON pivot — JSON has no byte string and no Infinity/NaN, so a document using a byte string or a non-finite float would be narrowed on the way in. In two places it goes further than MessagePack.

Encoding is the deterministic form of RFC 8949 §4.2. Integers and lengths are written shortest-form — the integer forms reach their major types' full ranges, so major type 1 goes down to −2⁶⁴ rather than stopping at i64::MIN — a float uses the shortest width that preserves it (half, single, or double; NaN is the canonical f97e00), and map keys are sorted by the bytewise lexicographic order of their encodings. That last one is subtle enough to state plainly: because the length prefix sorts first, "z" encodes before "aa". The payoff is that the same value always produces the same bytes — which is what makes CBOR usable for a signature or a content address over the encoding — and the cost is that a decode/encode round trip returns an object whose keys are in canonical order, not the order they were written in. An integer-valued number in the safe range takes an integer form, so a float that happens to be whole re-encodes as an integer. Past the safe range a number stays a float: an integer that large decodes to a BigInt, so writing a number as one would change its type on the round trip. cbor-x draws the same line, writing a large number as a float and reserving 64-bit integers for BigInt (D176).

An integer past JavaScript's safe range is a BigInt. A double cannot hold an integer past 2⁵³ exactly, so decode yields a number inside that range and a BigInt outside it: no CBOR integer is silently rounded, and a BigInt re-encodes as the same integer. This is where CBOR parts from MessagePack, whose JSON pivot has no BigInt and narrows the value instead. The two integer major types span [-2^64, 2^64-1]; a BigInt beyond that would need a bignum tag, which is refused, so encode throws rather than approximating. It carries the value across the host bridge as a BigInt(i128) — the variant MessagePack's target type never needed.

A map becomes a plain object, so a non-string key becomes a property name. CBOR allows any item as a key, but a JavaScript object's keys are strings, so a non-string key is coerced exactly as a property access coerces it: a number by String(n), an array or byte string as its elements joined with commas, a nested map as [object Object]. The cost is JavaScript's own — keys that coerce alike share a name, so [1, 2] and ["1", "2"] land on the same property. A caller who needs the keys kept distinct must write them as strings.

One spelling is fixed rather than String's exact behaviour: a nested map always spells [object Object]. String(plainObject) agrees, but a nested map carrying its own toString would make String call it — and throw on a non-callable one — and a codec should not run a decoded value's accessors during a structural walk.

Tags are refused. A tag (major type 6) is a number that gives a value a semantic meaning — a date, a bignum, a decimal fraction. No JavaScript type corresponds to one, and a wrapper object would be indistinguishable from real data the document contained, so decode reports the tag number and encode never writes one. The same reasoning refuses an unassigned simple value (major type 7), for which there is likewise no JavaScript counterpart.

Decoding is permissive over well-formed input. Definite and indefinite-length strings, arrays and maps all read, and the whole input must be consumed — a trailing byte is an error, because a decoder that stopped early would hide the difference between one document and two concatenated ones. Nesting is bounded at 256 levels, as everywhere else.

XML

Elements become object keys; attributes are prefixed @; an element's own text is $text; a name that repeats becomes an array. An empty element is the empty string:

XML.parse('<r a="1">text</r>');        // { r: { "@a": "1", $text: "text" } }
XML.parse("<r><i>1</i><i>2</i></r>");  // { r: { i: [{ $text: "1" }, { $text: "2" }] } }
XML.parse("<r><e/></r>");              // { r: { e: "" } }

A document must be well formed, and validate agrees with parse. Reaching the end of the input with elements still open is an error, and so is a document with no element at all. Both used to pass: "<r>" parsed to { r: {} } and "not xml" came back as that same string, so anything at all parsed "successfully" — and validate said true, because it only surfaced errors the reader raised and a truncated document raises none. It now tracks depth itself.

Nesting is bounded at 256 levels, matching libxml2's default, for the same reason MessagePack's is.

One reader, used by everything. quick-xml is only the tokenizer; what a document means is decided in crates/runtime/src/xml.rs, and parse, validate, the streaming decoder and the fuzz target all go through it. They used to walk quick-xml's events separately, and each lost something: since quick-xml reports a reference (&) as its own event, text silently dropped every escaped character (t< parsed to t), CDATA vanished, attribute values kept their raw &, and the stream skipped a malformed record instead of failing. One walk means one set of rules — the full list of what is refused is on the reference page.

A DTD is skipped, not processed. Only the five predefined entities and character references resolve; an entity a DTD declares is an error rather than nothing. Expanding DTD entities is how a document turns a few bytes into gigabytes (the "billion laughs"), and a document with no DTD loses nothing.

Trimming stops at what the author spelled out. Indentation is not content, so whitespace written as plain text at either edge of an element's text is trimmed. CDATA and character references are deliberate, so trimming stops at them: <a><![CDATA[ x ]]></a> keeps its spaces. The writer relies on the same rule in reverse and writes edge whitespace as  , so build then parse gives the text back.

The writer is the runtime's own, built straight from the value tree like TOML's, rather than through serde_json::Value and quick-xml's serde serializer. That pivot is where XML.build({ r: { a: 0.1 } }) failed: under serde_json's arbitrary_precision feature, which Cargo can switch on from anywhere in the build, every non-integer number becomes a private one-key map, and its key is not an XML name. Owning the writer also lets it be the reader's exact inverse: it escapes \r (the reader normalizes line ends) and attribute whitespace (attribute-value normalization would turn it into spaces), and it refuses a name, a character or an array it cannot spell instead of writing a document the reader would reject.

The stream finds records; the reader reads them. The decoder keeps only the record in progress and the structure around it (inside the root or not, the root's name). When a direct child of the root closes, its bytes go through the same reader as parse. A scan error that only means the chunk stopped mid-construct (an unclosed tag, comment, attribute value or reference) waits for more bytes; any other error ends the stream, and close checks that the root was closed. Bytes go through one streaming TextDecoder, so a character split across chunks decodes once.

YAML

The reader and writer are the runtime's own, in safe Rust (crates/runtime/src/yaml). The usual crate, serde_yaml, is deprecated and is built on unsafe-libyaml, a machine translation of libyaml's C into unsafe Rust. Its repository is archived, so bugs found in it will not be fixed (RUSTSEC-2024-0449). The structure follows the specification: characters become tokens, tokens become the event stream that the yaml-test-suite specifies, and events become values. Because the event layer exists, every one of the suite's 402 cases is checked in cargo test. A valid case must reproduce the suite's events exactly and an invalid one must be rejected. All 402 pass.

Where YAML leaves a choice to the application, the reader takes the one that cannot lose data silently:

• A duplicate key is an error. The specification requires unique keys; taking the last value hides typos in configuration.

• An unknown tag (!Ref, !custom) is an error. Ignoring it would return a value whose meaning was dropped.

• A sequence or mapping used as a key is an error, because a JavaScript object cannot have one as a key.

Merge keys are applied, although YAML 1.2 removed them from the type repository, because CI and compose files depend on them. A key written in the mapping beats a merged one wherever the << appears, which is what readers of those files expect.

Both resources an untrusted document can grow are bounded. Nesting stops at 256 levels, as in XML and MessagePack. Alias expansion stops at 1,000,000 nodes: each alias is a copy, so nine levels of ten aliases each (a few hundred bytes) would otherwise expand to a billion. The tokenizer and the event parser keep their state on the heap, so deep input cannot overflow the stack before it reaches the depth check.

The writer quotes for YAML 1.1 readers too. yes, no, on, off, y and n are plain strings in 1.2, but many readers still take them as booleans, so build quotes them, as js-yaml does by default.

Block-scalar chomping is the specification's, in all three modes — this is worth stating plainly because it reads like a bug when tested carelessly:

a: |          # clip (default): one trailing newline, if the source has one
a: |-         # strip: none
a: |+         # keep: every one

Clip preserves a final line break that exists. A block scalar that ends at end-of-file with no newline after it has none to preserve, so "l1\nl2" is the correct answer there — not a dropped newline.

CSV

The reader is the runtime's own, in safe Rust (crates/runtime/src/csv.rs), for the same reason YAML's is: the surface is small, the interesting cases are malformed, and it sits behind the op boundary where every byte is untrusted. A delimiter, a quote and a line break are all ASCII, so the byte scanner only ever splits on ASCII boundaries and never has to reason about code points.

Fields are strings and rows are arrays by default. That is the shape the general-purpose parsers converge on — Papa Parse, csv-parse, fast-csv, neat-csv and the Rust csv crate all yield rows as field sequences — and the one that loses nothing when the file has no header. header is opt-in, consumes the first row, and only then does a row become an object. Typing is opt-in for the same reason: no general-purpose reader infers types by default, and a guessed 1 — a phone number, a ZIP code — is harder to get back than to avoid.

Malformed input throws rather than collecting errors into an envelope. Only Papa Parse returns { data, errors, meta }, and only because it also handles files and streams; csv-parse, d3-dsv and the Rust csv reader return the value or an error. Every other runtime:serialization parser already throws, so CSV does too.

Strictness is the reader's, not the file's. A duplicate column name, an unterminated quote, and a stray character after a closing quote ("a"b) are all errors, and a row whose field count differs from the header is an error unless relaxColumnCount says otherwise. A field past maxFieldBytes or a record past maxColumns is a RangeError, so one very long line cannot be used to exhaust memory.

A round trip is preserved or refused, never silently lossy. comment drops a whole line, so build quotes any field that starts with the comment byte — otherwise writing then reading would lose the row. cast accepts only a plain decimal f64 can hold: 1e500 stays the string it was rather than becoming Infinity, which build would then refuse, and an integer past 2⁵³ − 1 — an ID, an account number — stays a string rather than coming back with its last digits changed (Papa Parse's dynamicTyping draws the same line). build writes a number as String(n) spells it and a BigInt exactly, so the text is what JavaScript would print. And an option set whose delimiter or quote is a line break, where the two are equal, or whose comment is either of them or a line break, is refused up front: such input has no unambiguous reading — a record starting with an empty field would read as a comment — so accepting it would emit rows the writer could not rebuild.

Streaming is a hybrid: the scan in Rust, the stream in JavaScript. A CSV field cannot be isolated by splitting on \n the way JSONL can — a quoted field may contain newlines and quotes, and a chunk boundary may fall between a \r and its \n, inside a doubled quote, or between the escape character and the byte it prefixes. So the byte scanner stays in Rust (csv.rs StreamDecoder) and is made resumable: it defers the few decisions that need the next byte and keeps the current field and record, so a chunk is consumed once and any split of an input yields the rows one parse would. CSV.DecoderStream is then a thin TransformStream that decodes bytes with a single, streaming TextDecoder (with ignoreBOM, so the bom option decides for bytes as it does for text) and forwards each completed row; a canceled or aborted stream releases the host state that flush would otherwise have freed. The encoder is the mirror image: the first row fixes the shape and the header (csv.rs StreamEncoder), and both reuse exactly the writer build uses, so a streamed document and a built one cannot drift. The decoder's flush returns the last record and reports an unterminated quote — the one error only the end of the stream can see — which is why csv_stream_close returns rows where XML's close returns nothing.

Being a TransformStream, the decoder is backpressured: a writer that runs ahead of the reader is throttled at the readable side's high water mark, so decoding a large document holds only what the consumer has not read, not the whole thing.

JSONL

Streams only: JSONL.DecoderStream and JSONL.EncoderStream, both TransformStreams. There is no parse/stringify pair, because a JSON Lines document is a sequence — the shape it is usually read in is a stream, and the whole-file case is text.split("\n").map(JSON.parse) without a helper.

JSONC and JSON5

One reader serves both, in its own crate (es-runtime-json), with the dialect decided at the call: JSONC refuses what JSON5 adds, because a tsconfig.json that JSONC.parse accepted with single quotes would still fail in tsc.

Parsing builds values in the engine, like BSON's decoder: the reader reports each value as it reads it, and the engine makes the JavaScript value at once, with no JSON in between. JSON could not carry JSON5's Infinity and NaN, and a tree built first and converted after is the per-call garbage the other formats pay. Each object starts empty and gets its properties one at a time, so it stays in V8's fast mode and objects with the same keys share a shape, as JSON.parse's do. V8's other way, an object made with all its properties in one call, comes out as a dictionary, and reading from it later is several times slower. The parser itself never builds a token: it looks at punctuation a byte at a time and only scans where a string, number or name starts.

Editing works on the text, not the value. edit finds the member the path names with the same scanners, reads the file's habits (its indent, the space after its colons, its trailing commas, its quotes), and returns the text with that one value spliced in. Writing the file again from a value cannot do this, however the comments are kept: a value has no record of blank lines, key spacing or where a comment sat. The value itself is written by the dialect's stringify, so edit and stringify cannot disagree about what JSON5 looks like. Why comments are not kept as data is recorded in DECISIONS D191.

The grammar is checked against the suites other parsers use: JSONTestSuite and the JSON5 project's own tests run in both dialects on every cargo test, and every case where a dialect deliberately accepts more is listed with its reason. Two fuzz targets check that whatever JSONC reads, JSON5 reads the same way, and that every edit means exactly the change asked for. scripts/jsonc_large.js and scripts/json5_large.js in bench/ compare each against what Bun, Deno and Node offer.

BSON

BSON is the one format that crosses no JSON and no value tree at all. Its encoder and decoder are engine builtins: the encoder reads the object where it lies and appends bytes, and the decoder builds objects straight from the bytes. A MongoDB driver encodes and decodes a document on every operation, so an intermediate tree there would be garbage made per query, which is what the database drivers were rewritten to avoid. The bytes themselves are read and written by a small crate that knows no engine type, and that crate is what the fuzzer drives.

Decoded objects stay in V8's fast mode. Each document and each class instance starts empty and gets its properties one at a time, so documents with the same fields share one shape, as JSON.parse's objects do. V8's one-call way of making an object with all its properties gives a dictionary instead: quicker to build, but larger, slower to read from, and slower to collect. For a collection's documents the shape is the better trade on every count but one, a document of plain fields only, which builds a little slower; bench/bson/decode.mjs measures each.

The BSON types are classes, recognized by prototype. ObjectId, Decimal128 and the rest are plain JavaScript classes. The engine compares an object's prototype against theirs and reads one property for the value, so an instance costs no call back into JavaScript. The classes hold no private state for the same reason: a decoded instance is built by the engine and never runs its constructor.

Each BSON type reads back as one JavaScript type. An int64 is always a bigint, even 5n, because the alternative of a number when it fits gives one field two types depending on its value, and a number for every int64 loses digits past 2⁵³. A regular expression reads as a BSONRegExp rather than a RegExp: the server's patterns are PCRE, which may not compile in JavaScript, its l and x options have no JavaScript flag, and RegExp.source rewrites / as \/, so a round trip would change the bytes. A JavaScript RegExp still writes. A number is the exception, because JavaScript has one number type for two BSON ones: a whole number writes as int32, so a stored double 1.0 comes back as int32, as it does with the official driver. BSON.Double writes a double on purpose.

Decimal128 text is exact or refused. new Decimal128("0.1") stores exactly 0.1. Text with more than 34 significant digits, or an exponent out of range, is a SyntaxError unless the excess is trailing zeros, which move into the exponent without changing the value. Nothing is rounded silently.

The implementation is checked against MongoDB's BSON corpus and binary-vector tests, the suites every official driver runs (tsr test:bson-corpus). The cases that differ are the ones above by design: documents holding a whole-number double, and a NaN whose payload bits a JavaScript number does not keep. The runner lists each, and fails if one starts passing. The corpus is fetched by mongodb-specs/fetch.sh rather than committed, because its license is not Apache-2.0.

Protobuf

Pure JavaScript and reflective: .proto source is compiled at runtime rather than generated ahead of time. proto3 and editions 2023/2024 only — proto2-specific constructs are rejected rather than half-supported. 64-bit integer fields arrive as BigInt, bytes as Uint8Array, enums as their value-name string.

What it costs

The module needs no capability: it is pure computation over values the guest already holds, and reads nothing. That is also its limit — nothing here opens a file. Parsing a document on disk is runtime:fs first, under its own grant.

The text parsers are Rust; Protobuf is JavaScript in the prelude bundle. Both run on the isolate's thread, so a very large document is a pause, not a background job.

See also

• runtime:serialization reference — signatures and options

• Internals: the fetch client — for bodies arriving over the network