| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
|
Review requested:
|
Sorry, something went wrong.
|
sorry, my mistake |
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
Summary
This PR adds node:kvstore, a new experimental core module: a NoSQL-style
key-value store built as a pure-JavaScript layer over the existing
node:sqlite module. It provides an openKv() factory returning a KVStore
instance with synchronous get, getMany, set, delete, clear, keys
(prefix and range iteration), watch (key-mutation reactive streams), and
publish/watch({ topic }) (in-process pub/sub). No native code of its
own — it inherits node:sqlite's platform support completely.
The module is gated behind --experimental-kvstore, defaulting to enabled
(opt-out via --no-experimental-kvstore) when SQLite is available, mirroring
how node:sqlite behaves since
nodejs/node#55890.
Why in core
key-value façade over it is the natural, minimal next step for the common
case where developers want "put a value under a key" rather than hand-rolled
SQL.
numerically correct — not lexicographic — within a type) and a rich-value
codec (v8.serialize, covering Date/Map/Set/BigInt/circular
references) are correctness-sensitive and easy to get subtly wrong. Doing
it once in core is better than in every userland package that wants this.
The ordering bug was real and confirmed empirically before being fixed (see
Design decisions: N1 below).
exactly this API shape on top of SQLite.
API at a glance
Key design decisions
Composite keys and codec
Keys are tuples of string | number | bigint | boolean encoded as a binary
BLOB using an order-preserving encoding. This was the hardest correctness
problem: decimal-string encoding ("n:2", "n:10") caused kv.keys({ start: [1], end: [100] }) to return [1, 10, 100] instead of
[1, 2, 3, …, 100] — confirmed empirically before being fixed.
Type-segment ordering is string < number < bigint < boolean (a deliberate
divergence from Deno KV's ordering, documented in doc/api/kvstore.md).
Numbers use sign-flipped float64 encoding. Bigints use [sign][length][BE magnitude], capped at 255 bytes (ERR_KVSTORE_BIGINT_TOO_LARGE beyond).
NaN is invalid; ±Infinity are valid; -0 normalizes to 0.
The property-test file (test-kvstore-key-codec-property.js) checks three
invariants across 3 000 random keys using an independently-written
reference comparator (not the codec's own logic), specifically mixing
ASCII, Latin-1, CJK, and astral-plane characters because UTF-16 and UTF-8
byte order diverge in exactly those cases.
Value codec
Values are serialized with v8.serialize / v8.deserialize (the same
algorithm as worker_threads.postMessage). This round-trips Date, RegExp,
Map, Set, typed arrays, BigInt, undefined, sparse arrays, and circular
references losslessly. Class identity does not survive. The on-disk format is
V8-internal: stable across Node versions, not portable cross-runtime.
Error model
Errors are plain Error / TypeError / RangeError instances with a stable
.code, following fs/http/net/crypto convention. No error classes are
exported. Two domain-specific codes are registered in lib/internal/errors.js:
All other validation errors reuse existing Node generic codes
(ERR_INVALID_ARG_TYPE, ERR_INVALID_ARG_VALUE, ERR_OUT_OF_RANGE,
ERR_INVALID_STATE).
Watch scope
key/keys/prefix watchers only observe mutations made through the
same KVStore instance in the same process. A file-backed store mutated
by a second openKv({ path }) call in the same process, or by another
process, produces no watch events on this instance — node:sqlite provides
no cross-connection change feed. This limitation is documented in
doc/api/kvstore.md and tested (see test-kvstore.js).
{ prefix: [] } — full store scan
An empty prefix is valid in kv.keys({ prefix: [] }) and matches every key
in the store. Key validation and prefix validation are separate code paths
(encodeKey vs encodePrefix) — the empty-prefix handling was a distinct
bug that was fixed and explicitly regression-tested.
Concurrent iterators
keys() range/prefix scans use a pooled prepared-statement approach
(lib/internal/kvstore/statements.js): each call acquires a statement from
the pool at iteration start and releases it on completion or break. Concurrent
and interleaved keys() calls therefore never share a cursor. (An earlier
single-shared-statement design corrupted concurrent scans — see SPEC-DECISIONS.md
N3 in the source repo.)
Security
v8.deserialize on user-controlled file paths. Every value read from
the store — including via kv.keys() range/prefix scans — passes through
v8.deserialize. Opening a path containing attacker-controlled bytes is
equivalent to deserializing attacker-controlled V8 payloads. doc/api/kvstore.md
documents this under kvstore.openKv(): "only open files you trust." A
future PR could add a magic-byte/format-version header to reject foreign
database files fast, but that is a file-format change with migration
implications and was deliberately left out of this first PR.
All SQL is fully parameterized. The only interpolated identifier is the
compile-time constant table name (kvstore_v1), never user input.
Known limitations and deliberately deferred items
These are not regressions — they are documented 1.0 constraints, explicitly
tracked for follow-up:
Open questions for reviewers
Error-code granularity. This PR maps broad validation categories to
existing generic codes (ERR_INVALID_ARG_TYPE, ERR_INVALID_ARG_VALUE,
ERR_OUT_OF_RANGE). An alternative would be finer-grained per-subsystem
codes (e.g. ERR_KVSTORE_INVALID_SELECTOR, ERR_KVSTORE_INVALID_KEY)
at the cost of more entries in lib/internal/errors.js. Which direction
does core prefer?
Sync-only API. All operations are synchronous, matching DatabaseSync.
Given node:sqlite's own experience with sync-only as a starting point,
is this acceptable for the initial experimental release, or should an
async variant be considered from the start?
watch() in-process scope. The current scope (same-instance-only) is
a strict but honest v1 constraint. Is this an acceptable limitation for
Stability: 1, or a blocker that requires a path to cross-connection
change feeds before shipping?
Feature scope. versionstamp (to disambiguate absent from stored
null) and atomic/CAS operations are both deferred. Does core want either
on day one given how much harder they are to add post-stabilization?
Testing
including error paths, watch lifecycle, backpressure/lag, topic pub/sub,
persistence with a file-backed store, and cross-instance watch isolation.
pinning the specific ordering cases (including the [1,10,100] vs
[1,2,3,…,100] N1 bug), codec round-trips, and boundary values.
random composite keys per run, three invariants (round-trip, order-preserving,
prefix containment), checked against an independently-written reference
comparator that cannot silently mirror a codec bug.
prepareDb() including a coverage gap not reachable through the public API
(getMany([]) early-return).
require('kvstore') → MODULE_NOT_FOUND; --no-experimental-kvstore
disables require('node:kvstore').
hasKVStore.
Benchmarks
benchmark/kvstore/ contains four scenarios (set, get, getMany, range-scan),
each using Node's standard createBenchmark() API. For reference, the
NoNoSQLite source repo's own benchmark (bench/run.mjs) compares these
operations against a hand-rolled node:sqlite KV table using naive
JSON-text keys and values — the overhead of node:kvstore's correctness layer
(order-preserving binary keys, v8 codec, validation) is approximately
1.5–2.6× across scenarios on an in-memory store.
Documentation
doc/api/kvstore.md covers:
file-backed store on a local SSD)
ESM and CJS
and migration note from the pre-1.0 positional form
security warning
queue, reactive audit log
Smoke test
After building:
Flag opt-out:
Schemeless access blocked: