FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

sqlite: add serialize() and deserialize() · nodejs/node@3af44ee · GitHub

/ node Public

Commit 3af44ee

Browse files
thisalihassan
authored and
aduh95
committed
sqlite: add serialize() and deserialize()
Add database.serialize() and database.deserialize() methods to DatabaseSync, wrapping the sqlite3_serialize and sqlite3_deserialize C APIs. These allow extracting an in-memory database as a Uint8Array and loading one back, enabling snapshots, cloning, and transfer of databases without filesystem I/O. Refs: #62575 PR-URL: #62579 Refs: #62575 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Edy Silva <edigleyssonsilva@gmail.com>
1 parent 57e59ea commit 3af44ee

4 files changed

Lines changed: 526 additions & 0 deletions

File tree

‎doc/api/sqlite.md‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -531,6 +531,89 @@ Opens the database specified in the `path` argument of the `DatabaseSync`
531531
constructor. This method should only be used when the database is not opened via
532532
the constructor. An exception is thrown if the database is already open.
533533

534+
### `database.serialize([dbName])`
535+
536+
<!-- YAML
537+
added: REPLACEME
538+
-->
539+
540+
* `dbName` {string} Name of the database to serialize. This can be `'main'`
541+
(the default primary database) or any other database that has been added with
542+
[`ATTACH DATABASE`][]. **Default:** `'main'`.
543+
* Returns: {Uint8Array} A binary representation of the database.
544+
545+
Serializes the database into a binary representation, returned as a
546+
`Uint8Array`. This is useful for saving, cloning, or transferring an in-memory
547+
database. This method is a wrapper around [`sqlite3_serialize()`][].
548+
549+
```mjs
550+
import { DatabaseSync } from 'node:sqlite';
551+
552+
const db = new DatabaseSync(':memory:');
553+
db.exec('CREATE TABLE t(key INTEGER PRIMARY KEY, value TEXT)');
554+
db.exec("INSERT INTO t VALUES (1, 'hello')");
555+
const buffer = db.serialize();
556+
console.log(buffer.length); // Prints the byte length of the database
557+
```
558+
559+
```cjs
560+
const { DatabaseSync } = require('node:sqlite');
561+
562+
const db = new DatabaseSync(':memory:');
563+
db.exec('CREATE TABLE t(key INTEGER PRIMARY KEY, value TEXT)');
564+
db.exec("INSERT INTO t VALUES (1, 'hello')");
565+
const buffer = db.serialize();
566+
console.log(buffer.length); // Prints the byte length of the database
567+
```
568+
569+
### `database.deserialize(buffer[, options])`
570+
571+
<!-- YAML
572+
added: REPLACEME
573+
-->
574+
575+
* `buffer` {Uint8Array} A binary representation of a database, such as the
576+
output of [`database.serialize()`][].
577+
* `options` {Object} Optional configuration for the deserialization.
578+
* `dbName` {string} Name of the database to deserialize into.
579+
**Default:** `'main'`.
580+
581+
Loads a serialized database into this connection, replacing the current
582+
database. The deserialized database is writable. Existing prepared statements
583+
are finalized before deserialization is attempted, even if the operation
584+
subsequently fails. This method is a wrapper around
585+
[`sqlite3_deserialize()`][].
586+
587+
```mjs
588+
import { DatabaseSync } from 'node:sqlite';
589+
590+
const original = new DatabaseSync(':memory:');
591+
original.exec('CREATE TABLE t(key INTEGER PRIMARY KEY, value TEXT)');
592+
original.exec("INSERT INTO t VALUES (1, 'hello')");
593+
const buffer = original.serialize();
594+
original.close();
595+
596+
const clone = new DatabaseSync(':memory:');
597+
clone.deserialize(buffer);
598+
console.log(clone.prepare('SELECT value FROM t').get());
599+
// Prints: { value: 'hello' }
600+
```
601+
602+
```cjs
603+
const { DatabaseSync } = require('node:sqlite');
604+
605+
const original = new DatabaseSync(':memory:');
606+
original.exec('CREATE TABLE t(key INTEGER PRIMARY KEY, value TEXT)');
607+
original.exec("INSERT INTO t VALUES (1, 'hello')");
608+
const buffer = original.serialize();
609+
original.close();
610+
611+
const clone = new DatabaseSync(':memory:');
612+
clone.deserialize(buffer);
613+
console.log(clone.prepare('SELECT value FROM t').get());
614+
// Prints: { value: 'hello' }
615+
```
616+
534617
### `database.prepare(sql[, options])`
535618

536619
<!-- YAML
@@ -1545,6 +1628,7 @@ callback function to indicate what type of operation is being authorized.
15451628
[`SQLTagStore`]: #class-sqltagstore
15461629
[`database.applyChangeset()`]: #databaseapplychangesetchangeset-options
15471630
[`database.createTagStore()`]: #databasecreatetagstoremaxsize
1631+
[`database.serialize()`]: #databaseserializedbname
15481632
[`database.setAuthorizer()`]: #databasesetauthorizercallback
15491633
[`sqlite3_backup_finish()`]: https://www.sqlite.org/c3ref/backup_finish.html#sqlite3backupfinish
15501634
[`sqlite3_backup_init()`]: https://www.sqlite.org/c3ref/backup_finish.html#sqlite3backupinit
@@ -1559,12 +1643,14 @@ callback function to indicate what type of operation is being authorized.
15591643
[`sqlite3_create_function_v2()`]: https://www.sqlite.org/c3ref/create_function.html
15601644
[`sqlite3_create_window_function()`]: https://www.sqlite.org/c3ref/create_function.html
15611645
[`sqlite3_db_filename()`]: https://sqlite.org/c3ref/db_filename.html
1646+
[`sqlite3_deserialize()`]: https://sqlite.org/c3ref/deserialize.html
15621647
[`sqlite3_exec()`]: https://www.sqlite.org/c3ref/exec.html
15631648
[`sqlite3_expanded_sql()`]: https://www.sqlite.org/c3ref/expanded_sql.html
15641649
[`sqlite3_get_autocommit()`]: https://sqlite.org/c3ref/get_autocommit.html
15651650
[`sqlite3_last_insert_rowid()`]: https://www.sqlite.org/c3ref/last_insert_rowid.html
15661651
[`sqlite3_load_extension()`]: https://www.sqlite.org/c3ref/load_extension.html
15671652
[`sqlite3_prepare_v2()`]: https://www.sqlite.org/c3ref/prepare.html
1653+
[`sqlite3_serialize()`]: https://sqlite.org/c3ref/serialize.html
15681654
[`sqlite3_set_authorizer()`]: https://sqlite.org/c3ref/set_authorizer.html
15691655
[`sqlite3_sql()`]: https://www.sqlite.org/c3ref/expanded_sql.html
15701656
[`sqlite3changeset_apply()`]: https://www.sqlite.org/session/sqlite3changeset_apply.html

‎src/node_sqlite.cc‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ namespace sqlite {
2323
using v8::Array;
2424
using v8::ArrayBuffer;
2525
using v8::BackingStoreInitializationMode;
26+
using v8::BackingStoreOnFailureMode;
2627
using v8::BigInt;
2728
using v8::Boolean;
2829
using v8::ConstructorBehavior;
@@ -1719,6 +1720,136 @@ void DatabaseSync::Location(const FunctionCallbackInfo<Value>& args) {
17191720
}
17201721
}
17211722

1723+
void DatabaseSync::Serialize(const FunctionCallbackInfo<Value>& args) {
1724+
DatabaseSync* db;
1725+
ASSIGN_OR_RETURN_UNWRAP(&db, args.This());
1726+
Environment* env = Environment::GetCurrent(args);
1727+
THROW_AND_RETURN_ON_BAD_STATE(env, !db->IsOpen(), "database is not open");
1728+
1729+
std::string db_name = "main";
1730+
if (!args[0]->IsUndefined()) {
1731+
if (!args[0]->IsString()) {
1732+
THROW_ERR_INVALID_ARG_TYPE(env->isolate(),
1733+
"The \"dbName\" argument must be a string.");
1734+
return;
1735+
}
1736+
db_name = Utf8Value(env->isolate(), args[0].As<String>()).ToString();
1737+
}
1738+
1739+
sqlite3_int64 size = 0;
1740+
unsigned char* data =
1741+
sqlite3_serialize(db->connection_, db_name.c_str(), &size, 0);
1742+
1743+
if (data == nullptr) {
1744+
if (size == 0) {
1745+
Local<ArrayBuffer> ab = ArrayBuffer::New(env->isolate(), 0);
1746+
args.GetReturnValue().Set(Uint8Array::New(ab, 0, 0));
1747+
return;
1748+
}
1749+
THROW_ERR_SQLITE_ERROR(env->isolate(), db);
1750+
return;
1751+
}
1752+
1753+
// V8 sandbox forbids external backing stores so allocate inside the
1754+
// sandbox and copy. Without sandbox wrap the output directly using
1755+
// sqlite3_free as the destructor to avoid the copy.
1756+
#ifdef V8_ENABLE_SANDBOX
1757+
auto free_data = OnScopeLeave([&] { sqlite3_free(data); });
1758+
auto store = ArrayBuffer::NewBackingStore(
1759+
env->isolate(),
1760+
size,
1761+
BackingStoreInitializationMode::kUninitialized,
1762+
BackingStoreOnFailureMode::kReturnNull);
1763+
if (!store) {
1764+
THROW_ERR_MEMORY_ALLOCATION_FAILED(env);
1765+
return;
1766+
}
1767+
memcpy(store->Data(), data, size);
1768+
Local<ArrayBuffer> ab = ArrayBuffer::New(env->isolate(), std::move(store));
1769+
#else
1770+
auto store = ArrayBuffer::NewBackingStore(
1771+
data, size, [](void* ptr, size_t, void*) { sqlite3_free(ptr); }, nullptr);
1772+
Local<ArrayBuffer> ab = ArrayBuffer::New(env->isolate(), std::move(store));
1773+
#endif
1774+
1775+
args.GetReturnValue().Set(Uint8Array::New(ab, 0, size));
1776+
}
1777+
1778+
void DatabaseSync::Deserialize(const FunctionCallbackInfo<Value>& args) {
1779+
DatabaseSync* db;
1780+
ASSIGN_OR_RETURN_UNWRAP(&db, args.This());
1781+
Environment* env = Environment::GetCurrent(args);
1782+
THROW_AND_RETURN_ON_BAD_STATE(env, !db->IsOpen(), "database is not open");
1783+
1784+
if (!args[0]->IsUint8Array()) {
1785+
THROW_ERR_INVALID_ARG_TYPE(env->isolate(),
1786+
"The \"buffer\" argument must be a Uint8Array.");
1787+
return;
1788+
}
1789+
1790+
Local<Uint8Array> input = args[0].As<Uint8Array>();
1791+
size_t byte_length = input->ByteLength();
1792+
1793+
if (byte_length == 0) {
1794+
THROW_ERR_INVALID_ARG_VALUE(env,
1795+
"The \"buffer\" argument must not be empty.");
1796+
return;
1797+
}
1798+
1799+
std::string db_name = "main";
1800+
if (args.Length() > 1 && !args[1]->IsUndefined()) {
1801+
if (!args[1]->IsObject()) {
1802+
THROW_ERR_INVALID_ARG_TYPE(env->isolate(),
1803+
"The \"options\" argument must be an object.");
1804+
return;
1805+
}
1806+
1807+
Local<Object> options = args[1].As<Object>();
1808+
Local<String> db_name_key = FIXED_ONE_BYTE_STRING(env->isolate(), "dbName");
1809+
Local<Value> db_name_value;
1810+
if (!options->Get(env->context(), db_name_key).ToLocal(&db_name_value)) {
1811+
return;
1812+
}
1813+
1814+
if (!db_name_value->IsUndefined()) {
1815+
if (!db_name_value->IsString()) {
1816+
THROW_ERR_INVALID_ARG_TYPE(
1817+
env->isolate(),
1818+
"The \"options.dbName\" argument must be a string.");
1819+
return;
1820+
}
1821+
db_name =
1822+
Utf8Value(env->isolate(), db_name_value.As<String>()).ToString();
1823+
}
1824+
}
1825+
1826+
// sqlite3_malloc64 is required because SQLITE_DESERIALIZE_FREEONCLOSE
1827+
// transfers ownership to SQLite, which calls sqlite3_free() on close.
1828+
// See: https://www.sqlite.org/c3ref/deserialize.html
1829+
unsigned char* buf =
1830+
static_cast<unsigned char*>(sqlite3_malloc64(byte_length));
1831+
if (buf == nullptr) {
1832+
THROW_ERR_MEMORY_ALLOCATION_FAILED(env);
1833+
return;
1834+
}
1835+
1836+
input->CopyContents(buf, byte_length);
1837+
1838+
db->FinalizeStatements();
1839+
1840+
int r = sqlite3_deserialize(
1841+
db->connection_,
1842+
db_name.c_str(),
1843+
buf,
1844+
byte_length,
1845+
byte_length,
1846+
SQLITE_DESERIALIZE_FREEONCLOSE | SQLITE_DESERIALIZE_RESIZEABLE);
1847+
if (r != SQLITE_OK) {
1848+
THROW_ERR_SQLITE_ERROR(env->isolate(), db);
1849+
return;
1850+
}
1851+
}
1852+
17221853
void DatabaseSync::AggregateFunction(const FunctionCallbackInfo<Value>& args) {
17231854
DatabaseSync* db;
17241855
ASSIGN_OR_RETURN_UNWRAP(&db, args.This());
@@ -3766,6 +3897,8 @@ static void Initialize(Local<Object> target,
37663897
isolate, db_tmpl, "enableDefensive", DatabaseSync::EnableDefensive);
37673898
SetProtoMethod(
37683899
isolate, db_tmpl, "loadExtension", DatabaseSync::LoadExtension);
3900+
SetProtoMethod(isolate, db_tmpl, "serialize", DatabaseSync::Serialize);
3901+
SetProtoMethod(isolate, db_tmpl, "deserialize", DatabaseSync::Deserialize);
37693902
SetProtoMethod(
37703903
isolate, db_tmpl, "setAuthorizer", DatabaseSync::SetAuthorizer);
37713904
SetSideEffectFreeGetter(isolate,

‎src/node_sqlite.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -195,6 +195,8 @@ class DatabaseSync : public BaseObject {
195195
static void EnableDefensive(const v8::FunctionCallbackInfo<v8::Value>& args);
196196
static void LimitsGetter(const v8::FunctionCallbackInfo<v8::Value>& args);
197197
static void LoadExtension(const v8::FunctionCallbackInfo<v8::Value>& args);
198+
static void Serialize(const v8::FunctionCallbackInfo<v8::Value>& args);
199+
static void Deserialize(const v8::FunctionCallbackInfo<v8::Value>& args);
198200
static void SetAuthorizer(const v8::FunctionCallbackInfo<v8::Value>& args);
199201
static int AuthorizerCallback(void* user_data,
200202
int action_code,

0 commit comments

Comments
 (0)
Back | FazBrowse Home | New Git URL