FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

child_process,cluster: allow using V8 serialization API · nodejs/node@4c317ce · GitHub

/ node Public

Commit 4c317ce

Browse files
addaleax
authored and
targos
committed
child_process,cluster: allow using V8 serialization API
Add an `serialization` option that allows child process IPC to use the (typically more powerful) V8 serialization API. Fixes: #10965 PR-URL: #30162 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: David Carlier <devnexen@gmail.com> Reviewed-By: Michaël Zasso <targos@protonmail.com>
1 parent 56188fe commit 4c317ce

13 files changed

Lines changed: 304 additions & 39 deletions

File tree

‎benchmark/cluster/echo.js‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,16 +7,25 @@ if (cluster.isMaster) {
77
workers: [1],
88
payload: ['string', 'object'],
99
sendsPerBroadcast: [1, 10],
10+
serialization: ['json', 'advanced'],
1011
n: [1e5]
1112
});
1213

13-
function main({ n, workers, sendsPerBroadcast, payload }) {
14+
function main({
15+
n,
16+
workers,
17+
sendsPerBroadcast,
18+
payload,
19+
serialization
20+
}) {
1421
const expectedPerBroadcast = sendsPerBroadcast * workers;
1522
var readies = 0;
1623
var broadcasts = 0;
1724
var msgCount = 0;
1825
var data;
1926

27+
cluster.settings.serialization = serialization;
28+
2029
switch (payload) {
2130
case 'string':
2231
data = 'hello world!';

‎doc/api/child_process.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -321,6 +321,9 @@ arbitrary command execution.**
321321
<!-- YAML
322322
added: v0.5.0
323323
changes:
324+
- version: REPLACEME
325+
pr-url: https://github.com/nodejs/node/pull/30162
326+
description: The `serialization` option is supported now.
324327
- version: v8.0.0
325328
pr-url: https://github.com/nodejs/node/pull/10866
326329
description: The `stdio` option can now be a string.
@@ -340,6 +343,9 @@ changes:
340343
* `execPath` {string} Executable used to create the child process.
341344
* `execArgv` {string[]} List of string arguments passed to the executable.
342345
**Default:** `process.execArgv`.
346+
* `serialization` {string} Specify the kind of serialization used for sending
347+
messages between processes. Possible values are `'json'` and `'advanced'`.
348+
See [Advanced Serialization][] for more details. **Default:** `'json'`.
343349
* `silent` {boolean} If `true`, stdin, stdout, and stderr of the child will be
344350
piped to the parent, otherwise they will be inherited from the parent, see
345351
the `'pipe'` and `'inherit'` options for [`child_process.spawn()`][]'s
@@ -386,6 +392,9 @@ The `shell` option available in [`child_process.spawn()`][] is not supported by
386392
<!-- YAML
387393
added: v0.1.90
388394
changes:
395+
- version: REPLACEME
396+
pr-url: https://github.com/nodejs/node/pull/30162
397+
description: The `serialization` option is supported now.
389398
- version: v8.8.0
390399
pr-url: https://github.com/nodejs/node/pull/15380
391400
description: The `windowsHide` option is supported now.
@@ -411,6 +420,9 @@ changes:
411420
[`options.detached`][]).
412421
* `uid` {number} Sets the user identity of the process (see setuid(2)).
413422
* `gid` {number} Sets the group identity of the process (see setgid(2)).
423+
* `serialization` {string} Specify the kind of serialization used for sending
424+
messages between processes. Possible values are `'json'` and `'advanced'`.
425+
See [Advanced Serialization][] for more details. **Default:** `'json'`.
414426
* `shell` {boolean|string} If `true`, runs `command` inside of a shell. Uses
415427
`'/bin/sh'` on Unix, and `process.env.ComSpec` on Windows. A different
416428
shell can be specified as a string. See [Shell Requirements][] and
@@ -998,6 +1010,11 @@ The `'message'` event is triggered when a child process uses
9981010
The message goes through serialization and parsing. The resulting
9991011
message might not be the same as what is originally sent.
10001012

1013+
If the `serialization` option was set to `'advanced'` used when spawning the
1014+
child process, the `message` argument can contain data that JSON is not able
1015+
to represent.
1016+
See [Advanced Serialization][] for more details.
1017+
10011018
### `subprocess.channel`
10021019
<!-- YAML
10031020
added: v7.1.0
@@ -1474,6 +1491,26 @@ the same requirement. Thus, in `child_process` functions where a shell can be
14741491
spawned, `'cmd.exe'` is used as a fallback if `process.env.ComSpec` is
14751492
unavailable.
14761493

1494+
## Advanced Serialization
1495+
<!-- YAML
1496+
added: REPLACEME
1497+
-->
1498+
1499+
Child processes support a serialization mechanism for IPC that is based on the
1500+
[serialization API of the `v8` module][v8.serdes], based on the
1501+
[HTML structured clone algorithm][]. This is generally more powerful and
1502+
supports more built-in JavaScript object types, such as `BigInt`, `Map`
1503+
and `Set`, `ArrayBuffer` and `TypedArray`, `Buffer`, `Error`, `RegExp` etc.
1504+
1505+
However, this format is not a full superset of JSON, and e.g. properties set on
1506+
objects of such built-in types will not be passed on through the serialization
1507+
step. Additionally, performance may not be equivalent to that of JSON, depending
1508+
on the structure of the passed data.
1509+
Therefore, this feature requires opting in by setting the
1510+
`serialization` option to `'advanced'` when calling [`child_process.spawn()`][]
1511+
or [`child_process.fork()`][].
1512+
1513+
[Advanced Serialization]: #child_process_advanced_serialization
14771514
[`'disconnect'`]: process.html#process_event_disconnect
14781515
[`'error'`]: #child_process_event_error
14791516
[`'exit'`]: #child_process_event_exit
@@ -1507,5 +1544,7 @@ unavailable.
15071544
[`subprocess.stdout`]: #child_process_subprocess_stdout
15081545
[`util.promisify()`]: util.html#util_util_promisify_original
15091546
[Default Windows Shell]: #child_process_default_windows_shell
1547+
[HTML structured clone algorithm]: https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Structured_clone_algorithm
15101548
[Shell Requirements]: #child_process_shell_requirements
15111549
[synchronous counterparts]: #child_process_synchronous_process_creation
1550+
[v8.serdes]: v8.html#v8_serialization_api

‎doc/api/cluster.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -724,6 +724,9 @@ values are `'rr'` and `'none'`.
724724
<!-- YAML
725725
added: v0.7.1
726726
changes:
727+
- version: REPLACEME
728+
pr-url: https://github.com/nodejs/node/pull/30162
729+
description: The `serialization` option is supported now.
727730
- version: v9.5.0
728731
pr-url: https://github.com/nodejs/node/pull/18399
729732
description: The `cwd` option is supported now.
@@ -746,6 +749,10 @@ changes:
746749
**Default:** `process.argv.slice(2)`.
747750
* `cwd` {string} Current working directory of the worker process. **Default:**
748751
`undefined` (inherits from parent process).
752+
* `serialization` {string} Specify the kind of serialization used for sending
753+
messages between processes. Possible values are `'json'` and `'advanced'`.
754+
See [Advanced Serialization for `child_process`][] for more details.
755+
**Default:** `false`.
749756
* `silent` {boolean} Whether or not to send output to parent's stdio.
750757
**Default:** `false`.
751758
* `stdio` {Array} Configures the stdio of forked processes. Because the
@@ -874,4 +881,5 @@ socket.on('data', (id) => {
874881
[`process` event: `'message'`]: process.html#process_event_message
875882
[`server.close()`]: net.html#net_event_close
876883
[`worker.exitedAfterDisconnect`]: #cluster_worker_exitedafterdisconnect
884+
[Advanced Serialization for `child_process`]: child_process.html#child_process_advanced_serialization
877885
[Child Process module]: child_process.html#child_process_child_process_fork_modulepath_args_options

‎doc/api/process.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,11 @@ the child process.
119119
The message goes through serialization and parsing. The resulting message might
120120
not be the same as what is originally sent.
121121

122+
If the `serialization` option was set to `advanced` used when spawning the
123+
process, the `message` argument can contain data that JSON is not able
124+
to represent.
125+
See [Advanced Serialization for `child_process`][] for more details.
126+
122127
### Event: `'multipleResolves'`
123128
<!-- YAML
124129
added: v10.12.0
@@ -2457,6 +2462,7 @@ cases:
24572462
[`require.resolve()`]: modules.html#modules_require_resolve_request_options
24582463
[`subprocess.kill()`]: child_process.html#child_process_subprocess_kill_signal
24592464
[`v8.setFlagsFromString()`]: v8.html#v8_v8_setflagsfromstring_flags
2465+
[Advanced Serialization for `child_process`]: child_process.html#child_process_advanced_serialization
24602466
[Android building]: https://github.com/nodejs/node/blob/master/BUILDING.md#androidandroid-based-devices-eg-firefox-os
24612467
[Child Process]: child_process.html
24622468
[Cluster]: cluster.html

‎lib/child_process.js‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -108,12 +108,12 @@ function fork(modulePath /* , args, options */) {
108108
return spawn(options.execPath, args, options);
109109
}
110110

111-
function _forkChild(fd) {
111+
function _forkChild(fd, serializationMode) {
112112
// set process.send()
113113
const p = new Pipe(PipeConstants.IPC);
114114
p.open(fd);
115115
p.unref();
116-
const control = setupChannel(process, p);
116+
const control = setupChannel(process, p, serializationMode);
117117
process.on('newListener', function onNewListener(name) {
118118
if (name === 'message' || name === 'disconnect') control.ref();
119119
});
@@ -547,7 +547,8 @@ function spawn(file, args, options) {
547547
envPairs: opts.envPairs,
548548
stdio: options.stdio,
549549
uid: options.uid,
550-
gid: options.gid
550+
gid: options.gid,
551+
serialization: options.serialization,
551552
});
552553

553554
return child;

‎lib/internal/bootstrap/pre_execution.js‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -326,7 +326,11 @@ function setupChildProcessIpcChannel() {
326326
// Make sure it's not accidentally inherited by child processes.
327327
delete process.env.NODE_CHANNEL_FD;
328328

329-
require('child_process')._forkChild(fd);
329+
const serializationMode =
330+
process.env.NODE_CHANNEL_SERIALIZATION_MODE || 'json';
331+
delete process.env.NODE_CHANNEL_SERIALIZATION_MODE;
332+
333+
require('child_process')._forkChild(fd, serializationMode);
330334
assert(process.send);
331335
}
332336
}

‎lib/internal/child_process.js‎

Lines changed: 27 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
'use strict';
22

3-
const { JSON, Object } = primordials;
3+
const { Object } = primordials;
44

55
const {
66
errnoException,
@@ -55,8 +55,6 @@ const {
5555

5656
const { SocketListSend, SocketListReceive } = SocketList;
5757

58-
// Lazy loaded for startup performance.
59-
let StringDecoder;
6058
// Lazy loaded for startup performance and to allow monkey patching of
6159
// internalBinding('http_parser').HTTPParser.
6260
let freeParser;
@@ -343,6 +341,15 @@ ChildProcess.prototype.spawn = function(options) {
343341
const ipcFd = stdio.ipcFd;
344342
stdio = options.stdio = stdio.stdio;
345343

344+
if (options.serialization !== undefined &&
345+
options.serialization !== 'json' &&
346+
options.serialization !== 'advanced') {
347+
throw new ERR_INVALID_OPT_VALUE('options.serialization',
348+
options.serialization);
349+
}
350+
351+
const serialization = options.serialization || 'json';
352+
346353
if (ipc !== undefined) {
347354
// Let child process know about opened IPC channel
348355
if (options.envPairs === undefined)
@@ -353,7 +360,8 @@ ChildProcess.prototype.spawn = function(options) {
353360
options.envPairs);
354361
}
355362

356-
options.envPairs.push('NODE_CHANNEL_FD=' + ipcFd);
363+
options.envPairs.push(`NODE_CHANNEL_FD=${ipcFd}`);
364+
options.envPairs.push(`NODE_CHANNEL_SERIALIZATION_MODE=${serialization}`);
357365
}
358366

359367
validateString(options.file, 'options.file');
@@ -446,7 +454,7 @@ ChildProcess.prototype.spawn = function(options) {
446454
this.stdio.push(stdio[i].socket === undefined ? null : stdio[i].socket);
447455

448456
// Add .send() method and start listening for IPC data
449-
if (ipc !== undefined) setupChannel(this, ipc);
457+
if (ipc !== undefined) setupChannel(this, ipc, serialization);
450458

451459
return err;
452460
};
@@ -513,7 +521,8 @@ class Control extends EventEmitter {
513521
}
514522
}
515523

516-
function setupChannel(target, channel) {
524+
let serialization;
525+
function setupChannel(target, channel, serializationMode) {
517526
target.channel = channel;
518527

519528
// _channel can be deprecated in version 8
@@ -528,12 +537,16 @@ function setupChannel(target, channel) {
528537

529538
const control = new Control(channel);
530539

531-
if (StringDecoder === undefined)
532-
StringDecoder = require('string_decoder').StringDecoder;
533-
const decoder = new StringDecoder('utf8');
534-
var jsonBuffer = '';
535-
var pendingHandle = null;
536-
channel.buffering = false;
540+
if (serialization === undefined)
541+
serialization = require('internal/child_process/serialization');
542+
const {
543+
initMessageChannel,
544+
parseChannelMessages,
545+
writeChannelMessage
546+
} = serialization[serializationMode];
547+
548+
let pendingHandle = null;
549+
initMessageChannel(channel);
537550
channel.pendingHandle = null;
538551
channel.onread = function(arrayBuffer) {
539552
const recvHandle = channel.pendingHandle;
@@ -545,21 +558,7 @@ function setupChannel(target, channel) {
545558
if (recvHandle)
546559
pendingHandle = recvHandle;
547560

548-
// Linebreak is used as a message end sign
549-
var chunks = decoder.write(pool).split('\n');
550-
var numCompleteChunks = chunks.length - 1;
551-
// Last line does not have trailing linebreak
552-
var incompleteChunk = chunks[numCompleteChunks];
553-
if (numCompleteChunks === 0) {
554-
jsonBuffer += incompleteChunk;
555-
this.buffering = jsonBuffer.length !== 0;
556-
return;
557-
}
558-
chunks[0] = jsonBuffer + chunks[0];
559-
560-
for (var i = 0; i < numCompleteChunks; i++) {
561-
var message = JSON.parse(chunks[i]);
562-
561+
for (const message of parseChannelMessages(channel, pool)) {
563562
// There will be at most one NODE_HANDLE message in every chunk we
564563
// read because SCM_RIGHTS messages don't get coalesced. Make sure
565564
// that we deliver the handle with the right message however.
@@ -574,9 +573,6 @@ function setupChannel(target, channel) {
574573
handleMessage(message, undefined, false);
575574
}
576575
}
577-
jsonBuffer = incompleteChunk;
578-
this.buffering = jsonBuffer.length !== 0;
579-
580576
} else {
581577
this.buffering = false;
582578
target.disconnect();
@@ -775,8 +771,7 @@ function setupChannel(target, channel) {
775771

776772
const req = new WriteWrap();
777773

778-
const string = JSON.stringify(message) + '\n';
779-
const err = channel.writeUtf8String(req, string, handle);
774+
const err = writeChannelMessage(channel, req, message, handle);
780775
const wasAsyncWrite = streamBaseState[kLastWriteWasAsync];
781776

782777
if (err === 0) {

0 commit comments

Comments
 (0)
Back | FazBrowse Home | New Git URL