FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

src: add support for externally shared js builtins · nodejs/node@6a01d39 · GitHub

/ node Public

Commit 6a01d39

Browse files
mhdawson
authored and
BethGriggs
committed
src: add support for externally shared js builtins
Refs: #44000 - add infra to support externally shared js builtins in support of distos that want to externalize deps that include JS/WASM instead of native code - add support for externalizing - cjs_module_lexer/lexer - cjs_module_lexer/dist/lexer - undici/undici Signed-off-by: Michael Dawson <mdawson@devrus.com> PR-URL: #44376 Backport-PR-URL: #45867 Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
1 parent 962a747 commit 6a01d39

6 files changed

Lines changed: 229 additions & 3 deletions

File tree

‎BUILDING.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1062,6 +1062,44 @@ To make `./myModule.js` available via `require('myModule')` and
10621062
> .\vcbuild link-module './myModule.js' link-module './myModule2.js'
10631063
```
10641064

1065+
## Building to use shared dependencies at runtime
1066+
1067+
By default Node.js is built so that all dependencies are bundled into
1068+
the Node.js binary itself. This provides a single binary that includes
1069+
the correct versions of all dependencies on which it depends.
1070+
1071+
Some Node.js distributions, however, prefer to manage dependencies.
1072+
A number of `configure` options are provided to support this use case.
1073+
1074+
* For dependencies with native code, the first set of options allow
1075+
Node.js to be built so that it uses a shared library
1076+
at runtime instead of building and including the dependency
1077+
in the Node.js binary itself. These options are in the
1078+
`Shared libraries` section of the `configure` help
1079+
(run `./configure --help` to get the complete list).
1080+
They provide the ability to enable the use of a shared library,
1081+
to set the name of the shared library, and to set the paths that
1082+
contain the include and shared library files.
1083+
1084+
* For dependencies with JavaScript code (including WASM), the second
1085+
set of options allow the Node.js binary to be built so that it loads
1086+
the JavaScript for dependencies at runtime instead of being built into
1087+
the Node.js binary itself. These options are in the `Shared builtins`
1088+
section of the `configure` help
1089+
(run `./configure --help` to get the complete list). They
1090+
provide the ability to set the path to an external JavaScript file
1091+
for the dependency to be used at runtime.
1092+
1093+
It is the responsibility of any distribution
1094+
shipping with these options to:
1095+
1096+
* ensure that the shared dependencies available at runtime
1097+
match what is expected by the Node.js binary. A
1098+
mismatch may result in crashes or unexpected behavior.
1099+
* fully test that Node.js operates as expected with the
1100+
external dependencies. There may be little or no test coverage
1101+
within the Node.js project CI for these non-default options.
1102+
10651103
## Note for downstream distributors of Node.js
10661104

10671105
The Node.js ecosystem is reliant on ABI compatibility within a major release.

‎configure.py‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,11 @@
5757
with open ('tools/icu/icu_versions.json') as f:
5858
icu_versions = json.load(f)
5959

60+
shareable_builtins = {'cjs_module_lexer/lexer': 'deps/cjs-module-lexer/lexer.js',
61+
'cjs_module_lexer/dist/lexer': 'deps/cjs-module-lexer/dist/lexer.js',
62+
'undici/undici': 'deps/undici/undici.js'
63+
}
64+
6065
# create option groups
6166
shared_optgroup = parser.add_argument_group("Shared libraries",
6267
"Flags that allows you to control whether you want to build against "
@@ -70,6 +75,9 @@
7075
"library you want to build against.")
7176
http2_optgroup = parser.add_argument_group("HTTP2",
7277
"Flags that allows you to control HTTP2 features in Node.js")
78+
shared_builtin_optgroup = parser.add_argument_group("Shared builtins",
79+
"Flags that allows you to control whether you want to build against "
80+
"internal builtins or shared files.")
7381

7482
# Options should be in alphabetical order but keep --prefix at the top,
7583
# that's arguably the one people will be looking for most.
@@ -427,6 +435,16 @@
427435

428436
parser.add_argument_group(shared_optgroup)
429437

438+
for builtin in shareable_builtins:
439+
builtin_id = 'shared_builtin_' + builtin + '_path'
440+
shared_builtin_optgroup.add_argument('--shared-builtin-' + builtin + '-path',
441+
action='store',
442+
dest='node_shared_builtin_' + builtin.replace('/', '_') + '_path',
443+
help='Path to shared file for ' + builtin + ' builtin. '
444+
'Will be used instead of bundled version at runtime')
445+
446+
parser.add_argument_group(shared_builtin_optgroup)
447+
430448
static_optgroup.add_argument('--static-zoslib-gyp',
431449
action='store',
432450
dest='static_zoslib_gyp',
@@ -2006,6 +2024,19 @@ def make_bin_override():
20062024
configure_inspector(output)
20072025
configure_section_file(output)
20082026

2027+
# configure shareable builtins
2028+
output['variables']['node_builtin_shareable_builtins'] = []
2029+
for builtin in shareable_builtins:
2030+
builtin_id = 'node_shared_builtin_' + builtin.replace('/', '_') + '_path'
2031+
if getattr(options, builtin_id):
2032+
if options.with_intl == 'none':
2033+
option_name = '--shared-builtin-' + builtin + '-path'
2034+
error(option_name + ' is incompatible with --with-intl=none' )
2035+
else:
2036+
output['defines'] += [builtin_id.upper() + '=' + getattr(options, builtin_id)]
2037+
else:
2038+
output['variables']['node_builtin_shareable_builtins'] += [shareable_builtins[builtin]]
2039+
20092040
# Forward OSS-Fuzz settings
20102041
output['variables']['ossfuzz'] = b(options.ossfuzz)
20112042

‎doc/contributing/maintaining-dependencies.md‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# Maintaining Dependencies
2+
3+
Node.js depends on additional components beyond the Node.js code
4+
itself. These dependencies provide both native and JavaScript code
5+
and are built together with the code under the `src` and `lib`
6+
directories to create the Node.js binaries.
7+
8+
All dependencies are located within the `deps` directory.
9+
10+
Any code which meets one or more of these conditions should
11+
be managed as a dependency:
12+
13+
* originates in an upstream project and is maintained
14+
in that upstream project.
15+
* is not built from the `preferred form of the work for
16+
making modifications to it` (see
17+
[GNU GPL v2, section 3.](https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html)
18+
when `make node` is run. A good example is
19+
WASM code generated from C (the preferred form).
20+
Typically generation is only supported on a subset of platforms, needs
21+
additional tools, and is pre-built outside of the `make node`
22+
step and then committed as a WASM binary in the directory
23+
for the dependency under the `deps` directory.
24+
25+
By default all dependencies are bundled into the Node.js
26+
binary, however, `configure` options should be available to
27+
use an externalized version at runtime when:
28+
29+
* the dependency provides native code and is available as
30+
a shared library in one or more of the common Node.js
31+
distributions.
32+
* the dependency provides JavaScript and is not built
33+
from the `preferred form of the work for making modifications
34+
to it` when `make node` is run.
35+
36+
Many distributions use externalized dependencies for one or
37+
more of these reasons:
38+
39+
1. They have a requirement to build everything that they ship
40+
from the `preferred form of the work for making
41+
modifications to it`. This means that they need to
42+
replace any pre-built components (for example WASM
43+
binaries) with an equivalent that they have built.
44+
2. They manage the dependency separately as it is used by
45+
more applications than just Node.js. Linking against
46+
a shared library allows them to manage updates and
47+
CVE fixes against the library instead of having to
48+
patch all of the individual applications.
49+
3. They have a system wide configuration for the
50+
dependency that all applications should respect.
51+
52+
## Supporting externalized dependencies with native code.
53+
54+
Support for externalized dependencies with native code for which a
55+
shared library is available can added by:
56+
57+
* adding options to `configure.py`. These are added to the
58+
shared\_optgroup and include an options to:
59+
* enable use of a shared library
60+
* set the name of the shared library
61+
* set the path to the directory with the includes for the
62+
shared library
63+
* set the path to where to find the shared library at
64+
runtime
65+
* add a call to configure\_library() to `configure.py` for the
66+
library at the end of list of existing configure\_library() calls.
67+
If there are additional libraries that are required it is
68+
possible to list more than one with the `pkgname` option.
69+
* in `node.gypi` guard the build for the dependency
70+
with `node_shared_depname` so that is is only built if
71+
the dependency is being bundled into Node.js itself. For example:
72+
73+
```text
74+
[ 'node_shared_brotli=="false"', {
75+
'dependencies': [ 'deps/brotli/brotli.gyp:brotli' ],
76+
}],
77+
```
78+
79+
## Supporting externalizable dependencies with JavaScript codeIZA
80+
81+
Support for an externalizable dependency with JavaScript code
82+
can be added by:
83+
84+
* adding an entry to the `sharable_builtins` map in
85+
`configure.py`. The path should correspond to the file
86+
within the deps directory that is normally bundled into
87+
Node.js. For example `deps/cjs-module-lexer/lexer.js`.
88+
This will add a new option for building with that dependency
89+
externalized. After adding the entry you can see
90+
the new option by running `./configure --help`.
91+
92+
* adding a call to `AddExternalizedBuiltin` to the constructor
93+
for BuildinLoader in `src/node_builtins.cc` for the
94+
dependency using the `NODE_SHARED_BUILTLIN` #define generated for
95+
the dependency. After running `./configure` with the new
96+
option you can find the #define in `config.gypi`. You can cut and
97+
paste one of the existing entries and then update to match the
98+
inport name for the dependency and the #define generated.
99+
100+
## Supporting non-externalized dependencies with JavaScript code
101+
102+
If the dependency consists of JavaScript in the
103+
`preferred form of the work for making modifications to it`, it
104+
can be added as a non-externalizable dependency. In this case
105+
simply add the path to the JavaScript file in the `deps_files`
106+
list in the `node.gyp` file.

‎node.gyp‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -50,9 +50,7 @@
5050
'deps/v8/tools/tickprocessor-driver.mjs',
5151
'deps/acorn/acorn/dist/acorn.js',
5252
'deps/acorn/acorn-walk/dist/walk.js',
53-
'deps/cjs-module-lexer/lexer.js',
54-
'deps/cjs-module-lexer/dist/lexer.js',
55-
'deps/undici/undici.js',
53+
'<@(node_builtin_shareable_builtins)',
5654
],
5755
'node_mksnapshot_exec': '<(PRODUCT_DIR)/<(EXECUTABLE_PREFIX)node_mksnapshot<(EXECUTABLE_SUFFIX)',
5856
'mkcodecache_exec': '<(PRODUCT_DIR)/<(EXECUTABLE_PREFIX)mkcodecache<(EXECUTABLE_SUFFIX)',

‎src/node_builtins.cc‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
#include "node_builtins.h"
2+
#include "node_builtins_env.h"
23
#include "debug_utils-inl.h"
34
#include "env-inl.h"
45
#include "node_external_reference.h"
@@ -23,6 +24,24 @@ BuiltinLoader BuiltinLoader::instance_;
2324

2425
BuiltinLoader::BuiltinLoader() : config_(GetConfig()) {
2526
LoadJavaScriptSource();
27+
#if defined(NODE_HAVE_I18N_SUPPORT)
28+
#ifdef NODE_SHARED_BUILTIN_CJS_MODULE_LEXER_LEXER_PATH
29+
AddExternalizedBuiltin(
30+
"internal/deps/cjs-module-lexer/lexer",
31+
STRINGIFY(NODE_SHARED_BUILTIN_CJS_MODULE_LEXER_LEXER_PATH));
32+
#endif // NODE_SHARED_BUILTIN_CJS_MODULE_LEXER_LEXER_PATH
33+
34+
#ifdef NODE_SHARED_BUILTIN_CJS_MODULE_LEXER_DIST_LEXER_PATH
35+
AddExternalizedBuiltin(
36+
"internal/deps/cjs-module-lexer/dist/lexer",
37+
STRINGIFY(NODE_SHARED_BUILTIN_CJS_MODULE_LEXER_DIST_LEXER_PATH));
38+
#endif // NODE_SHARED_BUILTIN_CJS_MODULE_LEXER_DIST_LEXER_PATH
39+
40+
#ifdef NODE_SHARED_BUILTIN_UNDICI_UNDICI_PATH
41+
AddExternalizedBuiltin("internal/deps/undici/undici",
42+
STRINGIFY(NODE_SHARED_BUILTIN_UNDICI_UNDICI_PATH));
43+
#endif // NODE_SHARED_BUILTIN_UNDICI_UNDICI_PATH
44+
#endif // NODE_HAVE_I18N_SUPPORT
2645
}
2746

2847
BuiltinLoader* BuiltinLoader::GetInstance() {
@@ -236,6 +255,29 @@ MaybeLocal<String> BuiltinLoader::LoadBuiltinSource(Isolate* isolate,
236255
#endif // NODE_BUILTIN_MODULES_PATH
237256
}
238257

258+
#if defined(NODE_HAVE_I18N_SUPPORT)
259+
void BuiltinLoader::AddExternalizedBuiltin(const char* id,
260+
const char* filename) {
261+
std::string source;
262+
int r = ReadFileSync(&source, filename);
263+
if (r != 0) {
264+
fprintf(
265+
stderr, "Cannot load externalized builtin: \"%s:%s\".\n", id, filename);
266+
ABORT();
267+
return;
268+
}
269+
270+
icu::UnicodeString utf16 = icu::UnicodeString::fromUTF8(
271+
icu::StringPiece(source.data(), source.length()));
272+
auto source_utf16 = std::make_unique<icu::UnicodeString>(utf16);
273+
BuiltinEnv::Add(id,
274+
UnionBytes(reinterpret_cast<const uint16_t*>((*source_utf16).getBuffer()),
275+
utf16.length()));
276+
// keep source bytes for builtin alive while BuiltinLoader exists
277+
GetInstance()->externalized_source_bytes_.push_back(std::move(source_utf16));
278+
}
279+
#endif // NODE_HAVE_I18N_SUPPORT
280+
239281
// Returns Local<Function> of the compiled module if return_code_cache
240282
// is false (we are only compiling the function).
241283
// Otherwise return a Local<Object> containing the cache.

‎src/node_builtins.h‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,14 @@
33

44
#if defined(NODE_WANT_INTERNALS) && NODE_WANT_INTERNALS
55

6+
#include <list>
67
#include <map>
78
#include <memory>
89
#include <set>
910
#include <string>
11+
#if defined(NODE_HAVE_I18N_SUPPORT)
12+
#include <unicode/unistr.h>
13+
#endif // NODE_HAVE_I18N_SUPPORT
1014
#include "node_mutex.h"
1115
#include "node_union_bytes.h"
1216
#include "v8.h"
@@ -76,11 +80,18 @@ class NODE_EXTERN_PRIVATE BuiltinLoader {
7680
const char* id,
7781
Result* result);
7882

83+
#if defined(NODE_HAVE_I18N_SUPPORT)
84+
static void AddExternalizedBuiltin(const char* id, const char* filename);
85+
#endif // NODE_HAVE_I18N_SUPPORT
86+
7987
static BuiltinLoader instance_;
8088
BuiltinCategories builtin_categories_;
8189
BuiltinSourceMap source_;
8290
BuiltinCodeCacheMap code_cache_;
8391
UnionBytes config_;
92+
#if defined(NODE_HAVE_I18N_SUPPORT)
93+
std::list<std::unique_ptr<icu::UnicodeString>> externalized_source_bytes_;
94+
#endif // NODE_HAVE_I18N_SUPPORT
8495

8596
// Used to synchronize access to the code cache map
8697
Mutex code_cache_mutex_;

0 commit comments

Comments
 (0)
Back | FazBrowse Home | New Git URL