FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

module: add a read-only mode to the compile cache · nodejs/node@97a437e · GitHub

/ node Public

Commit 97a437e

Browse files
codebytere
authored and
aduh95
committed
module: add a read-only mode to the compile cache
A compile cache generated ahead of time and shipped inside an application package should only ever be read: the package may be immutable or covered by an integrity check, and a cache directory that appears at run time would be a surprise. Add readOnly to module.enableCompileCache() and NODE_COMPILE_CACHE_READONLY=1: existing entries are loaded as before, nothing is serialized or persisted, flushCompileCache() is a no-op, and the cache directory is used as found rather than created, so enabling against a missing directory fails instead of making one. Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com> PR-URL: #65302 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: René <contact.9a5d6388@renegade334.me.uk>
1 parent d0447c9 commit 97a437e

9 files changed

Lines changed: 189 additions & 26 deletions

File tree

‎doc/api/cli.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3708,6 +3708,15 @@ When set to 1, the [module compile cache][] can be reused across different direc
37083708
locations as long as the module layout relative to the cache directory remains the same,
37093709
and by any user (the cache subdirectory is not suffixed with the creating user's uid).
37103710

3711+
### `NODE_COMPILE_CACHE_READONLY=1`
3712+
3713+
<!-- YAML
3714+
added: REPLACEME
3715+
-->
3716+
3717+
When set to 1, the [module compile cache][] only reads existing entries from
3718+
its directory: nothing is written to it and it is not created if missing.
3719+
37113720
### `NODE_DEBUG=module[,…]`
37123721

37133722
<!-- YAML

‎doc/api/module.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -427,6 +427,15 @@ There are two ways to enable the portable mode:
427427
428428
2. Setting the environment variable: [`NODE_COMPILE_CACHE_PORTABLE=1`][]
429429
430+
### Read-only compile cache
431+
432+
A cache that was generated ahead of time, for example at build time to be
433+
shipped inside an application package, can be enabled with `readOnly: true`
434+
(or [`NODE_COMPILE_CACHE_READONLY=1`][]). Node.js then loads whatever entries
435+
the directory holds and never writes to it: modules without a usable entry are
436+
compiled as usual but not persisted, [`module.flushCompileCache()`][] is a
437+
no-op, and the directory is not created if it is missing.
438+
430439
### Limitations of the compile cache
431440
432441
Currently when using the compile cache with [V8 JavaScript code coverage][], the
@@ -498,6 +507,9 @@ The following constants are returned as the `status` field in the object returne
498507
<!-- YAML
499508
added: v22.8.0
500509
changes:
510+
- version: REPLACEME
511+
pr-url: https://github.com/nodejs/node/pull/65302
512+
description: Add the `readOnly` option.
501513
- version: v25.4.0
502514
pr-url: https://github.com/nodejs/node/pull/60971
503515
description: This feature is no longer experimental.
@@ -522,6 +534,11 @@ changes:
522534
the cache can be reused even if the project directory is moved. This is a best-effort
523535
feature. If not specified, it will depend on whether the environment variable
524536
[`NODE_COMPILE_CACHE_PORTABLE=1`][] is set.
537+
* `readOnly` {boolean} Optional. If `true`, existing cache entries in `directory` are
538+
used but nothing is ever written to it, and the directory is not created when it does
539+
not exist (enabling then fails). Meant for caches generated ahead of time and shipped
540+
with an application. If not specified, it will depend on whether the environment
541+
variable [`NODE_COMPILE_CACHE_READONLY=1`][] is set.
525542
* Returns: {Object}
526543
* `status` {integer} One of the [`module.constants.compileCacheStatus`][]
527544
* `message` {string|undefined} If Node.js cannot enable the compile cache, this contains
@@ -2047,6 +2064,7 @@ returned object contains the following keys:
20472064
[`--require`]: cli.md#-r---require-module
20482065
[`NODE_COMPILE_CACHE=dir`]: cli.md#node_compile_cachedir
20492066
[`NODE_COMPILE_CACHE_PORTABLE=1`]: cli.md#node_compile_cache_portable1
2067+
[`NODE_COMPILE_CACHE_READONLY=1`]: cli.md#node_compile_cache_readonly1
20502068
[`NODE_DISABLE_COMPILE_CACHE=1`]: cli.md#node_disable_compile_cache1
20512069
[`NODE_V8_COVERAGE=dir`]: cli.md#node_v8_coveragedir
20522070
[`SourceMap`]: #class-modulesourcemap

‎doc/node.1‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1862,6 +1862,10 @@ When set to 1, the module compile cache can be reused across different directory
18621862
locations as long as the module layout relative to the cache directory remains the same,
18631863
and by any user (the cache subdirectory is not suffixed with the creating user's uid).
18641864
.
1865+
.It Ev NODE_COMPILE_CACHE_READONLY Ar 1
1866+
When set to 1, the module compile cache only reads existing entries from
1867+
its directory: nothing is written to it and it is not created if missing.
1868+
.
18651869
.It Ev NODE_DEBUG Ar module[,…]
18661870
\fB','\fR-separated list of core modules that should print debug information.
18671871
.

‎lib/internal/modules/helpers.js‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -466,21 +466,25 @@ function stringify(body) {
466466
* after this method is called.
467467
* This method accepts either:
468468
* - A string: path to the cache directory.
469-
* - An options object `{directory?: string, portable?: boolean}`:
469+
* - An options object `{directory?: string, portable?: boolean, readOnly?: boolean}`:
470470
* - `directory`: A string path to the cache directory.
471471
* - `portable`: If `portable` is true, the cache directory will be considered relative.
472472
* Defaults to `NODE_COMPILE_CACHE_PORTABLE === '1'`.
473+
* - `readOnly`: If `readOnly` is true, existing cache entries are used but nothing is
474+
* written, and the cache directory is not created. Defaults to
475+
* `NODE_COMPILE_CACHE_READONLY === '1'`.
473476
* If cache directory is undefined, it defaults to the `NODE_COMPILE_CACHE` environment variable.
474477
* If `NODE_COMPILE_CACHE` isn't set, it defaults to `path.join(os.tmpdir(), 'node-compile-cache')`.
475-
* @param {string | { directory?: string, portable?: boolean } | undefined} options
478+
* @param {string | { directory?: string, portable?: boolean, readOnly?: boolean } | undefined} options
476479
* @returns {{status: number, message?: string, directory?: string}}
477480
*/
478481
function enableCompileCache(options) {
479482
let portable;
483+
let readOnly;
480484
let directory;
481485

482486
if (typeof options === 'object' && options !== null) {
483-
({ directory, portable } = options);
487+
({ directory, portable, readOnly } = options);
484488
} else {
485489
directory = options;
486490
}
@@ -490,7 +494,10 @@ function enableCompileCache(options) {
490494
if (portable === undefined) {
491495
portable = process.env.NODE_COMPILE_CACHE_PORTABLE === '1';
492496
}
493-
const nativeResult = _enableCompileCache(directory, portable);
497+
if (readOnly === undefined) {
498+
readOnly = process.env.NODE_COMPILE_CACHE_READONLY === '1';
499+
}
500+
const nativeResult = _enableCompileCache(directory, portable, readOnly);
494501
const result = { status: nativeResult[0] };
495502
if (nativeResult[1]) {
496503
result.message = nativeResult[1];

‎src/compile_cache.cc‎

Lines changed: 52 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -257,7 +257,8 @@ CompileCacheEntry* CompileCacheHandler::GetOrInsert(Local<String> code,
257257
// If the portable cache is enabled and it seems possible to compute the
258258
// relative position from an absolute path, we use the relative position
259259
// in the cache key.
260-
if (portable_ == EnableOption::PORTABLE && IsAbsoluteFilePath(file_path)) {
260+
if (HasOption(portable_, EnableOption::PORTABLE) &&
261+
IsAbsoluteFilePath(file_path)) {
261262
// Normalize the path to ensure it is consistent.
262263
std::string normalized_file_path = NormalizeFileURLOrPath(env, file_path);
263264
if (normalized_file_path.empty()) {
@@ -332,6 +333,10 @@ void CompileCacheHandler::MaybeSaveImpl(CompileCacheEntry* entry,
332333
Debug("keeping the in-memory entry\n");
333334
return;
334335
}
336+
if (read_only_) {
337+
Debug("read-only, not serializing\n");
338+
return;
339+
}
335340
Debug("%s the in-memory entry\n",
336341
entry->cache == nullptr ? "initializing" : "refreshing");
337342

@@ -356,6 +361,9 @@ void CompileCacheHandler::MaybeSave(CompileCacheEntry* entry,
356361

357362
void CompileCacheHandler::MaybeSave(CompileCacheEntry* entry,
358363
std::string_view transpiled) {
364+
if (read_only_) {
365+
return;
366+
}
359367
CHECK(entry->type == CachedCodeType::kStrippedTypeScript);
360368
Debug("[compile cache] saving transpilation cache for %s %s\n",
361369
entry->type_name(),
@@ -388,6 +396,10 @@ void CompileCacheHandler::MaybeSave(CompileCacheEntry* entry,
388396
*/
389397
void CompileCacheHandler::Persist() {
390398
DCHECK(!compile_cache_dir_.empty());
399+
if (read_only_) {
400+
Debug("[compile cache] read-only, skipping persistence\n");
401+
return;
402+
}
391403

392404
// TODO(joyeecheung): do this using a separate event loop to utilize the
393405
// libuv thread pool and do the file system operations concurrently.
@@ -554,10 +566,11 @@ CompileCacheEnableResult CompileCacheHandler::Enable(Environment* env,
554566
cache_tag,
555567
cache_dir_with_tag);
556568

557-
if (!env->permission()->is_granted(
558-
env,
559-
permission::PermissionScope::kFileSystemWrite,
560-
cache_dir_with_tag)) [[unlikely]] {
569+
const bool read_only = HasOption(option, EnableOption::READ_ONLY);
570+
if (!read_only && !env->permission()->is_granted(
571+
env,
572+
permission::PermissionScope::kFileSystemWrite,
573+
cache_dir_with_tag)) [[unlikely]] {
561574
result.message = "Skipping compile cache because write permission for " +
562575
cache_dir_with_tag + " is not granted";
563576
result.status = CompileCacheEnableStatus::FAILED;
@@ -574,25 +587,46 @@ CompileCacheEnableResult CompileCacheHandler::Enable(Environment* env,
574587
return result;
575588
}
576589

577-
fs::FSReqWrapSync req_wrap;
578-
int err = fs::MKDirpSync(
579-
nullptr, &(req_wrap.req), cache_dir_with_tag, 0777, nullptr);
580-
if (is_debug_) {
581-
Debug("[compile cache] creating cache directory %s...%s\n",
590+
if (read_only) {
591+
// A read-only cache is used as found and never created: without the
592+
// directory there is nothing to read.
593+
uv_fs_t stat_req;
594+
int err =
595+
uv_fs_stat(nullptr, &stat_req, cache_dir_with_tag.c_str(), nullptr);
596+
// libuv normalizes st_mode across platforms; MSVC has no S_ISDIR macro.
597+
bool is_dir = err == 0 && (stat_req.statbuf.st_mode & S_IFMT) == S_IFDIR;
598+
uv_fs_req_cleanup(&stat_req);
599+
Debug("[compile cache] read-only cache directory %s...%s\n",
582600
cache_dir_with_tag,
583-
err < 0 ? uv_strerror(err) : "success");
584-
}
585-
if (err != 0 && err != UV_EEXIST) {
586-
result.message =
587-
"Cannot create cache directory: " + std::string(uv_strerror(err));
588-
result.status = CompileCacheEnableStatus::FAILED;
589-
return result;
601+
is_dir ? "found" : "not found");
602+
if (!is_dir) {
603+
result.message =
604+
"Cache directory does not exist (read-only): " + cache_dir_with_tag;
605+
result.status = CompileCacheEnableStatus::FAILED;
606+
return result;
607+
}
608+
} else {
609+
fs::FSReqWrapSync req_wrap;
610+
int err = fs::MKDirpSync(
611+
nullptr, &(req_wrap.req), cache_dir_with_tag, 0777, nullptr);
612+
if (is_debug_) {
613+
Debug("[compile cache] creating cache directory %s...%s\n",
614+
cache_dir_with_tag,
615+
err < 0 ? uv_strerror(err) : "success");
616+
}
617+
if (err != 0 && err != UV_EEXIST) {
618+
result.message =
619+
"Cannot create cache directory: " + std::string(uv_strerror(err));
620+
result.status = CompileCacheEnableStatus::FAILED;
621+
return result;
622+
}
590623
}
591624

592625
result.cache_directory = absolute_cache_dir_base;
593626
compile_cache_dir_ = cache_dir_with_tag;
594627
portable_ = option;
595-
if (option == EnableOption::PORTABLE) {
628+
read_only_ = read_only;
629+
if (HasOption(option, EnableOption::PORTABLE)) {
596630
normalized_compile_cache_dir_ =
597631
NormalizeFileURLOrPath(env, compile_cache_dir_);
598632
}

‎src/compile_cache.h‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,22 @@ struct CompileCacheEnableResult {
6060
std::string message; // Set in case of failure.
6161
};
6262

63-
enum class EnableOption : uint8_t { DEFAULT, PORTABLE };
63+
enum class EnableOption : uint8_t {
64+
DEFAULT = 0,
65+
PORTABLE = 1 << 0,
66+
// Only read existing cache entries: nothing is compiled into the in-memory
67+
// store for persisting, nothing is written to disk, and the cache directory
68+
// is not created if it does not exist.
69+
READ_ONLY = 1 << 1,
70+
};
71+
72+
inline constexpr EnableOption operator|(EnableOption a, EnableOption b) {
73+
return static_cast<EnableOption>(static_cast<uint8_t>(a) |
74+
static_cast<uint8_t>(b));
75+
}
76+
inline constexpr bool HasOption(EnableOption value, EnableOption flag) {
77+
return (static_cast<uint8_t>(value) & static_cast<uint8_t>(flag)) != 0;
78+
}
6479

6580
class CompileCacheHandler {
6681
public:
@@ -82,6 +97,7 @@ class CompileCacheHandler {
8297
bool rejected);
8398
void MaybeSave(CompileCacheEntry* entry, std::string_view transpiled);
8499
std::string_view cache_dir() { return compile_cache_dir_; }
100+
bool read_only() const { return read_only_; }
85101

86102
private:
87103
void ReadCacheFile(CompileCacheEntry* entry);
@@ -107,6 +123,7 @@ class CompileCacheHandler {
107123
std::string compile_cache_dir_;
108124
std::string normalized_compile_cache_dir_;
109125
EnableOption portable_ = EnableOption::DEFAULT;
126+
bool read_only_ = false;
110127
std::unordered_map<uint32_t, std::unique_ptr<CompileCacheEntry>>
111128
compiler_cache_store_;
112129
};

‎src/env.cc‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1195,8 +1195,14 @@ void Environment::InitializeCompileCache() {
11951195
DebugCategory::COMPILE_CACHE,
11961196
"[compile cache] using relative path\n");
11971197
}
1198-
EnableCompileCache(dir_from_env,
1199-
portable ? EnableOption::PORTABLE : EnableOption::DEFAULT);
1198+
std::string read_only_env;
1199+
bool read_only = credentials::SafeGetenv(
1200+
"NODE_COMPILE_CACHE_READONLY", &read_only_env, this) &&
1201+
read_only_env == "1";
1202+
EnableOption option = EnableOption::DEFAULT;
1203+
if (portable) option = option | EnableOption::PORTABLE;
1204+
if (read_only) option = option | EnableOption::READ_ONLY;
1205+
EnableCompileCache(dir_from_env, option);
12001206
}
12011207

12021208
CompileCacheEnableResult Environment::EnableCompileCache(

‎src/node_modules.cc‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -524,7 +524,10 @@ void EnableCompileCache(const FunctionCallbackInfo<Value>& args) {
524524

525525
EnableOption option = EnableOption::DEFAULT;
526526
if (args.Length() > 1 && args[1]->IsTrue()) {
527-
option = EnableOption::PORTABLE;
527+
option = option | EnableOption::PORTABLE;
528+
}
529+
if (args.Length() > 2 && args[2]->IsTrue()) {
530+
option = option | EnableOption::READ_ONLY;
528531
}
529532

530533
Utf8Value value(isolate, args[0]);

‎test/parallel/test-compile-cache-api-readonly.js‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
'use strict';
2+
3+
// This tests module.enableCompileCache({ directory, readOnly: true }): existing
4+
// entries are read, nothing is written, and a missing directory is not created.
5+
6+
const common = require('../common');
7+
const { spawnSyncAndAssert } = require('../common/child_process');
8+
const assert = require('assert');
9+
const fs = require('fs');
10+
const path = require('path');
11+
const tmpdir = require('../common/tmpdir');
12+
const fixtures = require('../common/fixtures');
13+
14+
tmpdir.refresh();
15+
const wrapper = fixtures.path('compile-cache-wrapper-options.js');
16+
const target = path.join(tmpdir.path, 'target.js');
17+
fs.writeFileSync(target, 'module.exports = 1;');
18+
const other = path.join(tmpdir.path, 'other.js');
19+
fs.writeFileSync(other, 'module.exports = 2;');
20+
const directory = path.join(tmpdir.path, 'cache');
21+
const list = () => fs.readdirSync(directory, { recursive: true }).sort();
22+
const run = (options, extraEnv, requires, check) => spawnSyncAndAssert(
23+
process.execPath,
24+
[...requires.flatMap((r) => ['-r', r]), target],
25+
{
26+
env: {
27+
...process.env,
28+
NODE_DEBUG_NATIVE: 'COMPILE_CACHE',
29+
NODE_TEST_COMPILE_CACHE_OPTIONS: JSON.stringify(options),
30+
...extraEnv,
31+
},
32+
},
33+
check);
34+
35+
// Read-only against a directory that does not exist: enabling fails and
36+
// nothing is created.
37+
run({ directory, readOnly: true }, {}, [wrapper], {
38+
stderr: /read-only cache directory .*\.\.\.not found/,
39+
});
40+
assert(!fs.existsSync(directory));
41+
42+
// A normal run generates the cache for target.js.
43+
run({ directory }, {}, [wrapper], {
44+
stderr: /writing cache for .*target\.js.*success/,
45+
});
46+
const generated = list();
47+
assert.notStrictEqual(generated.length, 0);
48+
49+
// Read-only against it: target.js's entry is accepted, other.js is compiled
50+
// but not written, and persistence is skipped.
51+
run({ directory, readOnly: true }, {}, [wrapper, other], {
52+
stderr: common.mustCall((output) => {
53+
assert.match(output, /cache for .*target\.js was accepted/);
54+
assert.match(output, /read-only, skipping persistence/);
55+
assert.doesNotMatch(output, /writing cache for/);
56+
return true;
57+
}),
58+
});
59+
assert.deepStrictEqual(list(), generated);
60+
61+
// The environment variable form.
62+
run({ directory }, { NODE_COMPILE_CACHE_READONLY: '1' }, [wrapper, other], {
63+
stderr: /read-only, skipping persistence/,
64+
});
65+
assert.deepStrictEqual(list(), generated);

0 commit comments

Comments
 (0)
Back | FazBrowse Home | New Git URL