FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

src: add an option to make compile cache portable · nodejs/node@94422e8 · GitHub

/ node Public

Commit 94422e8

Browse files
Aditi-1400
authored
src: add an option to make compile cache portable
Adds an option (NODE_COMPILE_CACHE_PORTABLE) for the built-in compile cache to encode the hashes with relative file paths. On enabling the option, the source directory along with cache directory can be bundled and moved, and the cache continues to work. When enabled, paths encoded in hash are relative to compile cache directory. PR-URL: #58797 Fixes: #58755 Refs: #52696 Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>
1 parent 22a864a commit 94422e8

15 files changed

Lines changed: 448 additions & 15 deletions

‎doc/api/cli.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3341,6 +3341,11 @@ added: v22.1.0
33413341
Enable the [module compile cache][] for the Node.js instance. See the documentation of
33423342
[module compile cache][] for details.
33433343

3344+
### `NODE_COMPILE_CACHE_PORTABLE=1`
3345+
3346+
When set to 1, the [module compile cache][] can be reused across different directory
3347+
locations as long as the module layout relative to the cache directory remains the same.
3348+
33443349
### `NODE_DEBUG=module[,…]`
33453350

33463351
<!-- YAML

‎doc/api/module.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -399,6 +399,28 @@ the [`NODE_COMPILE_CACHE=dir`][] environment variable if it's set, or defaults
399399
to `path.join(os.tmpdir(), 'node-compile-cache')` otherwise. To locate the compile cache
400400
directory used by a running Node.js instance, use [`module.getCompileCacheDir()`][].
401401
402+
By default, caches are invalidated when the absolute paths of the modules being
403+
cached are changed. To keep the cache working after moving the
404+
project directory, enable portable compile cache. This allows previously compiled
405+
modules to be reused across different directory locations as long as the layout relative
406+
to the cache directory remains the same. This would be done on a best-effort basis. If
407+
Node.js cannot compute the location of a module relative to the cache directory, the module
408+
will not be cached.
409+
410+
There are two ways to enable the portable mode:
411+
412+
1. Using the portable option in module.enableCompileCache():
413+
414+
```js
415+
// Non-portable cache (default): cache breaks if project is moved
416+
module.enableCompileCache({ path: '/path/to/cache/storage/dir' });
417+
418+
// Portable cache: cache works after the project is moved
419+
module.enableCompileCache({ path: '/path/to/cache/storage/dir', portable: true });
420+
```
421+
422+
2. Setting the environment variable: [`NODE_COMPILE_CACHE_PORTABLE=1`][]
423+
402424
Currently when using the compile cache with [V8 JavaScript code coverage][], the
403425
coverage being collected by V8 may be less precise in functions that are
404426
deserialized from the code cache. It's recommended to turn this off when
@@ -1789,6 +1811,7 @@ returned object contains the following keys:
17891811
[`--import`]: cli.md#--importmodule
17901812
[`--require`]: cli.md#-r---require-module
17911813
[`NODE_COMPILE_CACHE=dir`]: cli.md#node_compile_cachedir
1814+
[`NODE_COMPILE_CACHE_PORTABLE=1`]: cli.md#node_compile_cache_portable1
17921815
[`NODE_DISABLE_COMPILE_CACHE=1`]: cli.md#node_disable_compile_cache1
17931816
[`NODE_V8_COVERAGE=dir`]: cli.md#node_v8_coveragedir
17941817
[`SourceMap`]: #class-modulesourcemap

‎doc/node.1‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -725,6 +725,13 @@ Enable the
725725
.Sy module compile cache
726726
for the Node.js instance.
727727
.
728+
.It Ev NODE_COMPILE_CACHE_PORTABLE
729+
When set to '1' or 'true', the
730+
.Sy module compile cache
731+
will be hit as long as the location of the modules relative to the cache directory remain
732+
consistent. This can be used in conjunction with .Ev NODE_COMPILE_CACHE
733+
to enable portable on-disk caching.
734+
.
728735
.It Ev NODE_DEBUG Ar modules...
729736
Comma-separated list of core modules that should print debug information.
730737
.

‎lib/internal/modules/helpers.js‎

Lines changed: 19 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -402,18 +402,31 @@ function stringify(body) {
402402
}
403403

404404
/**
405-
* Enable on-disk compiled cache for all user modules being complied in the current Node.js instance
405+
* Enable on-disk compiled cache for all user modules being compiled in the current Node.js instance
406406
* after this method is called.
407-
* If cacheDir is undefined, defaults to the NODE_MODULE_CACHE environment variable.
408-
* If NODE_MODULE_CACHE isn't set, default to path.join(os.tmpdir(), 'node-compile-cache').
409-
* @param {string|undefined} cacheDir
407+
* This method accepts either:
408+
* - A string `cacheDir`: the path to the cache directory.
409+
* - An options object `{path?: string, portable?: boolean}`:
410+
* - `path`: A string path to the cache directory.
411+
* - `portable`: If `portable` is true, the cache directory will be considered relative. Defaults to false.
412+
* If cache path is undefined, it defaults to the NODE_MODULE_CACHE environment variable.
413+
* If `NODE_MODULE_CACHE` isn't set, it defaults to `path.join(os.tmpdir(), 'node-compile-cache')`.
414+
* @param {string | { path?: string, portable?: boolean } | undefined} options
410415
* @returns {{status: number, message?: string, directory?: string}}
411416
*/
412-
function enableCompileCache(cacheDir) {
417+
function enableCompileCache(options) {
418+
let cacheDir;
419+
let portable = false;
420+
421+
if (typeof options === 'object' && options !== null) {
422+
({ path: cacheDir, portable = false } = options);
423+
} else {
424+
cacheDir = options;
425+
}
413426
if (cacheDir === undefined) {
414427
cacheDir = join(lazyTmpdir(), 'node-compile-cache');
415428
}
416-
const nativeResult = _enableCompileCache(cacheDir);
429+
const nativeResult = _enableCompileCache(cacheDir, portable);
417430
const result = { status: nativeResult[0] };
418431
if (nativeResult[1]) {
419432
result.message = nativeResult[1];

‎src/compile_cache.cc‎

Lines changed: 50 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,9 @@
1313
#include <unistd.h> // getuid
1414
#endif
1515

16+
#ifdef _WIN32
17+
#include <windows.h>
18+
#endif
1619
namespace node {
1720

1821
using v8::Function;
@@ -223,13 +226,52 @@ void CompileCacheHandler::ReadCacheFile(CompileCacheEntry* entry) {
223226
Debug(" success, size=%d\n", total_read);
224227
}
225228

229+
static std::string GetRelativePath(std::string_view path,
230+
std::string_view base) {
231+
// On Windows, the native encoding is UTF-16, so we need to convert
232+
// the paths to wide strings before using std::filesystem::path.
233+
// On other platforms, std::filesystem::path can handle UTF-8 directly.
234+
#ifdef _WIN32
235+
std::filesystem::path module_path(
236+
ConvertToWideString(std::string(path), CP_UTF8));
237+
std::filesystem::path base_path(
238+
ConvertToWideString(std::string(base), CP_UTF8));
239+
#else
240+
std::filesystem::path module_path(path);
241+
std::filesystem::path base_path(base);
242+
#endif
243+
std::filesystem::path relative = module_path.lexically_relative(base_path);
244+
auto u8str = relative.u8string();
245+
return std::string(u8str.begin(), u8str.end());
246+
}
247+
226248
CompileCacheEntry* CompileCacheHandler::GetOrInsert(Local<String> code,
227249
Local<String> filename,
228250
CachedCodeType type) {
229251
DCHECK(!compile_cache_dir_.empty());
230252

253+
Environment* env = Environment::GetCurrent(isolate_->GetCurrentContext());
231254
Utf8Value filename_utf8(isolate_, filename);
232-
uint32_t key = GetCacheKey(filename_utf8.ToStringView(), type);
255+
std::string file_path = filename_utf8.ToString();
256+
// If the portable cache is enabled and it seems possible to compute the
257+
// relative position from an absolute path, we use the relative position
258+
// in the cache key.
259+
if (portable_ == EnableOption::PORTABLE && IsAbsoluteFilePath(file_path)) {
260+
// Normalize the path to ensure it is consistent.
261+
std::string normalized_file_path = NormalizeFileURLOrPath(env, file_path);
262+
if (normalized_file_path.empty()) {
263+
return nullptr;
264+
}
265+
std::string relative_path =
266+
GetRelativePath(normalized_file_path, normalized_compile_cache_dir_);
267+
if (!relative_path.empty()) {
268+
file_path = relative_path;
269+
Debug("[compile cache] using relative path %s from %s\n",
270+
file_path.c_str(),
271+
compile_cache_dir_.c_str());
272+
}
273+
}
274+
uint32_t key = GetCacheKey(file_path, type);
233275

234276
// TODO(joyeecheung): don't encode this again into UTF8. If we read the
235277
// UTF8 content on disk as raw buffer (from the JS layer, while watching out
@@ -500,7 +542,8 @@ CompileCacheHandler::CompileCacheHandler(Environment* env)
500542
// - $NODE_VERSION-$ARCH-$CACHE_DATA_VERSION_TAG-$UID
501543
// - $FILENAME_AND_MODULE_TYPE_HASH.cache: a hash of filename + module type
502544
CompileCacheEnableResult CompileCacheHandler::Enable(Environment* env,
503-
const std::string& dir) {
545+
const std::string& dir,
546+
EnableOption option) {
504547
std::string cache_tag = GetCacheVersionTag();
505548
std::string absolute_cache_dir_base = PathResolve(env, {dir});
506549
std::string cache_dir_with_tag =
@@ -548,6 +591,11 @@ CompileCacheEnableResult CompileCacheHandler::Enable(Environment* env,
548591

549592
result.cache_directory = absolute_cache_dir_base;
550593
compile_cache_dir_ = cache_dir_with_tag;
594+
portable_ = option;
595+
if (option == EnableOption::PORTABLE) {
596+
normalized_compile_cache_dir_ =
597+
NormalizeFileURLOrPath(env, compile_cache_dir_);
598+
}
551599
result.status = CompileCacheEnableStatus::ENABLED;
552600
return result;
553601
}

‎src/compile_cache.h‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,10 +62,14 @@ struct CompileCacheEnableResult {
6262
std::string message; // Set in case of failure.
6363
};
6464

65+
enum class EnableOption : uint8_t { DEFAULT, PORTABLE };
66+
6567
class CompileCacheHandler {
6668
public:
6769
explicit CompileCacheHandler(Environment* env);
68-
CompileCacheEnableResult Enable(Environment* env, const std::string& dir);
70+
CompileCacheEnableResult Enable(Environment* env,
71+
const std::string& dir,
72+
EnableOption option = EnableOption::DEFAULT);
6973

7074
void Persist();
7175

@@ -103,6 +107,8 @@ class CompileCacheHandler {
103107
bool is_debug_ = false;
104108

105109
std::string compile_cache_dir_;
110+
std::string normalized_compile_cache_dir_;
111+
EnableOption portable_ = EnableOption::DEFAULT;
106112
std::unordered_map<uint32_t, std::unique_ptr<CompileCacheEntry>>
107113
compiler_cache_store_;
108114
};

‎src/env.cc‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1128,11 +1128,21 @@ void Environment::InitializeCompileCache() {
11281128
dir_from_env.empty()) {
11291129
return;
11301130
}
1131-
EnableCompileCache(dir_from_env);
1131+
std::string portable_env;
1132+
bool portable = credentials::SafeGetenv(
1133+
"NODE_COMPILE_CACHE_PORTABLE", &portable_env, this) &&
1134+
!portable_env.empty() && portable_env == "1";
1135+
if (portable) {
1136+
Debug(this,
1137+
DebugCategory::COMPILE_CACHE,
1138+
"[compile cache] using relative path\n");
1139+
}
1140+
EnableCompileCache(dir_from_env,
1141+
portable ? EnableOption::PORTABLE : EnableOption::DEFAULT);
11321142
}
11331143

11341144
CompileCacheEnableResult Environment::EnableCompileCache(
1135-
const std::string& cache_dir) {
1145+
const std::string& cache_dir, EnableOption option) {
11361146
CompileCacheEnableResult result;
11371147
std::string disable_env;
11381148
if (credentials::SafeGetenv(
@@ -1149,7 +1159,7 @@ CompileCacheEnableResult Environment::EnableCompileCache(
11491159
if (!compile_cache_handler_) {
11501160
std::unique_ptr<CompileCacheHandler> handler =
11511161
std::make_unique<CompileCacheHandler>(this);
1152-
result = handler->Enable(this, cache_dir);
1162+
result = handler->Enable(this, cache_dir, option);
11531163
if (result.status == CompileCacheEnableStatus::ENABLED) {
11541164
compile_cache_handler_ = std::move(handler);
11551165
AtExit(

‎src/env.h‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1020,7 +1020,8 @@ class Environment final : public MemoryRetainer {
10201020
void InitializeCompileCache();
10211021
// Enable built-in compile cache if it has not yet been enabled.
10221022
// The cache will be persisted to disk on exit.
1023-
CompileCacheEnableResult EnableCompileCache(const std::string& cache_dir);
1023+
CompileCacheEnableResult EnableCompileCache(const std::string& cache_dir,
1024+
EnableOption option);
10241025
void FlushCompileCache();
10251026

10261027
void RunAndClearNativeImmediates(bool only_refed = false);

‎src/node_modules.cc‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -501,8 +501,14 @@ void EnableCompileCache(const FunctionCallbackInfo<Value>& args) {
501501
THROW_ERR_INVALID_ARG_TYPE(env, "cacheDir should be a string");
502502
return;
503503
}
504+
505+
EnableOption option = EnableOption::DEFAULT;
506+
if (args.Length() > 1 && args[1]->IsTrue()) {
507+
option = EnableOption::PORTABLE;
508+
}
509+
504510
Utf8Value value(isolate, args[0]);
505-
CompileCacheEnableResult result = env->EnableCompileCache(*value);
511+
CompileCacheEnableResult result = env->EnableCompileCache(*value, option);
506512
Local<Value> values[3];
507513
values[0] = v8::Integer::New(isolate, static_cast<uint8_t>(result.status));
508514
if (ToV8Value(context, result.message).ToLocal(&values[1]) &&

‎src/path.cc‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
11
#include "path.h"
22
#include <string>
33
#include <vector>
4+
#include "ada.h"
45
#include "env-inl.h"
56
#include "node_internals.h"
7+
#include "node_url.h"
68

79
namespace node {
810

@@ -88,6 +90,10 @@ std::string NormalizeString(const std::string_view path,
8890
}
8991

9092
#ifdef _WIN32
93+
constexpr bool IsWindowsDriveLetter(const std::string_view path) noexcept {
94+
return path.size() > 2 && IsWindowsDeviceRoot(path[0]) &&
95+
(path[1] == ':' && (path[2] == '/' || path[2] == '\\'));
96+
}
9197
constexpr bool IsWindowsDeviceRoot(const char c) noexcept {
9298
return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z');
9399
}
@@ -333,4 +339,44 @@ void FromNamespacedPath(std::string* path) {
333339
#endif
334340
}
335341

342+
// Check if a path looks like an absolute path or file URL.
343+
bool IsAbsoluteFilePath(std::string_view path) {
344+
if (path.rfind("file://", 0) == 0) {
345+
return true;
346+
}
347+
#ifdef _WIN32
348+
if (path.size() > 0 && path[0] == '\\') return true;
349+
if (IsWindowsDriveLetter(path)) return true;
350+
#endif
351+
if (path.size() > 0 && path[0] == '/') return true;
352+
return false;
353+
}
354+
355+
// Normalizes paths by resolving file URLs and converting to a consistent
356+
// format with forward slashes.
357+
std::string NormalizeFileURLOrPath(Environment* env, std::string_view path) {
358+
std::string normalized_string(path);
359+
constexpr std::string_view file_scheme = "file://";
360+
if (normalized_string.rfind(file_scheme, 0) == 0) {
361+
auto out = ada::parse<ada::url_aggregator>(normalized_string);
362+
auto file_path = url::FileURLToPath(env, *out);
363+
if (!file_path.has_value()) {
364+
return std::string();
365+
}
366+
normalized_string = file_path.value();
367+
}
368+
normalized_string = NormalizeString(normalized_string, false, "/");
369+
#ifdef _WIN32
370+
if (IsWindowsDriveLetter(normalized_string)) {
371+
normalized_string[0] = ToLower(normalized_string[0]);
372+
}
373+
for (char& c : normalized_string) {
374+
if (c == '\\') {
375+
c = '/';
376+
}
377+
}
378+
#endif
379+
return normalized_string;
380+
}
381+
336382
} // namespace node

0 commit comments

Comments
 (0)
Back | FazBrowse Home | New Git URL