fs: optimize `fs.cpSync` js calls by anonrig · Pull Request #53614 · nodejs/node · GitHub

#53612 is required to do a benchmark

A different approach to #53541

cc @nodejs/performance @nodejs/cpp-reviewers @billywhizz @jasnell @nodejs/fs

Benchmark CI: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1573/

Small win on happy path, and probably high win on error path:

                           confidence improvement accuracy (*)   (**)   (***)
fs/bench-cpSync.js n=1            ***     21.96 %       ±6.85% ±9.14% ±11.97%
fs/bench-cpSync.js n=100          ***     14.91 %       ±2.86% ±3.81%  ±4.97%
fs/bench-cpSync.js n=10000        ***      9.74 %       ±1.28% ±1.71%  ±2.23%

cc @nodejs/tsc @nodejs/fs appreciate your input on which path to take.

previous pr:

is a sem-ver major
introduces 2 different implementations (js only and cpp)
had limitations due to std::filesystem::copy

this pr:

is not a sem-ver major
optimizes all flags passed to cpSync
optimizes error cases as same as the previous pr
has a lot more optimization to do

lgtm

CI: https://ci.nodejs.org/job/node-test-pull-request/60175/

CI: https://ci.nodejs.org/job/node-test-pull-request/60191/

CI: https://ci.nodejs.org/job/node-test-pull-request/60190/

CI: https://ci.nodejs.org/job/node-test-pull-request/60257/

CI: https://ci.nodejs.org/job/node-test-pull-request/60274/

CI: https://ci.nodejs.org/job/node-test-pull-request/60277/

CI: https://ci.nodejs.org/job/node-test-pull-request/60366/

Windows compilation is failing:

C:\workspace\node-compile-windows-debug\node\deps\v8\src\base\bits.h(477,31): warning C4146: unary minus operator applied to unsigned type, result still unsigned [C:\workspace\node-compile-windows-debug\node\tools\v8_gypfiles\mksnapshot.vcxproj]
  embedded-file-writer.cc
  mksnapshot.cc
  embedded-empty.cc
  platform-embedded-file-writer-base.cc
  platform-embedded-file-writer-generic.cc
  platform-embedded-file-writer-win.cc
  platform-embedded-file-writer-aix.cc
  platform-embedded-file-writer-mac.cc
  snapshot-empty.cc
  static-roots-gen.cc
     Creating library ..\..\out\Debug\mksnapshot.lib and object ..\..\out\Debug\mksnapshot.exp
  mksnapshot.vcxproj -> ..\..\out\Debug\\mksnapshot.exe
  generating: "..\..\out\Debug\obj\v8_snapshot\/snapshot.cc" "..\..\out\Debug\obj\v8_snapshot\/embedded.S"
  Assembling ..\..\out\Debug\obj\v8_snapshot\\embedded.S...
  setup-isolate-deserialize.cc
C:\workspace\node-compile-windows-debug\node\deps\v8\src\base\bits.h(477,31): warning C4146: unary minus operator applied to unsigned type, result still unsigned [C:\workspace\node-compile-windows-debug\node\tools\v8_gypfiles\v8_snapshot.vcxproj]
  snapshot.cc
C:\workspace\node-compile-windows-debug\node\deps\v8\src\base\bits.h(477,31): warning C4146: unary minus operator applied to unsigned type, result still unsigned [C:\workspace\node-compile-windows-debug\node\tools\v8_gypfiles\v8_snapshot.vcxproj]
  v8_snapshot.vcxproj -> ..\..\out\Debug\lib\v8_snapshot.lib

@nodejs/platform-windows any idea why this build is failing?

Building locally, I got this:

C:\Users\vinic\Desktop\Projects\node\src\node_file.cc(3036,15): error C2664: 'void node::Environment::ThrowUVException(int,const char *,const char *,const char *,const char *)': cannot convert argument 4 from
 'const std::filesystem::path::value_type *' to 'const char *' [C:\Users\vinic\Desktop\Projects\node\libnode.vcxproj]

Building locally, I got this:

C:\Users\vinic\Desktop\Projects\node\src\node_file.cc(3036,15): error C2664: 'void node::Environment::ThrowUVException(int,const char *,const char *,const char *,const char *)': cannot convert argument 4 from
 'const std::filesystem::path::value_type *' to 'const char *' [C:\Users\vinic\Desktop\Projects\node\libnode.vcxproj]

This is also in the log from the CI. Based on the error message, this looks very similar to what we had in #53600 because std::filesystem::path::value_type is different on POSIX and Windows.

This is also in the log from the CI. Based on the error message, this looks very similar to what we had in #53600 because std::filesystem::path::value_type is different on POSIX and Windows.

It seems I forgot a src_path.c_str() there. Thank you! @StefanStojanovic @H4ad

CI: https://ci.nodejs.org/job/node-test-pull-request/60519/

CI: https://ci.nodejs.org/job/node-test-pull-request/60528/

Still LGTM

Landed in 88027e8

PR-URL: #53614 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

This looks wrong, the error code should be properly translated instead of being directly mapped to EEXIST, there are many other reasons why this could fail.

BufferValue converts the string to UTF8, on Windows this requires automatic conversion for char8_t to work which may not be supported by the version of MSVC we use, I think that may be the source of bugs like #54476

Actually no, this is simply broken on Windows, because this just gets returned in char. It needs to be returned as std::u8string/std::u8string_view instead.

…ths on Windows fs.cpSync({ recursive: true }) on Node 22 for Windows aborts the whole process (STATUS_STACK_BUFFER_OVERRUN / 0xC0000409, no JS exception) when the source path contains non-ASCII characters such as a CJK home directory (nodejs/node#54476, regression from nodejs/node#53614). The plugin declares engines "^22.19.0 || >=24.0.0", so Node 22 must work. Replace the two cpSync call sites with a per-entry recursive copy (mkdirSync + copyFileSync + utimesSync to preserve mtimes) and add a regression test using a CJK preset directory name.

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Jun 27, 2024

anonrig mentioned this pull request Jun 27, 2024

fs: improve cpSync performance #53541

Closed

anonrig force-pushed the optimize-cp-sync branch from 02e0883 to d7d4ae8 Compare June 27, 2024 18:44

anonrig requested a review from lemire June 27, 2024 18:44

anonrig force-pushed the optimize-cp-sync branch 2 times, most recently from bebf131 to 2d5fb50 Compare June 27, 2024 21:23

anonrig force-pushed the optimize-cp-sync branch from 2d5fb50 to 6a216bd Compare June 27, 2024 21:33

This comment was marked as outdated.

Sign in to view

mcollina approved these changes Jun 28, 2024

View reviewed changes

benjamingr approved these changes Jun 28, 2024

View reviewed changes

anonrig force-pushed the optimize-cp-sync branch from 6a216bd to 371d7a7 Compare July 11, 2024 18:54

anonrig force-pushed the optimize-cp-sync branch from 371d7a7 to e2c27ae Compare July 21, 2024 21:16

anonrig requested a review from jasnell July 21, 2024 21:17

This comment was marked as outdated.

Sign in to view

jasnell approved these changes Jul 21, 2024

View reviewed changes

anonrig added the author ready PRs that have at least one approval, no outstanding review comments, and a CI started. label Jul 21, 2024

fs: optimize fs.cpSync js calls

a590e84

anonrig force-pushed the optimize-cp-sync branch from e2c27ae to a590e84 Compare July 21, 2024 23:40

jasnell approved these changes Jul 22, 2024

View reviewed changes

anonrig added the commit-queue Add this label to land a pull request using GitHub Actions. label Jul 22, 2024

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Jul 22, 2024

nodejs-github-bot merged commit 88027e8 into nodejs:main Jul 22, 2024

targos pushed a commit that referenced this pull request Jul 30, 2024

fs: optimize fs.cpSync js calls

1b0d5d4

PR-URL: #53614 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

RafaelGSS mentioned this pull request Jul 30, 2024

v22.6.0 proposal #54123

Merged

RafaelGSS pushed a commit that referenced this pull request Aug 5, 2024

fs: optimize fs.cpSync js calls

5e03c17

PR-URL: #53614 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

joyeecheung reviewed Aug 22, 2024

View reviewed changes

joyeecheung mentioned this pull request Aug 22, 2024

v22.6.0 fs.cpSync crash #54476

Closed

markandrus mentioned this pull request Sep 11, 2024

[Bug?]: fsBinding.cpSyncCheckPaths error in Node 22 yarnpkg/berry#6488

Open

1 task

jazelly mentioned this pull request Sep 12, 2024

fs: translate error code properly in cpSync #54906

Merged

joyeecheung mentioned this pull request Sep 20, 2024

src: revert filesystem::path changes #55015

Merged

targos added the dont-land-on-v20.x label Sep 21, 2024

HAN102300 mentioned this pull request Aug 15, 2026

fix(liangshen): copy preset trees without fs.cpSync to survive CJK paths on Windows zhu1090093659/dsh-web#205

Merged

6 tasks

zhu1090093659 mentioned this pull request Aug 16, 2026

[Bug]: dsh-liangshen 触发 STATUS_STACK_BUFFER_OVERRUN (0xC0000409) 进程崩溃 zhu1090093659/dsh-web#302

Closed

1 task

+                                      ? std::filesystem::symlink_status(src_path, error_code)
+                                      : std::filesystem::status(src_path, error_code);
+                if (error_code) {
+                  return env->ThrowUVException(EEXIST, "lstat", nullptr, src.out());

+                ToNamespacedPath(env, &src);
+                THROW_IF_INSUFFICIENT_PERMISSIONS(
+                    env, permission::PermissionScope::kFileSystemRead, src.ToStringView());
+                auto src_path = std::filesystem::path(src.ToStringView());

fs: optimize `fs.cpSync` js calls by anonrig · Pull Request #53614 · nodejs/node · GitHub

Uh oh!

Uh oh!

fs: optimize fs.cpSync js calls - #53614

fs: optimize fs.cpSync js calls#53614nodejs-github-bot merged 1 commit intonodejs:mainnodejs/node:mainfrom anonrig:optimize-cp-syncanonrig/node:optimize-cp-syncCopy head branch name to clipboard

Conversation

anonrig commented Jun 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anonrig commented Jun 27, 2024

Uh oh!

This comment was marked as outdated.

mcollina left a comment

Choose a reason for hiding this comment

Uh oh!

nodejs-github-bot commented Jul 8, 2024

Uh oh!

nodejs-github-bot commented Jul 9, 2024

Uh oh!

nodejs-github-bot commented Jul 9, 2024

Uh oh!

nodejs-github-bot commented Jul 11, 2024

Uh oh!

nodejs-github-bot commented Jul 12, 2024

Uh oh!

nodejs-github-bot commented Jul 12, 2024

Uh oh!

nodejs-github-bot commented Jul 16, 2024

Uh oh!

aduh95 commented Jul 16, 2024

Uh oh!

anonrig commented Jul 17, 2024

Uh oh!

H4ad commented Jul 17, 2024

Uh oh!

StefanStojanovic commented Jul 17, 2024

Uh oh!

This comment was marked as outdated.

anonrig commented Jul 21, 2024

Uh oh!

nodejs-github-bot commented Jul 21, 2024

Uh oh!

nodejs-github-bot commented Jul 22, 2024

Uh oh!

jasnell left a comment

Choose a reason for hiding this comment

Uh oh!

nodejs-github-bot commented Jul 22, 2024

Uh oh!

joyeecheung Aug 22, 2024

Choose a reason for hiding this comment

Uh oh!

joyeecheung Aug 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joyeecheung Aug 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

fs: optimize fs.cpSync js calls#53614
nodejs-github-bot merged 1 commit into
nodejs:mainfrom
anonrig:optimize-cp-sync

anonrig commented Jun 27, 2024 •
edited
Loading

joyeecheung Aug 22, 2024 •
edited
Loading

joyeecheung Aug 22, 2024 •
edited
Loading