| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
|
cc @nodejs/performance @nodejs/fs @nodejs/cpp-reviewers |
Sorry, something went wrong.
|
cc @nodejs/tsc this pull-request is now a semver-major pull-request due to the following changes, and will not be ported to previous versions if we don't make an exception:
I appreciate if you could share your comments and thoughts on this. |
Sorry, something went wrong.
|
I'm okay with those changes as semver-major. I would be -1 on any sort of exception. |
Sorry, something went wrong.
|
I'm not sure, can't we re-map the thrown errors to the existing codes (e.g. in a catch, even if it's hard/degrates performance in C++ land (why?))? Wouldn't that only hurt the performance in the error case which isn't the interesting one? I'm wondering if our users for APIs like cpSync that are less performance sensitive (since they're blocking) don't care more breaking changes than a perf gain. (Just to be extra explicit - I'm not blocking this PR) |
Sorry, something went wrong.
|
Wait actually reading the code I think I understand since std::filesystem doesn't distinguish between those cases. |
Sorry, something went wrong.
These specific errors can only be caught if we traverse the filesystem one by one, and make the necessary copy operations. Since we don't do it, there is no way to know if the EEXIST was thrown from a symlink in src, or a symlink in destination that looks into a path in destination folder. The test cases are same, but they throw different errors.
We can catch it, but the problem is, we don't know which file is causing this error. |
Sorry, something went wrong.
|
Thanks, I'm +1 regarding the change and the breakage since I doubt people using cpSync are checking error codes this granularly. We may want to be prudent and trigger a CITGM run but I don't expect it'll find much. (It should still be semver major though) |
Sorry, something went wrong.
@benjamingr Actually, there is a std::filesystem::filesystem_error which I'm not sure how to catch it without try/catch. https://en.cppreference.com/w/cpp/filesystem/filesystem_error |
Sorry, something went wrong.
I love systems engineering. We can't use try/catch because we are passing -no-exceptions flag to GCC. ../../src/node_file.cc:2264:3: error: cannot use 'try' with exceptions disabled
try {
^
1 error generated.
|
Sorry, something went wrong.
|
That's for the overload that doesn't contain the error_code - and has the same info as the error_code so if it isn't there it doesn't distinguish between those cases. (Also CI should tell us that std::filesystem::copy gives these errors in a cross-platform friendly way) |
Sorry, something went wrong.
Even with that overloaded function, the following code does not compile due to disabled exception handling @benjamingr try {
std::filesystem::copy(src_path, dest_path, options);
} catch (const std::filesystem::filesystem_error& error) {
if (error.code() == std::errc::file_exists) {
std::string message = "File already exists";
return THROW_ERR_FS_CP_EEXIST(env, message.c_str());
}
std::string message = "Unhandled error " +
std::to_string(error_code.value()) + ": " +
error_code.message();
return THROW_ERR_FS_CP_EINVAL(env, message.c_str());
}
|
Sorry, something went wrong.
|
@anonrig you misunderstand me, sorry - I mean filesystem::copy has several overloads. The one you used in this PR already takes an error code reference here). If you do that, it doesn't throw and returns the error code through the parameter instead. If you remove that parameter it will throw. (There are valid important reasons why exceptions are disabled in the codebase, but feel free to disable that for testing) |
Sorry, something went wrong.
Yes, unfortunately std::error_code does not have path() property, but std::filesystem::filesystem_error does, and with exception handling disabled, it's not possible to get those paths, and make this a non-semver major pull-request. |
Sorry, something went wrong.
There was a problem hiding this comment.
This is excellent. Im in favor of this remaining a major change. Great work!
Sorry, something went wrong.
There was a problem hiding this comment.
lgtm
Sorry, something went wrong.
There was a problem hiding this comment.
LGTM with green CI. Since the implementation was adjusted, benchmark rerun would be appreciated.
Sorry, something went wrong.
There was a problem hiding this comment.
| code: 'ERR_FS_CP_EEXIST' | |
| code: 'ERR_FS_CP_EINVAL' |
AFAICT if symlink in src points to location in dest, cp will throw EINVAL due to presumably attempting of copying directory to its subdirectory. This might be a bug in the subdirectory checking implementation (isSrcSubdir(resolvedSrc, resolvedDest)).
Directory structure before copying:
├── copy_15
│ └── link -> node/test/.tmp.0/copy_16
├── copy_16
│ └── link -> node/test/.tmp.0/copy_16Directory structure after performing regular cp -r node/test/.tmp.0/copy_15 node/test/.tmp.0/copy_16:
├── copy_15
│ └── link -> node/test/.tmp.0/copy_16
├── copy_16
│ ├── copy_15
│ │ └── link -> node/test/.tmp.0/copy_16
│ └── link -> node/test/.tmp.0/copy_16It's not really related to this particular PR, but i guess creating cyclic symlink like this should be allowed.
Sorry, something went wrong.
There was a problem hiding this comment.
I think that this is nice enough. More detailed errors could be provided as part of a follow-up PR.
Sorry, something went wrong.
@LiviaMedeiros benchmark ci: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1572/ |
Sorry, something went wrong.
Sorry, something went wrong.
There was a problem hiding this comment.
I'm good with these changes as semver major. -1 on back porting.
Sorry, something went wrong.
|
I'm a little afraid of having a branch with two completely separate implementations of the feature depending on the options passed by the user. This LGTM if the end goal is to get rid of the JS one. |
Sorry, something went wrong.
| copy_options options = copy_options::copy_symlinks; | ||
|
|
||
| // When true timestamps from src will be preserved. | ||
| if (preserveTimestamps) options |= copy_options::create_hard_links; |
There was a problem hiding this comment.
is this correct?
afaict, this will create hard links in the destination directory to the original files. so, while it will appear upon inspecting the filesystem that the files have been copied and the timestamps have been preserved, the files will not have actually been copied, just linked as directory entries in the filesystem to the same file.
this means if you change any of the files in the source directory subsequently, the "files" in the destination will also change. i haven't looked at existing code in detail but am pretty sure this is not how it works with this flag.
i am also guessing without checking that this will only work if source and destination are on same filesystem? 🤔
if i am indeed correct, i guess simplest thing to do for now would be to use the slow path if "preserveTimestamps" is set?
from a quick google, i'm not sure there is an efficient way to copy and preserve timestamps without having to touch each file in a syscall and it just doesn't seem that there is an option rn in std::filesystem to emulate the current behaviour. happy to be wrong though - not a C++ expert.
https://en.cppreference.com/w/cpp/filesystem/copy_options
https://en.wikipedia.org/wiki/Hard_link
Sorry, something went wrong.
There was a problem hiding this comment.
i put some benches using the test/fixtures/copy directory from node.js and the same options as above here if anyone wants to try them out - should be easy to get them working on linux or macos.
https://github.com/just-js/lo-bench/blob/main/fs/README.md
some things to note:
on linux core i5, using /dev/shm to reduce filesystem/disk overhead
on macos m1 mini, using a ram disk
also, for the same workload:
so, without testing with other directory structures and sizes/depths but assuming we see similar results, we could assert the following:
another couple of things to note
let me know if i got anything wrong - i ran through this pretty quickly, but it should be easy to reproduce those results using the link above. 🙏
Sorry, something went wrong.
There was a problem hiding this comment.
Sounds good.
For now, I'll split this pull-request into multiple changes, and try to optimize the existing implementation before moving it into full C++.
Sorry, something went wrong.
There was a problem hiding this comment.
Here's a different approach: #53614
Sorry, something went wrong.
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
I'm extremely open to suggestions on improving the benchmarks, but with the current state, here are the results:
local benchmarks
macOS M1 Max
benchmark ci
benchmark ci: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1572/