| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
There was a problem hiding this comment.
Only very lightly reviewed. No opinion on whether this is a good or bad change.
Sorry, something went wrong.
There was a problem hiding this comment.
Missing word after 'such as.'
Sorry, something went wrong.
There was a problem hiding this comment.
var = re.sub(r'[\-./]', '_', name)
Sorry, something went wrong.
There was a problem hiding this comment.
'use strict';
Sorry, something went wrong.
There was a problem hiding this comment.
Operator should go on the previous line.
Sorry, something went wrong.
|
👍 strongly for merging this. My only comment would be if there might not be a maintenance burden being created by this approach that might be mitigated with enumeration of the exports at load time, provided we can ensure we don't expose the wrong things (perhaps filtering _... properties etc). Would be interested to hear thoughts on this approach. |
Sorry, something went wrong.
There was a problem hiding this comment.
I've included my comments... hope you don't mind all the questions :)
Sorry, something went wrong.
There was a problem hiding this comment.
Why do we need two forms of builtin? Can the builtin/esm not entirely replace the builtin/cjs? What is the use case for retaining builtin/cjs?
Sorry, something went wrong.
There was a problem hiding this comment.
When does this case ever happen?
Sorry, something went wrong.
There was a problem hiding this comment.
when the esm wrapper imports the actual cjs it looks like node:assert requesting assert, and thats the only time the builtin cjs should get used
Sorry, something went wrong.
There was a problem hiding this comment.
Is the assumption that this file is run periodically to update the lib folder? Is there a way to make this more automated? Why would you think this level of automation is preferable to runtime enumeration?
Sorry, something went wrong.
There was a problem hiding this comment.
hopefully it never needs to be used again, but i still included it since it might be useful at some point, maybe during a refactor or something in the far future
Sorry, something went wrong.
|
@guybedford i chose this over runtime enumeration because of the results in my last pr where i attempted forward evaluation but i guess if we can guarantee our libs won't have errors on evaluation it should be fine? i don't think our public api changes that much for it to be a real burden. this also lays groundwork for writing core parts of node in esm, although i don't know if that matters much to people. it should be noted that if we switch to runtime enumerating with out of order evaluation then the current issues i'm having with tests will also go away, and it will get rid of people who make loaders worrying about resolving both kinds builtins, which is definitely a confusing process. |
Sorry, something went wrong.
|
@devsnek thanks for clarifying. Personally I think the runtime creation would be better as it would avoid the need for the new builtin interpretation mode, remove the maintenance burden of maintaining the wrappers (generated or not), and it would effectively be exactly the same algorithm to build up the exports, saving an extra file load anyway to do that. |
Sorry, something went wrong.
|
@guybedford i just know there was significant opposition to out-of-order evaluation in my previous pr, but maybe @bmeck its fine in this case? |
Sorry, something went wrong.
|
Personally I don't seen an issue with considering core modules "preevaluated" (from the esm loader perspective). It's user dependencies with circular references and errors that we have to worry about for that issue. |
Sorry, something went wrong.
|
I don't like that the exports can go out of sync but am not blocking that at this point. I'd love it if the backing object could stay in sync for a variety of reasons, but all approaches I've tested are problematic for performance. |
Sorry, something went wrong.
|
@devsnek could you make the ESM facades eagerly populate even when required so that they stay the true primordial form of the exports? |
Sorry, something went wrong.
|
@bmeck i'm not sure what you mean by "true primordial form" but my method would basically be: loaders.set('builtin', async (url) => {
const module = InternalModule.require(url.slice(5));
const properties = Object.getOwnPropertyDescriptors(module);
const keys = ['default'];
for (const [name, prop] of Object.entries(properties)) {
if (!prop.enumerable || !prop.value)
continue;
if (/(^_)|(_$)/.test(name))
continue;
keys.push(name);
}
return createDynamicModule(keys, url, (reflect) => {
reflect.exports.default.set(module);
for (const key of keys)
reflect.exports[key].set(module[key]);
});
}); |
Sorry, something went wrong.
|
@devsnek I just want to be sure that if you mutate fs it does not mutate the named export value depending on timing. const fs = require('fs');
fs.readFile = () => {}import {readFile} from 'fs';
// should never be that noop function (even if this file is loaded after the one above) |
Sorry, something went wrong.
|
@bmeck i don't know of any way to guarantee at all, even requiring every builtin before user code runs and keeping a cache would still allow user mutation of the exports object. |
Sorry, something went wrong.
|
@devsnek the same sort of technique we use to eagerly inject CJS modules into the loader registry for this should work here for the core modules I think? |
Sorry, something went wrong.
|
(I know it's not pretty, but it provides the invariants) |
Sorry, something went wrong.
There was a problem hiding this comment.
Just a thought... could we not devise a mechanism for generating these automatically during the build so we do not need to keep them manually in sync? The surface area of core modules is fixed at time of build.
Sorry, something went wrong.
There was a problem hiding this comment.
@jasnell we could somehow with static parsing maybe, but various things like getter/setters would need to be excluded.
Sorry, something went wrong.
There was a problem hiding this comment.
This appears to be missing the new strict export.
Sorry, something went wrong.
There was a problem hiding this comment.
it might not have been in my branch when i generated these, but since i'm changing this to generate the exports at runtime it shouldn't be an issue. (stay tuned!! 😄)
Sorry, something went wrong.
There was a problem hiding this comment.
We should decide if we really want to export pure aliases or take the opportunity to begin limiting access to those
Sorry, something went wrong.
There was a problem hiding this comment.
can you expand on the reason to limit access to things?
Sorry, something went wrong.
There was a problem hiding this comment.
Should we export deprecated modules at all?
Sorry, something went wrong.
There was a problem hiding this comment.
+0 to removing them, not blocking on it
Sorry, something went wrong.
|
New approach looks good. |
Sorry, something went wrong.
There was a problem hiding this comment.
lgtm
Sorry, something went wrong.
There was a problem hiding this comment.
what is "setter" and "getter"? Property descriptors have set and get.
Sorry, something went wrong.
|
This feature has got go-ahead in the modules meeting when we discussed it.
I believe we’re currently blocked on a proxy issue.
…On Tue, 10 Apr 2018 at 03:11, Gus Caplan ***@***.***> wrote:
@BridgeAR <https://github.com/BridgeAR> modules team isn't merging
anything atm while we talk about use cases etc
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
<#18131 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAkiyhTmZ4kVypWXPPcZwqX-fEuv3_M3ks5tnAasgaJpZM4RdLYy>
.
|
Sorry, something went wrong.
|
@guybedford there is no technical limitation here, if nothing bureaucratic is holding this back i'd love to finish it up |
Sorry, something went wrong.
|
Definitely, let's get this going then. I think we've solved all the main issues here with this mutatable default approach. I was just thinking, it would be nice to do some simple benchmarks of this for existing code, and perhaps comparing that to having setters on module.exports for reassignments instead of a proxy, just in terms of retaining property access performance on builtins. If proxy is fast enough at this point then it shouldn't be a problem though. |
Sorry, something went wrong.
|
Can we remove the filter for setters too since there is no technical block on them. |
Sorry, something went wrong.
|
@devsnek is this pending further review then? |
Sorry, something went wrong.
|
@guybedford its been a very long time since i looked at this so i kinda need to go through the whole thing again and i was on vacation all last week. i hope to do something with this in a few days. |
Sorry, something went wrong.
|
I'd be pretty bummed if setters were still filtered. It complicates and diminishes something intended as an ecosystem olive branch by making things more complex for tools and users. For example, projects like webpack or babel 7 that want to allow limited .mjs use will now have to maintain a list (not great) or runtime check (complicates things) to throw errors when a user requests one of these filtered names. If instead, getters/setters were handled via the process established for APIs then tools wouldn't have to bake in more rules here and changes to the API would shake out naturally in this compat feature without extra work. |
Sorry, something went wrong.
|
@jdalton don't worry I'll address that, I'm just trying to figure out this implementation a bit more... I'm not very happy with it at the moment |
Sorry, something went wrong.
|
i rewrote this a bit if people wanna take a look. i consolidated stuff and made it a bit more streamlined and easier to reason with |
Sorry, something went wrong.
There was a problem hiding this comment.
I’d think it’d be sufficient to show that fs.readFileSync === readFileSync?
Sorry, something went wrong.
There was a problem hiding this comment.
Should this use hasOwnProperty instead of in?
Sorry, something went wrong.
There was a problem hiding this comment.
This won’t be robust against Function.toString.call, fwiw
Sorry, something went wrong.
There was a problem hiding this comment.
that's just left over from some other stuff i was doing, i'll remove it
Sorry, something went wrong.
There was a problem hiding this comment.
Similarly, this doesn’t have to actually have a function - it just needs something to ===
Sorry, something went wrong.
There was a problem hiding this comment.
because this implementation wraps functions, i feel like testing return value is worthwhile
Sorry, something went wrong.
There was a problem hiding this comment.
👆 Why is events special cased?
Sorry, something went wrong.
There was a problem hiding this comment.
i don't remember anymore but at some point everyone was fine with this :/
Sorry, something went wrong.
There was a problem hiding this comment.
Can we side on removing unnecessary filtering please.
Sorry, something went wrong.
There was a problem hiding this comment.
👆 In the if (valueDescriptor && condition you can make it if (valueDescriptor) { then inside the block capture the result of Reflect.defineProperty(target, prop, descriptor), call update and return the captured result.
Sorry, something went wrong.
There was a problem hiding this comment.
👆 By returning false you're making the CJS export objects non-delete able. Instead you could allow the operation to happen, capturing the result of Reflect.deleteProperty(target, prop), calling update, then returning the captured result. If a property is deleted its updated value is undefined or whatever is exposed on its prototype.
Sorry, something went wrong.
There was a problem hiding this comment.
calling update
calling it with what exactly? the property is gone if we let it get deleted
Sorry, something went wrong.
There was a problem hiding this comment.
calling it with what exactly? the property is gone if we let it get deleted
Whatever the value of target[prop] that's remaining.
Sorry, something went wrong.
There was a problem hiding this comment.
I take a slightly different approach to wrapping. Instead of creating a new wrap function, I proxy the original.
Sorry, something went wrong.
There was a problem hiding this comment.
Is there a reason why you would prefer that?
Sorry, something went wrong.
There was a problem hiding this comment.
@TimothyGu Yes. I first avoid wrapping for the 99% case (plain or bound functions) and then only wrap with a proxy for the native case. For the 1% case (native method) wrapping with a proxy lets all other traps passthru to the original function while I just hook the apply trap.
Beyond the fact that less wrapping is good in general this is important to the esm loader because folks can opt for CJS named export support beyond buitlins. Avoiding the function/proxy wrap means more methods are === to other references a user might store.
Sorry, something went wrong.
There was a problem hiding this comment.
I'll have to re-audit our exports to make sure we only need to bind for native methods, but that does seem like a better case
Sorry, something went wrong.
There was a problem hiding this comment.
@jdalton your proxy unfortunately breaks in certain cases which i think could be a v8 bug so i'll keep using my wrap function for now
@nodejs/v8
> Reflect.apply(EventEmitter.call, EventEmitter, []) TypeError: Reflect.apply is not a function
Sorry, something went wrong.
There was a problem hiding this comment.
@jdalton i wasn't running any esm, it broke regular usage of event emitter. it seems like v8 doesn't enjoy Function.prototype.call and Reflect.apply together, this isn't just an issue with EventEmitter.
var EventEmitter = require('events');
// probably some ghetto es3 function extending EventEmitter
EventEmitter.call(...); i can special case the thisArg variable like value === Function.prototype.call but that feels nasty and the wrap works
Sorry, something went wrong.
There was a problem hiding this comment.
Dev error? Proxies should work.
Can you show how you attempted proxies. Might be able to spot the issue.
In my implementation
EventEmitter.call({})works. The .call is proxy wrapped.
I have smth like this for the handler (poke around the implementation here)
wrapper = new Proxy(value, {
apply(funcTarget, thisArg, args) {
if (thisArg === proxy ||
thisArg === entry.esmNamespace) {
thisArg = target
}
return Reflect.apply(value, thisArg, args)
}
})When EventEmitter.call({}) is called the thisArg === proxy condition is met because var EventEmitter is the events module module.exports proxy. The thisArg is set to the original target (the unwrapped module.exports of events). This results in a successful invocation.
Sorry, something went wrong.
There was a problem hiding this comment.
@devsnek Can you find an isolated reproduction of the bug? The following seems to be working just fine here:
$ node
> const { EventEmitter } = events
undefined
> Reflect.apply(EventEmitter.call, EventEmitter, [])
TypeError: Cannot set property 'domain' of undefined
at EventEmitter.init (domain.js:401:15)
at EventEmitter (events.js:27:21)
Sorry, something went wrong.
There was a problem hiding this comment.
@jdalton i copied yours in, except for using regular proxies instead of your OwnProxy, which i doubt makes any difference in this case
Sorry, something went wrong.
There was a problem hiding this comment.
ok i just came back to this and it seems like the proxy works now so i'm going to just assume i need more sleep... i'll push in a few minutes after some more testing
Sorry, something went wrong.
There was a problem hiding this comment.
I don't think we should make a distinction between "our proxies" versus "their proxies".
We should either disable proxy showing by default, or just show the proxy.
Sorry, something went wrong.
There was a problem hiding this comment.
@TimothyGu It's cosmetic and can totally be tackled in a follow-up PR. The idea from earlier in the thread was that folks thought it would be less-good displaying the Proxy prefix when inspecting builtin module exports in the repl.
Sorry, something went wrong.
There was a problem hiding this comment.
I'd like to argue in the other way: because it is cosmetic, this change can be done in a follow-up PR. Changes should be atomic and focused on one topic at a time.
Sorry, something went wrong.
There was a problem hiding this comment.
I'd like to argue in the other way: because it is cosmetic, this change can be done in a follow-up PR.
We're on the same page. I was saying that the masking of builtin exports could be done in a follow-up PR instead of this one 😁
Sorry, something went wrong.
There was a problem hiding this comment.
Oops 😛
Sorry, something went wrong.
There was a problem hiding this comment.
Good progress, but a bit more work is still needed.
Also, how much does this slow down require()ing an internal module (and Node.js startup) from CJS with the --experimental-modules flag turned on? We could get this in with a performance hit, but we need to be able to quantify that and know what to fix in the future.
Sorry, something went wrong.
There was a problem hiding this comment.
Last , target is unneeded.
Sorry, something went wrong.
There was a problem hiding this comment.
What exactly does this do? A comment would help with what exactly this truncates. Also we generally use .slice() (substr is part of Annex B).
Sorry, something went wrong.
There was a problem hiding this comment.
No need to import URL at all in Node.js v10.x :)
Sorry, something went wrong.
There was a problem hiding this comment.
you never know where this might be backported to
Sorry, something went wrong.
There was a problem hiding this comment.
Core code rarely use the functional array methods. Let's make this imperative instead.
this.namespace = [];
for (const key of Object.getOwnPropertyNames(this.exports)) {
const desc = Object.getOwnPropertyDescriptor(this.exports, key);
if (!desc.enumerable)
continue;
namespace.push(key);
}
Sorry, something went wrong.
There was a problem hiding this comment.
mfw thats what i was using before 😢 will change back
Sorry, something went wrong.
There was a problem hiding this comment.
For consistency with defineProperty, use update(prop, Reflect.get(target, prop)). (This behavior is observable through getters/setters.)
Sorry, something went wrong.
There was a problem hiding this comment.
set hook has a fourth receiver argument that you are not handling.
You need to at least make sure to forward the receiver argument to Reflect.set.
Sorry, something went wrong.
There was a problem hiding this comment.
You need to handle receiver here as well.
Sorry, something went wrong.
There was a problem hiding this comment.
You might need to add this handling for getOwnPropertyDescriptor as well.
Sorry, something went wrong.
There was a problem hiding this comment.
Still need to test:
Sorry, something went wrong.
There was a problem hiding this comment.
Instead of getOwnPropertyNames+enumerable check you could use Object.keys
Sorry, something went wrong.
|
average startup time with child process + loop + console.log(perf_hooks.performance.nodeTiming) + shameless rounding: without proxy (1000 runs) 79.71845515598729 i didn't bother testing time of an individual require because it can only happen at max once. its also worth nothing this increased time should be more than handled by snapshots, whenever we finish those. |
Sorry, something went wrong.
There was a problem hiding this comment.
Besides non-functions and bound-functions you can also skip non-native functions.
I have an inference method here for reference. (a v8 helper for this would be rad++)
Sorry, something went wrong.
There was a problem hiding this comment.
we can't skip native functions. for instance if you do module.exports = new Map() and we don't wrap the exports then Map.prototype.* will have improper receivers and throw
Sorry, something went wrong.
There was a problem hiding this comment.
we can't skip native functions.
I know, I'm saying skip non-native.
Sorry, something went wrong.
There was a problem hiding this comment.
so the only functions we bind then are unbound native functions? what about member functions written in js
Sorry, something went wrong.
There was a problem hiding this comment.
so the only functions we bind then are unbound native functions? what about member functions written in js
We aren't binding functions. The wrapper juggles the thisArg around for the one, maybe two, cases that cause native methods grief but beyond that we forward the thisArg along. So it makes sense to only wrap the methods that need the thisArg juggling in the first place (native methods).
Sorry, something went wrong.
There was a problem hiding this comment.
You might end up needing a thisArg === nsObj check.
A test for calling a native method on the namespace object would cover it.
Sorry, something went wrong.
|
alright so this is the 400th comment in this thread and i'm starting to consistently get the github unicorn when loading this. at the moment it works and this pr seems to be ready. that being said, i need some approvals on this. i would love to land this by the end of next week. |
Sorry, something went wrong.
|
👍 As an initial landing of an experimental feature I think it's great. Every time I look at this implementation I find something in my own to improve! There are some things to dry-up and tweak here but they can be tackled in follow-up PRs. |
Sorry, something went wrong.
provide named exports for all builtin libraries so that the libraries may be
imported in a nicer way for esm users: `import { readFile } from 'fs'`
instead of importing the entire namespace, `import fs from 'fs'`, and
calling `fs.readFile`. the default export is left as the entire
namespace (module.exports)
|
And, given that it's not a semver change (I think...), I believe there
should be no problem to land it?
…On Sun, Apr 29, 2018 at 2:26 AM John-David Dalton ***@***.***> wrote:
👍 As an initial landing of an experimental feature I think it's great.
Every time I look at this implementation I find something in my own to
improve! There are some things to dry-up and tweak here but they can be
tackled in follow-up PRs.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#18131 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAYnRN0X4rkV7THOoKvx1uxnqEepfnzRks5ttPq_gaJpZM4RdLYy>
.
|
Sorry, something went wrong.
| this.exportKeys = Object.keys(this.exports); | ||
|
|
||
| const update = (property, value) => { | ||
| if (this.reflect !== undefined && this.exportKeys.includes(property)) |
There was a problem hiding this comment.
is it ok that delete Array.prototype.includes can break this code?
If not, you could copy Array.prototype.includes to be an own property on this.exportKeys, perhaps?
(same question on has/get/set on collection instances)
Sorry, something went wrong.
There was a problem hiding this comment.
Sorry, something went wrong.
There was a problem hiding this comment.
Fair point, just wanted to call it out :-)
Sorry, something went wrong.
| NativeModule.require(id); | ||
| const module = NativeModule.getCached(id); | ||
| return createDynamicModule( | ||
| [...module.exportKeys, 'default'], url, (reflect) => { |
There was a problem hiding this comment.
i'm not sure if the ordering matters here at all - is default always last?
https://tc39.github.io/ecma262/#sec-modulenamespacecreate step 7 suggests that all export keys, including "default" if present, should be alphabetically sorted. (i do see at least one test that validates the ordering, but i'm not sure if that test covers this code or not)
Sorry, something went wrong.
There was a problem hiding this comment.
the order doesn't matter there actually as it just gets injected into a generated source text
Sorry, something went wrong.
|
My greatest concern here is the potential 10% performance slowdown for NodeJS app startup having all core modules as proxies in CommonJS code, as Gus provided in some numbers at #18131 (comment). It may turn out that creating setter-based core modules could be an alternative to the proxy approach that is also faster, so I do think this would still be worth seriously considering, or at least comparing for performance. The benefit of a proxy over just a setter is supporting dynamic properties and object.defineProperty configuration cases. But dynamic properties are already not supported, so perhaps benefits may be worth the loss of configuration hooks. |
Sorry, something went wrong.
|
@guybedford as long as the experimental flag is around i'd rather take it as an opportunity to experiment with the behaviour rather than perf optimisation. come time to ship it we can always make it more performant if needed. i'm also exceedingly hopeful that we will finish up snapshots by that time and then we won't have to worry this at all |
Sorry, something went wrong.
|
There is an important distinction between experimentation that doesn't
affect legacy at all, and experimentation that creates a performance
regression for existing code. I really don't think a 10% reduction in
NodeJS startup time for existing codebases is a trivial matter to push
through. Yes lets move fast with the modules work, but let's not slow down
existing NodeJS apps in the process.
I'm also starting to get the unicorn for this page consistently - it is
probably worth closing this PR and opening a new one at this point I think
to reset discussion for review.
…On Sun, Apr 29, 2018 at 3:28 PM Gus Caplan ***@***.***> wrote:
@guybedford <https://github.com/guybedford> as long as the experimental
flag is around i'd rather take it as an opportunity to experiment with the
behaviour rather than perf optimisation. come time to ship it we can always
make it more performant if needed. i'm also exceedingly hopeful that we
will finish up snapshots by that time and then we won't have to worry this
at all
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#18131 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAkiykOyOSII3TlhpYZvz4DcudoJs4Sjks5ttcAKgaJpZM4RdLYy>
.
|
Sorry, something went wrong.
|
@guybedford this code doesn't run unless the flag is given, and yes i would agree a new pr is probably a good idea |
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
provide named exports for all builtin libraries so that the libraries may be
imported in a nicer way for esm users: import { readFile } from 'fs'
instead of importing the entire namespace, import fs from 'fs', and
calling fs.readFile
Checklist