esm: remove CLI flag limitation to programmatic registration by JakobJingleheimer · Pull Request #48439 · nodejs/node · GitHub

This is a follow-up to #46826

We see 2 potential approaches here when we detect a user is trying to register a loader and the worker etc have not be set up:

Replace the existing DefaultModuleLoader with a CustomizedModuleLoader
Merge the two ModuleLoaders back together and trigger setups when certain methods are called (eg ModuleLoader::register())

The first seems the simplest. However, it's currently facing an issue where ModuleJob maintains a reference to the loader that instantiated it; in option 1, that reference becomes stale. Further work to be done is to sever their direct connection.

Review requested:

@nodejs/loaders
@nodejs/modules

Suggested change

      
                  initializeImportMeta: (meta, wrap) => this.importMetaInitialize(meta, { url }),
          
                  initializeImportMeta: (meta, wrap) => this.importMetaInitialize(meta, { __proto__: null, url }),

I think this does not matter here: we provide and destructure known keys, so it would never reach the prototype.

that would mean it doesn't matter now, but if anyone ever read a new key in the future without adding that key to every single callsite, it'd be a vulnerability.

That's valid but a larger issue, that I think is out of scope here.

Is there any advantage to not coding defensively here, especially in the module system?

I believe @aduh95 recently said it's a de-op

I'd say it can be a de-op only if you sometimes pass objects with null prototype and sometimes don't.

Let’s please not haggle over cleanup stuff yet. We need to get this PR working first.

Also isn’t there a lint rule for __proto__: null? So this will get fixed before the PR lands, in order to get the lint check to pass.

The linter rule only enforces which style is used for null objects - it doesn't enforce that null objects are used.

Totally fair tho to not haggle over cleanup stuff yet; there's no rush to address this.

Nit:

createModuleLoader makes the final decision of whether default module loader is created or customized module loader is created. But this comment and this function is called when we know CustomizedModuleLoader is going to be created. I think we should decouple the condition of conditionally creating customized or default module loader creation to their own functions and directly create them. This would potentially improve the readability and maintainability of loaders, since if in the future the behavior of createModuleLoader changes, we also need to update this line of comment.

Suggested change

      
                importAssertions = { __proto__: null },
          
                importAssertions = kEmptyObject,

The import attributes may be modified by a user loader, so using kEmptyObject wouldn't fly here. Also that code was there before this PR. (maybe let's not make unrelated whitespace changes?)

I changed multiple args to the constructor, so it wasn't just fixing the whitespace mangling. If you'd like the mangling fixed separately, I can re-mangle the 2 lines and fix in a follow-up (but the purpose of this PR is also clean-up that was deferred from the previous PR).

esmLoader getter directly calls createModuleLoader if not exist, which makes it really hard to keep track of the execution flow. Would you mind adding some comments about the assumption of the state of esmLoader attribute in here?

On Slack we were discussing creating a function getModuleLoader which would return the current active module loader instance, whether it was an instance of the customized or the default one.

Also in general this PR isn’t ready for review yet, I think it’s just where @JakobJingleheimer finished at the end of the night. It doesn’t work yet, and we’re not sure which of the two approaches described in the top post will work or be more readable.

Yeah, it's a draft. Not sure why GitHub tagged people for review.

It's the bot who does the tagging, not GitHub. The webhook is executed when you open a PR, regardless of its status. You probably already know that, but you don't have to open a PR, you can also push commits to your forks and open a PR later if you don't want comments for now.

Ah, it doesn't have to be configured that way (at work, we have the same kind of thing, and it only runs when the PR is not a draft—eg opened as non-draft or switches from draft to non-draft).

I didn’t realize there was a conversation was going on Slack. I mostly left these comments because I’m trying to familiarize myself with the loader implementation and I believe asking/commenting is the best way of learning.

Sorry for any disturbance caused by my comments.

I just meant this isn't ready for cleanup notes yet. We need to figure out which approach to take and get it working. If you have suggestions for that (see initial post) please feel free 😀

Note that this PR is effectively unflagging the loader implementation 🤔 not sure if it's ready for that just yet.

Suggested change

      
              it('works without a CLI flag', async () => {
          
              it('works without --experimental-loader flag', async () => {

Oh! whoops 😅

Suggested change

      
               * @param {*} importAssertions
          
               * @param {Record<string, string>} importAssertions

Or use whatever JSDoc we use for these elsewhere.

i'm confused why a map's "has" function would ever take more than one argument. is moduleMap something that's not a Map?

Good question! I also had to ask what this is.

The extra argument (import assertion type) is per spec required for matching cache. Under the hood, the extra argument is stored as part of the value (or maybe the key?).

I'll add some code doc for it after I get this stable/passing.

I think it’s very confusing to have something called “map” whose method names violate the expected interface contract for a Map, but i assume that predates this PR. (iow, i think instead of using has/set/etc it should have brand new method names)

Yes, this PR does not introduce ModuleMap ;)

Major functional changes: - Allow `import()` to work within loaders that require other loaders, - Unflag the use of `Module.register`. A new interface `Customizations` has been created in order to unify `ModuleLoader` (previously `DefaultModuleLoader`), `Hooks` and `CustomizedModuleLoader` all of which now implement it: ```ts interface LoadResult { format: ModuleFormat; source: ModuleSource; } interface ResolveResult { format: string; url: URL['href']; } interface Customizations { allowImportMetaResolve: boolean; load(url: string, context: object): Promise<LoadResult> resolve( originalSpecifier: string, parentURL: string, importAssertions: Record<string, string> ): Promise<ResolveResult> resolveSync( originalSpecifier: string, parentURL: string, importAssertions: Record<string, string> ) ResolveResult; register(specifier: string, parentUrl: string): any; forceLoadHooks(): void; importMetaInitialize(meta, context, loader): void; } ``` The `ModuleLoader` class now has `setCustomizations` which takes an object of this shape and delegates its responsibilities to this object if present. Note that two properties `allowImportMetaResolve` and `resolveSync` exist now as a mechanism for `import.meta.resolve` – since `Hooks` does not implement `resolveSync` other loaders cannot use `import.meta.resolve`; `allowImportMetaResolve` is a way of checking for that case instead of invoking `resolveSync` and erroring. Fixes nodejs#48515 Closes nodejs#48439

Major functional changes: - Allow `import()` to work within loaders that require other loaders, - Unflag the use of `Module.register`. A new interface `Customizations` has been created in order to unify `ModuleLoader` (previously `DefaultModuleLoader`), `Hooks` and `CustomizedModuleLoader` all of which now implement it: ```ts interface LoadResult { format: ModuleFormat; source: ModuleSource; } interface ResolveResult { format: string; url: URL['href']; } interface Customizations { allowImportMetaResolve: boolean; load(url: string, context: object): Promise<LoadResult> resolve( originalSpecifier: string, parentURL: string, importAssertions: Record<string, string> ): Promise<ResolveResult> resolveSync( originalSpecifier: string, parentURL: string, importAssertions: Record<string, string> ) ResolveResult; register(specifier: string, parentUrl: string): any; forceLoadHooks(): void; importMetaInitialize(meta, context, loader): void; } ``` The `ModuleLoader` class now has `setCustomizations` which takes an object of this shape and delegates its responsibilities to this object if present. Note that two properties `allowImportMetaResolve` and `resolveSync` exist now as a mechanism for `import.meta.resolve` – since `Hooks` does not implement `resolveSync` other loaders cannot use `import.meta.resolve`; `allowImportMetaResolve` is a way of checking for that case instead of invoking `resolveSync` and erroring. Fixes nodejs#48515 Closes nodejs#48439 PR-URL: nodejs#48559 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>

Major functional changes: - Allow `import()` to work within loaders that require other loaders, - Unflag the use of `Module.register`. A new interface `Customizations` has been created in order to unify `ModuleLoader` (previously `DefaultModuleLoader`), `Hooks` and `CustomizedModuleLoader` all of which now implement it: ```ts interface LoadResult { format: ModuleFormat; source: ModuleSource; } interface ResolveResult { format: string; url: URL['href']; } interface Customizations { allowImportMetaResolve: boolean; load(url: string, context: object): Promise<LoadResult> resolve( originalSpecifier: string, parentURL: string, importAssertions: Record<string, string> ): Promise<ResolveResult> resolveSync( originalSpecifier: string, parentURL: string, importAssertions: Record<string, string> ) ResolveResult; register(specifier: string, parentUrl: string): any; forceLoadHooks(): void; importMetaInitialize(meta, context, loader): void; } ``` The `ModuleLoader` class now has `setCustomizations` which takes an object of this shape and delegates its responsibilities to this object if present. Note that two properties `allowImportMetaResolve` and `resolveSync` exist now as a mechanism for `import.meta.resolve` – since `Hooks` does not implement `resolveSync` other loaders cannot use `import.meta.resolve`; `allowImportMetaResolve` is a way of checking for that case instead of invoking `resolveSync` and erroring. Fixes #48515 Closes #48439 PR-URL: #48559 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>

Major functional changes: - Allow `import()` to work within loaders that require other loaders, - Unflag the use of `Module.register`. A new interface `Customizations` has been created in order to unify `ModuleLoader` (previously `DefaultModuleLoader`), `Hooks` and `CustomizedModuleLoader` all of which now implement it: ```ts interface LoadResult { format: ModuleFormat; source: ModuleSource; } interface ResolveResult { format: string; url: URL['href']; } interface Customizations { allowImportMetaResolve: boolean; load(url: string, context: object): Promise<LoadResult> resolve( originalSpecifier: string, parentURL: string, importAssertions: Record<string, string> ): Promise<ResolveResult> resolveSync( originalSpecifier: string, parentURL: string, importAssertions: Record<string, string> ) ResolveResult; register(specifier: string, parentUrl: string): any; forceLoadHooks(): void; importMetaInitialize(meta, context, loader): void; } ``` The `ModuleLoader` class now has `setCustomizations` which takes an object of this shape and delegates its responsibilities to this object if present. Note that two properties `allowImportMetaResolve` and `resolveSync` exist now as a mechanism for `import.meta.resolve` – since `Hooks` does not implement `resolveSync` other loaders cannot use `import.meta.resolve`; `allowImportMetaResolve` is a way of checking for that case instead of invoking `resolveSync` and erroring. Fixes nodejs#48515 Closes nodejs#48439 PR-URL: nodejs#48559 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>

Major functional changes: - Allow `import()` to work within loaders that require other loaders, - Unflag the use of `Module.register`. A new interface `Customizations` has been created in order to unify `ModuleLoader` (previously `DefaultModuleLoader`), `Hooks` and `CustomizedModuleLoader` all of which now implement it: ```ts interface LoadResult { format: ModuleFormat; source: ModuleSource; } interface ResolveResult { format: string; url: URL['href']; } interface Customizations { allowImportMetaResolve: boolean; load(url: string, context: object): Promise<LoadResult> resolve( originalSpecifier: string, parentURL: string, importAssertions: Record<string, string> ): Promise<ResolveResult> resolveSync( originalSpecifier: string, parentURL: string, importAssertions: Record<string, string> ) ResolveResult; register(specifier: string, parentUrl: string): any; forceLoadHooks(): void; importMetaInitialize(meta, context, loader): void; } ``` The `ModuleLoader` class now has `setCustomizations` which takes an object of this shape and delegates its responsibilities to this object if present. Note that two properties `allowImportMetaResolve` and `resolveSync` exist now as a mechanism for `import.meta.resolve` – since `Hooks` does not implement `resolveSync` other loaders cannot use `import.meta.resolve`; `allowImportMetaResolve` is a way of checking for that case instead of invoking `resolveSync` and erroring. Fixes #48515 Closes #48439 PR-URL: #48559 Backport-PR-URL: #50669 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>

Major functional changes: - Allow `import()` to work within loaders that require other loaders, - Unflag the use of `Module.register`. A new interface `Customizations` has been created in order to unify `ModuleLoader` (previously `DefaultModuleLoader`), `Hooks` and `CustomizedModuleLoader` all of which now implement it: ```ts interface LoadResult { format: ModuleFormat; source: ModuleSource; } interface ResolveResult { format: string; url: URL['href']; } interface Customizations { allowImportMetaResolve: boolean; load(url: string, context: object): Promise<LoadResult> resolve( originalSpecifier: string, parentURL: string, importAssertions: Record<string, string> ): Promise<ResolveResult> resolveSync( originalSpecifier: string, parentURL: string, importAssertions: Record<string, string> ) ResolveResult; register(specifier: string, parentUrl: string): any; forceLoadHooks(): void; importMetaInitialize(meta, context, loader): void; } ``` The `ModuleLoader` class now has `setCustomizations` which takes an object of this shape and delegates its responsibilities to this object if present. Note that two properties `allowImportMetaResolve` and `resolveSync` exist now as a mechanism for `import.meta.resolve` – since `Hooks` does not implement `resolveSync` other loaders cannot use `import.meta.resolve`; `allowImportMetaResolve` is a way of checking for that case instead of invoking `resolveSync` and erroring. Fixes nodejs/node#48515 Closes nodejs/node#48439 PR-URL: nodejs/node#48559 Backport-PR-URL: nodejs/node#50669 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>

esm: remove CLI flag limitation to programmatic registration

28055ff

JakobJingleheimer added this to the ESM Loaders: Stable (CJS parity) milestone Jun 12, 2023

JakobJingleheimer self-assigned this Jun 12, 2023

nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Jun 12, 2023

ljharb reviewed Jun 12, 2023

View reviewed changes

anonrig reviewed Jun 13, 2023

View reviewed changes

WIP: disentangle loader from ModuleJob

d4a5dd8

JakobJingleheimer force-pushed the esm/instantiate-worker-on-demand branch from 9b4c85c to d4a5dd8 Compare June 13, 2023 08:15

aduh95 reviewed Jun 13, 2023

View reviewed changes

Comment thread lib/internal/modules/esm/module_job.js Outdated

aduh95 reviewed Jun 13, 2023

View reviewed changes

JakobJingleheimer mentioned this pull request Jun 14, 2023

module: implement register utility #46826

Merged

4 tasks

JakobJingleheimer added 7 commits June 15, 2023 22:21

WIP: continue to detangle ModuleJob & ESMLoader

378c5a5

WIP: add debugs

d2ad85b

WIP: await getting module job (now always async)

02c6090

WIP: move initialize import.meta out of module loader

cb72d94

WIP: tweak debugs

77e91dc

correct ModuleJob(url → responseURL)

582f50a

move setCallbackForWrap back into translator('module')

ec5d529

GeoffreyBooth reviewed Jun 19, 2023

View reviewed changes

JakobJingleheimer added 4 commits June 19, 2023 10:10

fix: restore setCallbackForWrap in ModuleLoader::eval()

e6e8cac

fix: restore gate of import.meta.resolve within loader worker

3819d08

fix: existing job detection

9ff1de7

WIP: investigate missing/tardy job entry in ModuleMap

6fa7dc7

ljharb reviewed Jun 23, 2023

View reviewed changes

GeoffreyBooth mentioned this pull request Jun 26, 2023

esm: unflag Module.register and allow nested loader import() #48559

Merged

JakobJingleheimer mentioned this pull request Jun 29, 2023

[Node 20] ESM Loaders no longer have process.argv access #48577

Open

JakobJingleheimer mentioned this pull request Jul 10, 2023

Replace globalPreload nodejs/loaders#147

Closed

This was referenced Jul 15, 2023

module: ensure successful dynamic import returns the same result #46662

Merged

Node.js Loaders Team Meeting 2023-07-18 nodejs/loaders#149

Closed

nodejs-github-bot closed this in a2fc4a3 Jul 19, 2023

JakobJingleheimer mentioned this pull request Aug 6, 2023

module: make CJS load from ESM loader #47999

Merged

-                    commonEvals.register(fixtures.fileURL('es-module-loaders', 'loader-load-passthru.mjs')) +
-                    commonEvals.dynamicImport('console.log("Hello from dynamic import");'),
-                  ]);
+                it('works without a CLI flag', async () => {

+                   // `job` may be a Promise, which the engine INSANELY _sometimes_ considers undefined and
+                   // sometimes not. This is why we use `has` instead of `get` to check for its existence.
+                   // ! Do NOT try to check against its value; the engine will **gladly** screw you over.
+                  if (!this.moduleMap.has(url, importAssertions.type)) {

	initializeImportMeta: (meta, wrap) => this.importMetaInitialize(meta, { url }),
	initializeImportMeta: (meta, wrap) => this.importMetaInitialize(meta, { __proto__: null, url }),

	importAssertions = { __proto__: null },
	importAssertions = kEmptyObject,

	it('works without a CLI flag', async () => {
	it('works without --experimental-loader flag', async () => {

+                 *
+                 * @param {URL['href']} url The resolved url of the module.
+                 * @param {ModuleWrap} moduleWrapper
+                 * @param {*} importAssertions

	* @param {*} importAssertions
	* @param {Record<string, string>} importAssertions

esm: remove CLI flag limitation to programmatic registration by JakobJingleheimer · Pull Request #48439 · nodejs/node · GitHub

Uh oh!

Uh oh!

esm: remove CLI flag limitation to programmatic registration - #48439

esm: remove CLI flag limitation to programmatic registration#48439JakobJingleheimer wants to merge 13 commits intonodejs:mainnodejs/node:mainfrom JakobJingleheimer:esm/instantiate-worker-on-demandJakobJingleheimer/node:esm/instantiate-worker-on-demandCopy head branch name to clipboard

Conversation

JakobJingleheimer commented Jun 12, 2023

Uh oh!

nodejs-github-bot commented Jun 12, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

esm: remove CLI flag limitation to programmatic registration#48439
JakobJingleheimer wants to merge 13 commits into
nodejs:mainfrom
JakobJingleheimer:esm/instantiate-worker-on-demand