| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
|
Review requested:
|
Sorry, something went wrong.
Sorry, something went wrong.
|
I'm not sure I understand what this is supposed to do? |
Sorry, something went wrong.
|
Currently the map operator is using queue based strategy which means the order matter, sometimes you have a situation where you don't care about the order of the items, so you want pool based strategy which means as soon as one finishes another one starts You can have more information in the attached issue. If you can be more specific I would appreciate it 😀 |
Sorry, something went wrong.
Sorry, something went wrong.
There was a problem hiding this comment.
I don't really understand why this is called pool and what it is trying to achieve exactly.
Sorry, something went wrong.
There was a problem hiding this comment.
can you explain why?
Sorry, something went wrong.
There was a problem hiding this comment.
sorry Raz I don't understand it either,
Sorry, something went wrong.
|
Let's take for example this situation, we have 6 items and use concurrency of 3: currently, the execution would look like that:
1 .......... 2 ......................... 3 ..... ░░░░░░░░░░ 4 ............... ░░░░░░░░░░░░░░░░░░░░░░░░░ 5 .............................. ░░░░░░░░░░░░░░░░░░░░░░░░░ 6 ... Notice that 4 does not start when the earliest of 1, 2 and 3 (which is 3 in this case) finish but only after 1 finish the problem is that what if we don't care about the order, and we just wanna do 3 items in parallel in every single moment this is what pool means as opposed to queue which imply the order is kept with this pr, when passing pool: true we will have this: 1 .......... 2 ......................... 3 ..... ░░░░░ 4 ............... ░░░░░░░░░░ 5 .............................. ░░░░░░░░░░░░░░░░░░░░░░░ 6 ... as soon as something is finished another one starts (in this case 3 finished so 4 start immediately) If you still have questions let me know |
Sorry, something went wrong.
|
I don't understand why we wait at all for 1 to finish before starting 4, it seems like we should just change that instead of adding an option? |
Sorry, something went wrong.
|
I agree with @benjamingr, I don't see a reason for the old behavior. |
Sorry, something went wrong.
|
if we don't keep the old behavior but we modify the new one to push items in the order they came it can cause memory leak when not consuming first items and changing the behavior to not emit items in the order they can is a breaking change that I don't think we should do as it's valuable |
Sorry, something went wrong.
How? What sort of leak?
The API is experimental and my expectation around concurrency is that once you pass it the output order stays the same and the consumption order is the same but things run well... concurrently, I would have 0 expectation of something like the following taking 10 seconds and not 5: Readable.from([5000, 1, 1, 1, 1, 5000]).map(async x => {
await setTimeout(x);
}, { concurrency: 4 }).toArray();But looking at the code my guess is that it does |
Sorry, something went wrong.
sort of leak is that if we don't consume the first item and we just consume all the rest we gonna accumulate data
{
Readable.from([5000, 1, 1, 1, 1, 5000]).map(async x => {
await setTimeout(x);
return x;
}, { concurrency: 4 }).toArray().then(val => console.log('queue', val));
}
{
Readable.from([5000, 1, 1, 1, 1, 5000]).map(async x => {
await setTimeout(x);
return x;
}, { concurrency: 4, pool: true }).toArray().then(val => console.log('pool', val));
}The output is: pool [ 1, 1, 1, 1, 5000, 5000 ] queue [ 5000, 1, 1, 1, 1, 5000 ] so the output is not the same... |
Sorry, something went wrong.
|
Right, here is an alternative: which tells the stream how much to buffer, we re-add highWaterMark and default it to something reasonable. We keep the output order as long as we can depending on highWaterMark, as soon as it's too much to buffer we wait for the buffer to be consumed before doing further work. |
Sorry, something went wrong.
|
but what if we don't care about the order? |
Sorry, something went wrong.
|
Buffering more is fine, changing what we wait for when we consume is fine but we shouldn't change the actual output of .map in options IMO. |
Sorry, something went wrong.
|
Changing order of map output is something that I would never expect, and most likely no one else as well 🥲 |
Sorry, something went wrong.
|
What reasonable highWaterMark should be? |
Sorry, something went wrong.
|
When you say adding highWaterMark you mean implementing it? As by itself I don't see how it would help as we don't push the next item until the previous finish |
Sorry, something went wrong.
|
We call it highWatermark and it means how much the operator buffers and not just the resulting stream? |
Sorry, something went wrong.
I implemented highWaterMark buffering, WDYT? is that what you meant? (Need to update the tests but wanna know if that's what you meant) |
Sorry, something went wrong.
Sorry, something went wrong.
The ones that are not causing failures for the last few bug fix suggestions I made. |
Sorry, something went wrong.
Sorry, something went wrong.
all the bug fixes are already covered from what I checked |
Sorry, something went wrong.
Commit Queue failed- Loading data for nodejs/node/pull/49249 ✔ Done loading data for nodejs/node/pull/49249 ----------------------------------- PR info ------------------------------------ Title stream: add `highWaterMark` to the `map` and `filter` operator (#49249) ⚠ Could not retrieve the email or name of the PR author's from user's GitHub profile! Branch rluvaton:add-stream-map-pool -> nodejs:main Labels stream, needs-ci Commits 10 - stream: add `highWaterMark` for the map operator - stream: update test to check output order - stream: update and rename - stream: filter empty in the right place - stream: remove unnecessary code - stream: allow user to pass `highWaterMark` to `map` and `filter` fns - stream: update docs after adding `highWaterMark` - stream: fix highWaterMark validation and fix test - stream: updated CR - stream: fix concurrency and test Committers 1 - Raz Luvaton <16746759+rluvaton@users.noreply.github.com> PR-URL: https://github.com/nodejs/node/pull/49249 Fixes: https://github.com/nodejs/node/issues/46132 Reviewed-By: Matteo Collina Reviewed-By: Benjamin Gruenbaum Reviewed-By: Robert Nagy ------------------------------ Generated metadata ------------------------------ PR-URL: https://github.com/nodejs/node/pull/49249 Fixes: https://github.com/nodejs/node/issues/46132 Reviewed-By: Matteo Collina Reviewed-By: Benjamin Gruenbaum Reviewed-By: Robert Nagy -------------------------------------------------------------------------------- ℹ This PR was created on Sat, 19 Aug 2023 23:11:27 GMT ✔ Approvals: 3 ✔ - Matteo Collina (@mcollina) (TSC): https://github.com/nodejs/node/pull/49249#pullrequestreview-1592477937 ✔ - Benjamin Gruenbaum (@benjamingr) (TSC): https://github.com/nodejs/node/pull/49249#pullrequestreview-1593092273 ✔ - Robert Nagy (@ronag) (TSC): https://github.com/nodejs/node/pull/49249#pullrequestreview-1593077530 ✔ Last GitHub CI successful ℹ Last Full PR CI on 2023-08-24T09:20:44Z: https://ci.nodejs.org/job/node-test-pull-request/53539/ - Querying data for job/node-test-pull-request/53539/ ✔ Last Jenkins CI successful -------------------------------------------------------------------------------- ✔ No git cherry-pick in progress ✔ No git am in progress ✔ No git rebase in progress -------------------------------------------------------------------------------- - Bringing origin/main up to date... From https://github.com/nodejs/node * branch main -> FETCH_HEAD ✔ origin/main is now up-to-date - Downloading patch for 49249 From https://github.com/nodejs/node * branch refs/pull/49249/merge -> FETCH_HEAD ✔ Fetched commits as da197d189021..4b0707997951 -------------------------------------------------------------------------------- [main 0034c0e5f8] stream: add `highWaterMark` for the map operator Author: Raz Luvaton <16746759+rluvaton@users.noreply.github.com> Date: Sun Aug 20 01:42:56 2023 +0300 2 files changed, 159 insertions(+), 17 deletions(-) [main d02a503624] stream: update test to check output order Author: Raz Luvaton <16746759+rluvaton@users.noreply.github.com> Date: Wed Aug 23 18:14:20 2023 +0300 1 file changed, 6 insertions(+), 3 deletions(-) [main d41d4506f7] stream: update and rename Author: Raz Luvaton <16746759+rluvaton@users.noreply.github.com> Date: Wed Aug 23 18:16:36 2023 +0300 1 file changed, 8 insertions(+), 12 deletions(-) [main 281e41769b] stream: filter empty in the right place Author: Raz Luvaton <16746759+rluvaton@users.noreply.github.com> Date: Wed Aug 23 21:46:58 2023 +0300 1 file changed, 5 insertions(+), 4 deletions(-) [main bd13b8d45a] stream: remove unnecessary code Author: Raz Luvaton <16746759+rluvaton@users.noreply.github.com> Date: Thu Aug 24 00:16:31 2023 +0300 1 file changed, 2 insertions(+), 3 deletions(-) [main 1e131966c9] stream: allow user to pass `highWaterMark` to `map` and `filter` fns Author: Raz Luvaton <16746759+rluvaton@users.noreply.github.com> Date: Thu Aug 24 00:31:55 2023 +0300 2 files changed, 60 insertions(+), 5 deletions(-) Auto-merging doc/api/stream.md [main 0b8bc71a8c] stream: update docs after adding `highWaterMark` Author: Raz Luvaton <16746759+rluvaton@users.noreply.github.com> Date: Thu Aug 24 00:43:55 2023 +0300 1 file changed, 12 insertions(+) [main d2d8025628] stream: fix highWaterMark validation and fix test Author: Raz Luvaton <16746759+rluvaton@users.noreply.github.com> Date: Thu Aug 24 10:38:54 2023 +0300 3 files changed, 6 insertions(+), 3 deletions(-) [main d78a1158ee] stream: updated CR Author: Raz Luvaton <16746759+rluvaton@users.noreply.github.com> Date: Thu Aug 24 10:57:31 2023 +0300 1 file changed, 4 insertions(+), 4 deletions(-) [main 89215b6aa9] stream: fix concurrency and test Author: Raz Luvaton <16746759+rluvaton@users.noreply.github.com> Date: Thu Aug 24 11:22:42 2023 +0300 2 files changed, 13 insertions(+), 14 deletions(-) ✔ Patches applied There are 10 commits in the PR. Attempting autorebase. Rebasing (2/20)https://github.com/nodejs/node/actions/runs/5963058918 |
Sorry, something went wrong.
this is done so we don't wait for the first items to finish before starting new ones Fixes: #46132 Co-authored-by: Robert Nagy <ronagy@icloud.com> PR-URL: #49249 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Robert Nagy <ronagy@icloud.com>
this is done so we don't wait for the first items to finish before starting new ones Fixes: #46132 Co-authored-by: Robert Nagy <ronagy@icloud.com> PR-URL: #49249 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Robert Nagy <ronagy@icloud.com>
this is done so we don't wait for the first items to finish before starting new ones Fixes: nodejs/node#46132 Co-authored-by: Robert Nagy <ronagy@icloud.com> PR-URL: nodejs/node#49249 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Robert Nagy <ronagy@icloud.com>
this is done so we don't wait for the first items to finish before starting new ones Fixes: nodejs/node#46132 Co-authored-by: Robert Nagy <ronagy@icloud.com> PR-URL: nodejs/node#49249 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Robert Nagy <ronagy@icloud.com>
| Back | FazBrowse Home | New Git URL |
Fixes: #46132
Co-authored-by: Robert Nagy ronagy@icloud.com