FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

stream: restore flow if there are 'data' handlers after once('readable') by mcollina · Pull Request #22209 · nodejs/node · GitHub

/ node Public

stream: restore flow if there are 'data' handlers after once('readable') - #22209

Closed
mcollina wants to merge 1 commit into
nodejs:masterfrom
mcollina:fix-stream-alt-readable
Closed

stream: restore flow if there are 'data' handlers after once('readable')#22209
mcollina wants to merge 1 commit into
nodejs:masterfrom
mcollina:fix-stream-alt-readable

Conversation

mcollina commented Aug 9, 2018

Copy link
Copy Markdown
Member

In #18994, we made 'readable'  take precedence over 'data'/resume() and pause(). However, we didn't take into account situation where both 'readable'  and 'data' event handler were present at the same time. This PR addresses it by starting flowing after 'readable' is removed.

Fixes: #21398

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines

mcollina requested review from addaleax and mafintosh August 9, 2018 12:14

Copy link
Copy Markdown
Collaborator

nodejs-github-bot added the stream Issues and PRs related to the stream subsystem. label Aug 9, 2018

mcollina commented Aug 9, 2018
edited
Loading

Copy link
Copy Markdown
Member Author

This is an alternative implementation of #21696.
cc @lundibundi

mcollina commented Aug 9, 2018

Copy link
Copy Markdown
Member Author

Copy link
Copy Markdown
Member

It seems like this one has the same flaws my PR had at the beginning

  • ignoring user pause
  • piping with .on('readable') fails

But the first one is documented now.
So basically there is no way to pause stream if you use 'readable' and pipe doesn't work. The latter is as before but I don't think it's a good behavior, though if we go with this PR I think it has to be documented explicitly in .pipe doc, as well as that stream will be set to paused if you add 'data' listener after 'readable'.

mcollina commented Aug 9, 2018

Copy link
Copy Markdown
Member Author

ignoring user pause

That works exactly as expected. If resume() is ignored, why pause()  shouldn't?
stream.pause() and stream.resume() are methods to be used with .on('data').

piping with on('readable') fails

Which is as expected. on('readable') is a method of consuming data from a stream with backpressure, so a user has to consume it via stream.read() calls. Side note, without #18994 stream.read() always returned null when used with stream.pipe().

Copy link
Copy Markdown
Member

Yeah, this may be better as it simplifies the implementation a bit and signifies the fact that you shouldn't use 'readable' with 'data'. But as I said we have to better document it as issues of incorrect usage of streams are constantly popping up.

mcollina force-pushed the fix-stream-alt-readable branch from 8fc5f64 to a7e57aa Compare August 9, 2018 17:24

mcollina commented Aug 9, 2018

Copy link
Copy Markdown
Member Author

I'll add some more docs for review tomorrow.

CI: https://ci.nodejs.org/job/node-test-pull-request/16301/

Copy link
Copy Markdown
Member Author

Comment thread doc/api/stream.md Outdated

vsemozhetbyt Aug 11, 2018
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

[`readable.read()`][]-> [`readable.read()`][stream-read]?

Comment thread doc/api/stream.md Outdated

vsemozhetbyt Aug 11, 2018
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

[`data`][] -> [`'data'`][]?

Copy link
Copy Markdown
Member Author

CITGM seems ok.

Copy link
Copy Markdown
Member Author

mcollina force-pushed the fix-stream-alt-readable branch from 58b5968 to 2956304 Compare August 11, 2018 10:15

Copy link
Copy Markdown
Member Author

@vsemozhetbyt PTAL

Copy link
Copy Markdown
Contributor

Doc part LGTM.

Copy link
Copy Markdown
Member Author

@mafintosh @addaleax PTAL.

Copy link
Copy Markdown
Member Author

oyyd commented Aug 21, 2018

Copy link
Copy Markdown
Contributor

@mcollina I think this relates to #21122. What if we don't call read() inside the callback of 'readable' ? This might be regarded as an undefined behavior but calling on('data') before on('readable') like below:

const fs = require('fs')
const { resolve } = require('path')

const source = resolve(__dirname, 'index.js')
const target = resolve(__dirname, 'copy.js')

const reader = fs.createReadStream(source)
const writer = fs.createWriteStream(target);

reader.on('close', () => {
    console.log('reader closed');
});

writer.on('close', () => {
    console.log('writer closed');
});

reader.on('data', () => {
  console.log('receive data')
})

reader.on('readable', () => {
  // DO NOT `reader.read()`
})

Will print:

receive data
reader closed

However, change the order of binding these two events will result in different behaviors:

const fs = require('fs')
const { resolve } = require('path')

const source = resolve(__dirname, 'index.js')
const target = resolve(__dirname, 'copy.js')

const reader = fs.createReadStream(source)
const writer = fs.createWriteStream(target);

reader.on('close', () => {
    console.log('reader closed');
});

writer.on('close', () => {
    console.log('writer closed');
});

reader.on('readable', () => {
  // DO NOT `reader.read()`
})

reader.on('data', () => {
  console.log('receive data')
})

Will print nothing and exit.

Also, replace reader.on('data') with reader.pipe(writer) will cause similar results.

Copy link
Copy Markdown
Member Author

@oyyd yes it does. Unfortunately the documentation for #18994 was not clear enough. This resolves an edge case for that PR and it improves the docs.

I've also replied in the issue you linked.

oyyd commented Aug 21, 2018

Copy link
Copy Markdown
Contributor

I see, thanks.

mcollina force-pushed the fix-stream-alt-readable branch from 2956304 to 861124a Compare August 21, 2018 15:20

Copy link
Copy Markdown
Member Author

Copy link
Copy Markdown
Member

LGTM, nice fix

Copy link
Copy Markdown
Member Author

Copy link
Copy Markdown
Member Author

@nodejs/build I can't get centos7-64-gcc6 to pass, see https://ci.nodejs.org/job/node-test-commit-linux/20945/nodes=centos7-64-gcc6/console.

This looks like an infra issue, see nodejs/build#1468.

jasnell added the author ready PRs that have at least one approval, no outstanding review comments, and a CI started. label Aug 21, 2018

Trott commented Aug 22, 2018

Copy link
Copy Markdown
Member

Trott pushed a commit to Trott/io.js that referenced this pull request Aug 22, 2018
Fixes: nodejs#21398
See: nodejs#21696

PR-URL: nodejs#22209
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Mathias Buus <mathiasbuus@gmail.com>

Trott commented Aug 22, 2018

Copy link
Copy Markdown
Member

Resume Build resulted in a green CI run.

Landed in 98cf84f.

Trott closed this Aug 22, 2018
targos pushed a commit that referenced this pull request Aug 24, 2018
Fixes: #21398
See: #21696

PR-URL: #22209
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Mathias Buus <mathiasbuus@gmail.com>
targos pushed a commit that referenced this pull request Sep 3, 2018
Fixes: #21398
See: #21696

PR-URL: #22209
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Mathias Buus <mathiasbuus@gmail.com>
daprahamian added a commit to mongodb/node-mongodb-native that referenced this pull request Sep 17, 2018
Fixes test failure causes by combination of nodejs/node#22209 and
an ambiguous test.
daprahamian added a commit to mongodb/node-mongodb-native that referenced this pull request Sep 17, 2018
Fixes test failure causes by combination of nodejs/node#22209 and
an ambiguous test.
mbroadst pushed a commit to mongodb/node-mongodb-native that referenced this pull request Sep 18, 2018
Fixes test failure causes by combination of nodejs/node#22209 and
an ambiguous test.
daprahamian added a commit to mongodb/node-mongodb-native that referenced this pull request Sep 18, 2018
Fixes test failure causes by combination of nodejs/node#22209 and
an ambiguous test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author ready PRs that have at least one approval, no outstanding review comments, and a CI started. stream Issues and PRs related to the stream subsystem.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants


Back | FazBrowse Home | New Git URL