FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

stream: readable stream continues to read when pushing a empty string by MoonBall · Pull Request #18211 · nodejs/node · GitHub

/ node Public

stream: readable stream continues to read when pushing a empty string - #18211

Closed
MoonBall wants to merge 1 commit into
nodejs:masterfrom
MoonBall:readable-stream-continue-to-read-when-pushing-undefined
Closed

stream: readable stream continues to read when pushing a empty string#18211
MoonBall wants to merge 1 commit into
nodejs:masterfrom
MoonBall:readable-stream-continue-to-read-when-pushing-undefined

Conversation

Copy link
Copy Markdown
Member
Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines
Affected core subsystem(s)

stream

nodejs-github-bot added the stream Issues and PRs related to the stream subsystem. label Jan 17, 2018

jasnell commented Jan 17, 2018

Copy link
Copy Markdown
Member

ping @nodejs/streams @mcollina

Copy link
Copy Markdown
Member

From reading your test I'm not sure I completely understand what the problem is?

MoonBall commented Jan 18, 2018
edited
Loading

Copy link
Copy Markdown
Member Author

@mafintosh From the view of a nodejs user, I think a readable stream should end if only calling read() in a 'readable' listener. The key code is as below:

const r = new Readable();
r.on('readable', () => {
  r.read();
});
r.on('end', () => {
  // calling `read()` in a 'readable' listener should ultimately emit the 'end' event.
});

But if _read() pushes a empty string or undefined, the key code doesn't emit the 'end' event. But I'm not sure that the situation exists. It can be avoided by stream implementors.

Copy link
Copy Markdown
Member

I am 👎 to this change. undefined is a legit value in object mode stream, and it should not terminate.

Copy link
Copy Markdown
Member

Yes, I'm agreeing with @mcollina as well. Stream termination should be as explicit as possible. I think we should consider having an explicit API for it at some point.

Copy link
Copy Markdown
Member Author

@mcollina @mafintosh My purpose may be misunderstood. I updated the test so that you can better understand it. I am so sorry for my poor english.

MoonBall commented Jan 18, 2018
edited
Loading

Copy link
Copy Markdown
Member Author

We can just consider that this.push() pushes a empty string. I updated the title of the PR.

MoonBall changed the title stream: readable stream continues to read when pushing undefined stream: readable stream continues to read when pushing a empty string Jan 18, 2018

Copy link
Copy Markdown
Member

Here are some automated runs:

CI: https://ci.nodejs.org/job/node-test-pull-request/12611/
CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1209/

Anyway, if we want to relax this it needs to be done earlier: https://github.com/MoonBall/node/blob/49ed0814f2750075fb64ab167f9a129f70f37d52/lib/_stream_readable.js#L234 esplicitly checks for a string longer than zero. This check was made pretty explicit, so if we want to make this change we should edit it there.

I'm not super-confident in adding '' as a valid chunk, because you have actually written nothing and the stream knows it should push things. However I see why it could improve the API.

cc @mafintosh?

Copy link
Copy Markdown
Member

The linter seems to have failed with:

10:49:31 not ok 36 - /usr/home/iojs/build/workspace/node-test-linter/test/parallel/test-stream-readable-event.js
10:49:31 ---
10:49:31 message: '''underlyingData'' is never reassigned. Use ''const'' instead.'
10:49:31 severity: error
10:49:31 data:
10:49:31 line: 90
10:49:31 column: 7
10:49:31 ruleId: prefer-const

Copy link
Copy Markdown
Member Author

@addaleax I fixed it.
@mcollina I changed the code as you suggested.

Copy link
Copy Markdown
Member

Can you also rebase on top of master?

Copy link
Copy Markdown
Member Author

ok.

MoonBall force-pushed the readable-stream-continue-to-read-when-pushing-undefined branch 2 times, most recently from eef4179 to 9547feb Compare January 18, 2018 17:40

Copy link
Copy Markdown
Member Author

Rebased.

MoonBall commented Jan 19, 2018
edited
Loading

Copy link
Copy Markdown
Member Author

@mcollina In non-objectMode, we can call this.push() without a param. Is it valid?
The code of the PR will throw Error, if pushing a undefined in non-objectMode.

I fixed that pushing undefined in another PR(#18244).

Copy link
Copy Markdown
Member

What happens if you push undefined with this change?

Copy link
Copy Markdown
Member

MoonBall commented Jan 19, 2018
edited
Loading

Copy link
Copy Markdown
Member Author

@mcollina
if I push undefined, the following code will execute.

      if (typeof chunk !== 'string' &&
          !state.objectMode &&
          Object.getPrototypeOf(chunk) !== Buffer.prototype) {
        chunk = Stream._uint8ArrayToBuffer(chunk);
      }

The error is:

_stream_readable.js:237
          Object.getPrototypeOf(chunk) !== Buffer.prototype) {
                 ^

TypeError: Cannot convert undefined or null to object
    at Function.getPrototypeOf (<anonymous>)
    at readableAddChunk (_stream_readable.js:237:18)
    at Readable.push (_stream_readable.js:215:10)

I doesn't fix it, because I think undefined is invalid. But, if we allow to push undefined in non-object mode, I will fix it.

MoonBall force-pushed the readable-stream-continue-to-read-when-pushing-undefined branch from 9547feb to aaf9bc7 Compare January 22, 2018 15:31

Copy link
Copy Markdown
Member Author

@mcollina I modified the code because that push() maybe pass a undefined chunk.

Copy link
Copy Markdown
Member

I'm not super convinced by this change, I'll have more time to review it in a week.
Also, I would like to fix #18294 before landing this, as they are definitely connected.

@mafintosh can you check this?

Copy link
Copy Markdown
Member

Let me try and look into it tonight

BridgeAR commented Feb 1, 2018

Copy link
Copy Markdown
Member

@mcollina can you please have another look? #18294 got fixed and some time passed by.

mcollina left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I'm not convinced that changing this is the right thing to do, yet. Maybe somebody else should weight in on why.

BTW, this is semver-major.

mcollina added the semver-major PRs that contain breaking changes and should be released in the next major version. label Feb 2, 2018

mafintosh commented Feb 2, 2018
edited
Loading

Copy link
Copy Markdown
Member

@MoonBall @mcollina The tests doesn't pass for me when rebasing on top of latest master, do they for you? Also the test isn't completely clear still imo. Does this make it emit an empty string on read?

Copy link
Copy Markdown
Member

My bad with the tests, was using the wrong branch. They pass.

Copy link
Copy Markdown
Member

@MoonBall what happens if you do push(Buffer.alloc(0)) ?

MoonBall commented Feb 3, 2018

Copy link
Copy Markdown
Member Author

@mafintosh The result is same if I push() a empty Buffer, a empty string or undefined because of the condition.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

could you collect the chunks in an array and deepEqual those and the end instead of the concatenated string? That'd make it a bit easier for me (and others) to understand and show that it isn't returning an empty string here ever :)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

There is a misunderstanding. This change only fixed that the 'end' event isn't emitted when we push a empty string, a empty buffer or undefined.

MoonBall Feb 4, 2018
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Currently, a readable stream in non-object mode doesn't inform users a empty string or a empty buffer. It is reasonable.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I'd like it if the test was a bit more explicit.

result += data doesn't tell me if r.read() returns an empty buffer, which from reading the test, you might expect.

Doing result.push(data) and then on end assert.deepEquals(result, [...]) would help

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

@mafintosh I did it.

Copy link
Copy Markdown
Member

@MoonBall Thanks for the explanation! Added a comment on your test case. @mcollina I'm leaning 👍 on this, think it makes sense.

MoonBall force-pushed the readable-stream-continue-to-read-when-pushing-undefined branch from aaf9bc7 to 29554ed Compare February 5, 2018 15:03
MoonBall force-pushed the readable-stream-continue-to-read-when-pushing-undefined branch from 29554ed to 77a66d3 Compare February 5, 2018 15:16
BridgeAR requested review from a team and mcollina February 13, 2018 05:46

mcollina left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM

Copy link
Copy Markdown
Member

Copy link
Copy Markdown
Member

Landing

Copy link
Copy Markdown
Member

Landed as faeee11.

mcollina closed this Feb 15, 2018
mcollina pushed a commit that referenced this pull request Feb 15, 2018
PR-URL: #18211
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
MayaLekova pushed a commit to MayaLekova/node that referenced this pull request May 8, 2018
PR-URL: nodejs#18211
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

semver-major PRs that contain breaking changes and should be released in the next major version. stream Issues and PRs related to the stream subsystem.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants


Back | FazBrowse Home | New Git URL