FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

buffer: add buffer.transcode by jasnell · Pull Request #9038 · nodejs/node · GitHub

/ node Public

buffer: add buffer.transcode - #9038

Closed
jasnell wants to merge 1 commit into
nodejs:masterfrom
jasnell:buffer-transcode
Closed

buffer: add buffer.transcode#9038
jasnell wants to merge 1 commit into
nodejs:masterfrom
jasnell:buffer-transcode

Conversation

jasnell commented Oct 11, 2016
edited
Loading

Copy link
Copy Markdown
Member
Checklist
  • make -j8 test (UNIX), or vcbuild test nosign (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines
Affected core subsystem(s)

buffer

Description of change

Add buffer.transcode(source, from, to) method. Primarily uses ICU to transcode a buffer's content from one of Node.js' supported encodings to another.

Originally part of a proposal to add a new unicode module. Decided to refactor the approach towrds individual PRs without a new module.

Refs: #8075
/cc @trevnorris @addaleax

jasnell added the buffer Issues and PRs related to the buffer subsystem. label Oct 11, 2016
nodejs-github-bot added build Issues and PRs related to build files or the CI. c++ Issues and PRs that require attention from people who are familiar with C++. tools Issues and PRs related to the tools directory. labels Oct 11, 2016

jasnell commented Oct 11, 2016

Copy link
Copy Markdown
Member Author

jasnell added the semver-minor PRs that contain new features and should be released in the next minor version. label Oct 11, 2016
Comment thread lib/internal/buffer.js Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

what about possibly placing this in lib/internal/buffer-transcode.js and conditionally require()'ing it. purely cosmetic though, to prevent an extra level of indent. or you could just return early. :)

if (!process.binding('config').hasIntl)
  return;

Comment thread lib/internal/buffer.js Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

We going to have a complaint about not supporting SharedArrayBuffer for this?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Eventually, perhaps. Not too worried about that for now.

jasnell commented Oct 12, 2016

Copy link
Copy Markdown
Member Author

Updated

Comment thread doc/api/buffer.md Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Maybe give an example of transcoding is not possible?

Comment thread src/node_i18n.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Hm – this could be a ThrowICUError function, right?

Comment thread src/node_i18n.cc Outdated

addaleax Oct 11, 2016
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

This is fine but at some point this might become a member of the MaybeStackBuffer class? I realize that would conflict a bit with the MaybeStackBuffer<UChar> overload, maybe leave a TODO here?

Comment thread lib/internal/buffer.js Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Is that final to residue from editing?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Yeah, slight brain malfunction there I think ;-)

Comment thread lib/internal/buffer.js Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Mhhh this returns a string or a Buffer depending on the target encoding? I don’t think binary-to-text encodings should be allowed here, .toString() is the right method for them.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Yeah, you're right. I'll pull these back out.

jasnell commented Oct 12, 2016

Copy link
Copy Markdown
Member Author

@addaleax ... updated! PTAL

addaleax left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM, nice!

Comment thread src/node_i18n.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Could this use SwapBytes16?

Comment thread src/node_i18n.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

(ditto)

Comment thread src/node_i18n.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

The 1024 seem kind of magic here, although I realize that is largely my fault. 😄 (Not sure if there’s anything to do about that)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Should be fixed now!

Comment thread lib/internal/buffer.js Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Style: s/from_enc/fromEncoding/ and s/to_enc/toEncoding/. Ditto for cnv_from and cnv_to.

Comment thread src/node_i18n.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I realize you adapted this code from elsewhere but using snprintf() to format the error message will be much more efficient.

Comment thread src/node_i18n.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Needs a if (e.Empty()) return Local<Value>();.

Comment thread src/node_i18n.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I think this should be if (!ret.Empty()) buf->Release(); - it's leaking memory now when the buffer can't be created.

Comment thread src/node_i18n.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Ditto.

Comment thread src/node_i18n.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Same as above: strict aliasing violation and prone to crashing.

Comment thread src/node_i18n.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

It seems like there is ample opportunity to share code between Ucs2FromUtf8 and Utf8FromUcs2, they are 80% identical.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Perhaps. For now I'm more inclined to keep these separate as it makes finding and tweaking bugs a bit easier. I'll take another pass in a separate PR to condense things down.

Comment thread src/util.h Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Shouldn't this also reset length_?

Comment thread src/util.h Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

If you move this into a common header, you might want to give it a slightly less generic name; e.g. SPREAD_BUFFER_ARG.

Comment thread tools/icu/icu-generic.gyp Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Just remove the 'defines' block instead of commenting it out.

jasnell force-pushed the buffer-transcode branch 3 times, most recently from 9e0464b to 2da087e Compare October 13, 2016 15:11

jasnell commented Oct 13, 2016
edited
Loading

Copy link
Copy Markdown
Member Author

jasnell commented Oct 13, 2016
edited
Loading

Copy link
Copy Markdown
Member Author

Another CI run after cleanups: https://ci.nodejs.org/job/node-test-pull-request/4507/ ... that last run was less than successful....
Trying another: https://ci.nodejs.org/job/node-test-pull-request/4508/

jasnell commented Oct 13, 2016

Copy link
Copy Markdown
Member Author

CI looks good. @bnoordhuis PTAL... LGTY?

jasnell commented Oct 17, 2016

Copy link
Copy Markdown
Member Author

ping @bnoordhuis

Comment thread src/node_i18n.cc Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

if you're returning empty handles as an indicator it's probably be more "V8-ish" have the return signature as a MaybeLocal<Value> instead. been trying to do that in other locations myself.

Comment thread src/node_i18n.cc Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

creating Local<Value>'s but there's no HandleScope. if this callback is expected to always be called within an existing HandleScope (like MakeCallback), mind putting a comment at the top. also like MakeCallback (see src/node.h).

Comment thread src/node_i18n.cc Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

if we know this is a v8::Object then can use e.As<Object>(). that's also more explicit that no extra handle is being created.

Comment thread src/node_i18n.cc Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

The new v8::Maybe<T> API for v8::Object::Set() is annoying and ugly, but if if we're going to use some of the new API might as well use all of it.

Comment thread src/node_i18n.cc Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

if you're manipulating the original memory, why bother take a copy?

Comment thread src/node_i18n.cc Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

if this operation fails, do we want to abort or throw?

Comment thread src/node_i18n.cc Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

for future performance enhancement, detect the alignment of the pointer and perform as many swaps that can be done in a single go.

Comment thread lib/internal/buffer.js Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

i'm a little confused by this whole object, but right here if we're converting ascii to latin1 shouldn't we be passing 'latin1' as the encoding argument to Buffer.from()?

jasnell force-pushed the buffer-transcode branch 3 times, most recently from 6f70820 to 65144bb Compare October 18, 2016 21:43

jasnell commented Oct 18, 2016

Copy link
Copy Markdown
Member Author

@bnoordhuis ... ok, reworked the implementation with an eye towards simplification and reducing duplication. PTAL
@trevnorris and @addaleax ... if I could trouble each of you to take another look also, I'd appreciate it.

jasnell commented Oct 21, 2016

Copy link
Copy Markdown
Member Author

ping @bnoordhuis @trevnorris @addaleax

jasnell commented Oct 24, 2016

Copy link
Copy Markdown
Member Author

Thank you for the follow up review @addaleax. Will get this landed tomorrow if there are no further objections.

Add buffer.transcode(source, from, to) method. Primarily uses ICU
to transcode a buffer's content from one of Node.js' supported
encodings to another.

Originally part of a proposal to add a new unicode module. Decided
to refactor the approach towrds individual PRs without a new module.

Refs: nodejs#8075

jasnell commented Oct 25, 2016

Copy link
Copy Markdown
Member Author

New CI run after squashing: https://ci.nodejs.org/job/node-test-pull-request/4665/

jasnell commented Oct 25, 2016

Copy link
Copy Markdown
Member Author

green except for unrelated failures. landing

jasnell dismissed bnoordhuis’s stale review October 25, 2016 17:11

PR updated after view

jasnell added a commit that referenced this pull request Oct 25, 2016
Add buffer.transcode(source, from, to) method. Primarily uses ICU
to transcode a buffer's content from one of Node.js' supported
encodings to another.

Originally part of a proposal to add a new unicode module. Decided
to refactor the approach towrds individual PRs without a new module.

Refs: #8075
PR-URL: #9038
Reviewed-By: Anna Henningsen <anna@addaleax.net>

jasnell commented Oct 25, 2016

Copy link
Copy Markdown
Member Author

Landed in e8eaaa7

jasnell closed this Oct 25, 2016
srl295 mentioned this pull request Oct 25, 2016
4 tasks
evanlucas pushed a commit that referenced this pull request Nov 3, 2016
Add buffer.transcode(source, from, to) method. Primarily uses ICU
to transcode a buffer's content from one of Node.js' supported
encodings to another.

Originally part of a proposal to add a new unicode module. Decided
to refactor the approach towrds individual PRs without a new module.

Refs: #8075
PR-URL: #9038
Reviewed-By: Anna Henningsen <anna@addaleax.net>

Copy link
Copy Markdown
Member

If this is backported to any of the other release lines, it needs to come with #9838

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

buffer Issues and PRs related to the buffer subsystem. build Issues and PRs related to build files or the CI. c++ Issues and PRs that require attention from people who are familiar with C++. semver-minor PRs that contain new features and should be released in the next minor version. tools Issues and PRs related to the tools directory.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants


Back | FazBrowse Home | New Git URL