Repository navigation
Mac utf8 Rendering Issue #515
Description
Scrolling through a vertical vim split that has a lot of multibyte utf8 characters causes horizontal overflows.
Mac OS X 10.11.6
Tried using both tmux 2.2 and HEAD (installed with homebrew)
$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL="en_US.UTF-8"$ echo $TERM
tmux-256colorHad same results with non-italics $TERM screen-256color.
Issue occurs in both Terminal.app and iTerm2.app and in both nvim and vim.
Running :redraw! (in vim) fixes the issue. Running refresh-client (in tmux) never fixes it.
The issue does not occur outside of tmux.
Activity
nicm commented on Aug 25, 2016
OS X has many missing widths for Unicode characters, tmux will assume width 1 for those. I guess you are using some of them that should actually be 2, but without a log it is hard to tell.
joshuarubin commented on Aug 25, 2016
I'm happy to provide data for you to reproduce. What do you need from me? What kind of logs?
nicm commented on Aug 25, 2016
tmux logs with -vvvv
Make sure you kill tmux entirety and start with -vvvv to get the server log
On 25 Aug 2016 7:11 p.m., "Joshua Rubin" notifications@github.com wrote:
joshuarubin commented on Aug 29, 2016
What was the issue?
nicm commented on Aug 29, 2016
No logs
On 29 Aug 2016 2:49 p.m., "Joshua Rubin" notifications@github.com wrote:
joshuarubin commented on Aug 29, 2016
Huh, I thought I posted them, not sure what happened, here:
There were also issues with this char in my status line (unicode 2705): ✅
That caused some wcwidth() == -1 issues in the log. I removed that and still had drawing issues.
Also, you might want to warn people that the logs contain their env vars. I had things like my github api key there. I had to jump through some hoops to sanitize the logs to be safe to post publicly,
Here are the logs.
tmux-server-23676.log.txt
nicm commented on Aug 29, 2016
This log is from tmux 2.2? Please reproduce using a tmux built from Git
master. Anyway, there are no obvious problems in the log and it is too
big to tell anything specific.
Please give me a smaller log - stop tmux entirely, then start tmux with
-vvv, print ONE of the problem characters or at most a small sample,
then exit tmux and send me that log. Also please tell me what characters
you are seeing the problem with.
On Mon, Aug 29, 2016 at 06:58:56AM -0700, Joshua Rubin wrote:
joshuarubin commented on Aug 29, 2016
OK, here is the log with only 3 -v and my tmux plugins disabled. Probably still too big for you. I have no way of printing just the problem char since I don't know what it is.
tmux-server-37901.log.txt
Here is the file I use to reproduce the problem. In tmux, open vim with this file (might want to rename to .go), make a vim vertical split and scroll up and down and you'll see rendering issues.
lex_test.go.txt
joshuarubin commented on Aug 29, 2016
oh, btw:
This is built using homebrew --HEAD
nicm commented on Aug 29, 2016
Can you cat this file OUTSIDE tmux please and show me a screenshot of what it looks like? x.txt
joshuarubin commented on Aug 29, 2016
OK, here you go.
nicm commented on Aug 29, 2016
can you do this please:
echo "aaa$(cat x.txt)bbb"
nicm commented on Aug 29, 2016
Never mind that actually, the problem is as I suspected.
nicm commented on Aug 29, 2016
So the problem is that there are lots of width 0 characters in this file, but your platform is telling tmux they are width 1, for example:
1472485122.766917 input_utf8_close 3 '\340\262\277' (width 1)
Usually the problem is that OS X just tells tmux that the character has no width and tmux assumes 1, but in this case wcwidth() is actually telling us it is width 1.
This particular character is U+0CBF which was introduced with Unicode 1.1 in 1993.
joshuarubin commented on Aug 29, 2016
That's great, thank you! What can be done about it?
Also, I removed the line with that char and there are still rendering issues.
nicm commented on Aug 29, 2016
You can try to report this to Apple.
These are the zero width characters in the file, you will probably need to remove some of the others:
\340\244\202
\340\245\202
\340\245\207
\340\245\210
\340\262\277
\340\263\215
\340\270\261
\340\270\265
\340\271\211
\342\200\215
joshuarubin commented on Aug 29, 2016
OK, so I assume this issue can be considered "won't fix"?
For posterity, these chars are:
U+0902: ं (devanagari sign anusvara)
U+0942: ू (devanagari vowel sign uu)
U+0947: े (devanagari vowel sign e)
U+0948: ै (devanagari vowel sign ai)
U+0CBF: ಿ (kannada vowel sign i)
U+0CCD: ್ (kannada sign virama)
U+0E31: ั (thai character mai han-akat)
U+0E35: ี (thai character sara ii)
U+0E49: ้ (thai character mai tho)
U+200D: (zero width joiner)
nicm commented on Aug 29, 2016
The fix needs to happen in the system locale implementation.
nicm commented on Aug 29, 2016
There was some talk of providing a version of wcwidth()/mbtowc() using utf8proc (http://julialang.org/utf8proc/) but nothing ever came of it and I don't have any plans to do it.
joshuarubin commented on Aug 29, 2016
OK, understood, thank you for your time. Just as one more last ditch attempt, would it be possible to include a different implementation of wcwidth (if only on mac)? Here is a similar discussion from zsh:
http://www.zsh.org/mla/workers/2008/msg00505.html
Which suggests using this implementation:
http://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c
nicm commented on Aug 29, 2016
Markus Kuhn's implementation is not actively maintained AFAIK and it lacks mbtowc().
joshuarubin commented on Aug 29, 2016
I'd be happy to try to integrate utf8proc. Is it only to replace the wcwidth and mbtowc functions? If so, that shouldn't be too hard and I can make a pull request. If there is more to it, I'd still be happy to help if you can give me a bit of guidance about what else would need to be updated.
nicm commented on Aug 29, 2016
Yes those are the only two functions that tmux uses (in utf8.c). Note that this should not replace them on platforms where they are working, just a single file in compat/ that provides alternative implementations of wcwidth() and mbtowc() to use on platforms with a broken or missing locale implementation.
lock commented on Feb 16, 2020
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.