Was looking for kitty support in tmux and saw this PR. Not sure if i'm doing something wrong, but getting:

❯ ./configure --with-image-support
...
❯ make
make: *** No rule to make target `image-kitty.c', needed by `image-kitty.o'.  Stop.

Oops sorry, forgot to add the new files. Added now. Please try again.

Both kitty and sixel images seem to work. Plese let me know. There's also a rudimentary fallback to ascii in terminals with no image support at all.

@senthil-instrumentl found another missing .h file which is now fixed (included inside image-kitty.h). Please give it a try. Feedback welcome. Thanks!

I am using it inside Windows Terminal for development. I tried it briefly in kitty term. I fixed a flashing issue but that was already pushed yesterday. I can try it in wezterm probably tomorrow.

Can you attach the png file or tell me where to get it?

Hopefully we can fix the flashing everywhere. We will get this to work. Thanks for your help testing it!

I Deleted my previous comments. Apologies for the email noise. My previous tests were invalid.

However the PR sill does not work as expected for me. The images are rendered at incorrect position and do not move as the terminal scrolls. It also causes occasionally the statusbar to disappear when rendering or replaced by newline symbol.

.tmux.conf has

set -ga terminal-features ',*:RGB:sixel:sync:kitty'

Example:

Using chafa produces similar results.

I can not get kitty to work either. It will either do nothing or report unsupported.

❯ kitty +kitten icat ~/Pictures/bitwig_layer_transparency.png

~/Pictures
❯ kitty +kitten icat ~/Pictures/bitwig_layer_transparency.png
Error: This terminal does not support the graphics protocol use a terminal such as kitty, WezTerm or Konsole that does. If you are running inside a terminal multiplexer such as tmux or screen that might be interfering as well.

I am using WezTerm built from master couple days ago. All of these work flawlessly outside tmux.

I reproduced this in wezterm locally. I've added a fix for this to this branch, please pull it and recompile.

IMPORTANT: you should remove this from your .tmux.conf

set -ga terminal-features ',*:RGB:sixel:sync:kitty'

Then fully restart the tmux server. tmux will detect each attached terminal separately:

• WezTerm: RGB plus sixel.
• Kitty: RGB plus Kitty placeholders.
• Non-image terminals: text fallback.

If you have a separate reason to force sync, it should be configured only for the appropriate terminal pattern, not with *. The global line incorrectly claims every client supports both image protocols, which defeats per-client backend selection.

I like the initiative, but there are a few bugs.
First, a minor one: HLS conversion has the usual bug where it doesn't match Sixel semantics. (Hue is rotated -120 degrees compared to common use today, test with img2sixel -t hls ...). Also, rounding.

diff --git a/image-sixel.c b/image-sixel.c
index 577ba51a..595c3836 100644
--- a/image-sixel.c
+++ b/image-sixel.c
@@ -429,9 +429,9 @@ sixel_colour_to_rgb(u_int colour, u_char *r, u_char *g, u_char *b)
 	double	h, l, s, p, q;
 
 	if (type == 2) {
-		*r = ((colour >> 16) & 0x1ff) * 255 / 100;
-		*g = ((colour >> 8) & 0xff) * 255 / 100;
-		*b = (colour & 0xff) * 255 / 100;
+		*r = (((colour >> 16) & 0xff) * 255 + 50) / 100;
+		*g = (((colour >> 8) & 0xff) * 255 + 50) / 100;
+		*b = ((colour & 0xff) * 255 + 50) / 100;
 		return;
 	}
 	if (type != 1) {
@@ -448,9 +448,9 @@ sixel_colour_to_rgb(u_int colour, u_char *r, u_char *g, u_char *b)
 	}
 	q = l < 0.5 ? l * (1 + s) : l + s - l * s;
 	p = 2 * l - q;
-	*r = sixel_hue(p, q, h + 1.0 / 3) * 255;
-	*g = sixel_hue(p, q, h) * 255;
-	*b = sixel_hue(p, q, h - 1.0 / 3) * 255;
+	*r = sixel_hue(p, q, h) * 255 + 0.5;
+	*g = sixel_hue(p, q, h - 1.0 / 3) * 255 + 0.5;
+	*b = sixel_hue(p, q, h + 1.0 / 3) * 255 + 0.5;
 }
 
 /* Convert decoded SIXEL data into the protocol-neutral immutable image. */

Second, sixel_from_image is... not pretty. The way you'd normally go about this is some kind of image quantization algorithm (I've had luck with octrees), with the current nearest neighbor approach the output looks awful, as demonstrated by your own screenshot. Or at the very least you should apply some error diffusion, ref.

Third, sizing is broken. For example,

printf '\033P0;1q"1;1;26;26#0;2;50;50;50#1;2;0;0;0#2;2;49;49;49#3;2;33;33;33#4;2;47;47;47#5;2;44;44;44#6;2;48;48;48#2?O?C!5?S!4?__?_?__$#0?_wW!5KGwo_-#0?~~!8?!10@BB}w$#2!12?a_A?A?A?A??@E-#0?~~!7?KK~~KK!7?~~$#2!9?K!6?K-#0?F^[owowowowowowowowooS^F$#2?G!10?@@!7?G??G-\033\'

should print a 26x26 image, but it gets scaled up to the cell size, so if I have 20x10 pixel cells then it blows up to 40x30. From a quick look at the code, draw_rectangle only gets sizes in cells, so perhaps that's related.

Finally, maybe I'm wrong but it looks like the cropping Kitty parameters (X/Y/w/h (edit: sorry, I meant x/y, although I use X/Y too but it's less important)) aren't respected? Those are kind of necessary for efficient TUIs, e.g. I use them extensively in chawan so that I don't have to re-send images on scroll.

• Correct DEC SIXEL HLS hue ordering and rounding in image-sixel.c.
• Preserve original pixel dimensions so partial edge cells are not stretched.
• Adaptive median-cut colour quantization with Floyd–Steinberg error diffusion.
• Kitty’s actual source-cropping keys, lowercase x/y/w/h, plus c/r placement sizing.
  Uppercase X/Y are destination offsets within the first cell, not crop coordinates,
  according to the Kitty protocol (https://sw.kovidgoyal.net/kitty/graphics-protocol/).
• Cropped Kitty placements use zero-copy immutable pixel views in image.c, and row-
  strided views are correctly streamed when uploaded.

@bptato, Did this address all your concerns? The sixel image definitely looks better to me now, thanks!

It's better, thanks. Some issues left:

• In my Sixel encoder I also do error diffusion on the alpha channel; the result looks a bit strange, but at least when you have an image with alpha<128 everywhere you don't end up with an invisible image.
• Kitty input is still wonky, e.g. with cha -o buffer.images=true https://developers.google.com/speed/webp/gallery1 you'll see partial images flowing into each other etc. At least part of this may be because the cursor is desync'ing, I think if you emulate the regular protocol with the unicode placeholders then you'll have to move the cursor back after output. (But I may be wrong here.)

Also, even with the error diffusion there are some pretty bad artifacts (image, up: kitty -> tmux -> sixel on this branch, bottom: sixel passed through tmux master):

Apples to oranges of course, as the palettizer algorithm is different. But you can see the stripes per line, which I think happens because the palette & dither is recomputed for each line. At least the latter must be kept.
(That aside, I'm a bit suspicious about "Sixel-per-line"'s performance characteristics, ref. Would it be possible to output the whole thing in one go?)

Wow but it's really close! I see it gets dithery above his head and up his arm. What do you think is causing that? I can try to figure it out but if you have some ideas that would be super helpful. Have a look in image-sixel.c and see if it makes sense.

This is the first step to get this to paint the image in stripes because that's the way the current screen-redraw works. It splits the screen up into a "scene" of "spans". When each span is output, the sub-rectangle of the image is output in that stripe. This way, we treat the image just exactly like characters on the screen. They can be scrolled up into the copy buffer, moved around on the screen in floating panes, and partially obscured by floating panes or popups, everything just works.

In a near future revision of this, I hope to be able to join up ajoining spans into larger rectangles. Still though, an image may be painted with multiple sub-rectangles if it is partly obscured by one or more floating panes. But it should improve the efficiency of sending these images to the terminal.

But I don't think this is the solution to this image aberation. It looks like either an encoder or decoder issue and it well could be the encoding into these stripes that is causing this.

Can you show me some of the other examples of images flowing into one another? It sounds like the sub-rectangles are not properly scaled for the terminal or something like that.

By the way, this idea of using small rectangles is based on how Kitty does images. Each of those special unicode characters carries with it which piece of which puzzle belongs on top of that character.

Rendering is pretty broken atm. Example:

Original:

I still can not get kitty images to show within tmux even after correcting my .tmux.conf per instructions. I tried with default config and no dice with that either.

Wow but it's really close! I see it gets dithery above his head and up his arm. What do you think is causing that? I can try to figure it out but if you have some ideas that would be super helpful. Have a look in image-sixel.c and see if it makes sense.

Sorry, I wasn't really clear about it:

1. Right now, the dither starts from scratch for every (terminal grid) line, meaning the error is periodically zeroed out. Somehow the current/next arrays should be retained.
2. Similarly, recomputing the palette for every line makes color approximation unstable; ideally it should be computed for the entire image at once (maybe store it in struct image?)

That should make the stripes disappear.
(Related: point 2 also makes an optimization possible where, when receiving Sixel input, you just keep a copy of the original palette in case you ever need Sixel output again. Then sixel -> tmux -> sixel would stay lossless.)

This is the first step to get this to paint the image in stripes because that's the way the current screen-redraw works. It splits the screen up into a "scene" of "spans". When each span is output, the sub-rectangle of the image is output in that stripe. This way, we treat the image just exactly like characters on the screen. They can be scrolled up into the copy buffer, moved around on the screen in floating panes, and partially obscured by floating panes or popups, everything just works.

In a near future revision of this, I hope to be able to join up ajoining spans into larger rectangles. Still though, an image may be painted with multiple sub-rectangles if it is partly obscured by one or more floating panes. But it should improve the efficiency of sending these images to the terminal.

That makes a lot of sense, thanks.

Can you show me some of the other examples of images flowing into one another? It sounds like the sub-rectangles are not properly scaled for the terminal or something like that.

With the aforementioned command, after scrolling around the page using Kitty 0.48.0 for a bit:

If I keep scrolling, eventually the entire layout collapses into a mess. I get similar results with xterm, and setting cha output to sixel (-o display.image-mode=sixel) mostly alleviates the issue, hence my suspicion that it's a cursor desync in the Kitty input handler. But that doesn't explain the stray image remnants, so there's probably more to it...

(I say mostly; there are still some minor sizing problems in Sixel too. I'll try to post a reduced example later. (edit: never mind, Sixel input seems to work correctly, only Kitty has issues.))

@bptato let me know if this last push fixes the sixel issues.

Well, it fixes it for Sixel input. But quantization is still broken for kitty -> tmux -> sixel (image):

What is the algorithm based off of, anyway?

By the way, I've found the source of my Kitty cursor positioning issue: the patch doesn't respect 'C=1', which is supposed to keep the cursor where it was. I suppose it should set a flag and suppress cursor movement in image_write.

Apropos image_write, I've noticed this:

	sx = im->sx;
	if (sx > screen_size_x(s) - cx)
		sx = screen_size_x(s) - cx;
	sy = im->sy;
	if (sy > screen_size_y(s) - 1)
		sy = screen_size_y(s) - 1; // <-- ??
	if (sx == 0 || sy == 0)
		return;

	if (screen_size_y(s) - cy <= sy) {
		lines = sy - (screen_size_y(s) - cy) + 1;
		screen_write_scrollup(ctx, lines, bg);
		if (lines > cy)
			screen_write_cursormove(ctx, -1, 0, 0);
		else
			screen_write_cursormove(ctx, -1, cy - lines, 0);
		cy = s->cy;
	}

Surely we don't want to crop the image to the screen height when moving the cursor?

I believe what should happen here is that if the image overflows the screen, it should first scroll down by lines, and then the image should be cropped from the top so it fits into the screen (i.e. set some kind of "y origin" variable).
Except of course if cursor movement is suppressed, in that case we can crop at the bottom and skip scrolling.

@bptato Ok I made progress on the kitty images and the kitty->tmux->sixel support. Can you please restest this branch.

I can't seem to reproduce this issue: "scrolling, eventually the entire layout collapses into a mess". If it's still messing up when you scroll up and down, please can you provide me with a list of instructions how to reproduce that? I have the 1.wepb image you used. If you used a different image, please provide me a link.

I tested in kitty with this:

kitty +kitten icat --scale-up --fit=both 1.webp

to make the image fit full width.

Ok I made progress on the kitty images

C=1 indeed works now, thanks.

and the kitty->tmux->sixel support.

From what I can see, you removed the dither, which I find puzzling because that's the only part that worked. Now the image looks like:

The part that is broken is the quantization, i.e., your (?) median cut routine. In other words, the colors are being selected incorrectly.

I can't seem to reproduce this issue: "scrolling, eventually the entire layout collapses into a mess". If it's still messing up when you scroll up and down, please can you provide me with a list of instructions how to reproduce that? I have the 1.wepb image you used. If you used a different image, please provide me a link.

The bug(s) I was talking about manifested when viewing the webp gallery in chawan as described above. By "scroll" I referred to scrolling inside the TUI, which in the background both scrolls the text and moves the placements. (This is necessary because every terminal I tested handled scroll + Kitty placements differently.)
Anyway, layout being messed up was caused by the lack of C=1 and is now fixed. Remnants of images still linger on the screen, which seems to be related to moved placements not being deleted (?); I'll try to reduce that too.

I tested in kitty with this:

kitty +kitten icat --scale-up --fit=both 1.webp

I'm pretty sure icat discriminates against tmux, so this would use the passthrough sequence by default, i.e., this doesn't really test anything. (I've tried passing --transfer-mode=stream --passthrough=none but then it doesn't print anything for some reason.)

I'll try to reduce that too.

#!/bin/sh
printf '\033_GC=1,s=26,v=26,p=1,q=2,i=1,a=T,f=100,m=0;iVBORw0KGgoAAAANSUhEUgAAABoAAAAaCAYAAACpSkzOAAACmUlEQVR4XmOgF2CEWVRfX8/w//9/BkZGRgy7YeKNjY1kuwtsam1tLQMTE5MG0MBsoEWW6KYBxf8DxScD6UVNTU1kWcZYU1MD0qjLzMx8EmgYJy5TgJb8/ffvn+2vX7+Od3Z2kmwZC9AnIE0FIEuAhl0H4ulA9k9kk4Bi4UB1TkA8H2iRYXl5+Xd2dna8loGCG2Q2LLgZQXED1PEIaLjs379/g4Ds9VDLwQYBxRiAPpFmZWW9DFQjCDRgLxDfJeQloJpdQHPWA9X9A1kGs+gZ0BBJoKQtUOIIeqSDgheoKR6IF5ASZkAH3gKaCzLzFQsxGltaWhjq6uoWguIJ6hi82oCG8wIV+AEdpga0LBfIrmUh1oWg1AYM5iVA9UuI1JMGVDcTiCNIsghkOChIQ0NDwfasXr0ap33QeL8GUgD0HTeIZmIgAUAtYQZqYYZZSKx2oi2CWaKpqbkIhEm1jOg40tLSAjleEBgUUSAGkJ8PpN8Q6yOCFoF8AvQBTvOAqZHh+vXrDPjijGAcIcXJUqBPQOXda5iNQP5rkBiQv5SYYMTrIxUVFZC5IkDCCk8QWQHVgdS8xBeMeBNDe3s7A7CwfQl0uSmwKDIBYhdQGQbCIDZIDCQHUgNSS7ZFII2gUgEU6UADzwJz+kUgDa6zoOyzIDmoGgaKLAJpBheKkArxC7BIeQDCQPEvIEuJrQyJzkdQA38ADbcHYaBFP0ipcVGqCWDYBwPxOkYs1TkDCeA/pEkQBDRnLZD9GKhVjgUkCGTsBgomAekWIJYG4p8MlAFQrZgJMgJo/m4QzQJ1fT9QIBLI1gTiSRRaAm/gAM38DjSvH2QeuHECLW1xNk7IsRhoyXGgJVOBem+AExPMEHzNLTItYiAlVTJQCwAAJxkMkgbHo2UAAAAASUVORK5CYII=\033\\\n'
sleep 1
printf '\033_GC=1,s=26,v=26,p=1,q=2,i=1,a=p;\033\\\n'

The first printf should draw the icon, then the second one should move it to the next line.
So I guess it's happening because the patch doesn't handle placement ids (in this case, p=1). ref

I've reverted ab591ae and hopefully now I have fixed the quantization, sorry about that misunderstanding. Hopefully the quantization bug is fixed.

I tested both using icat in kitty and displaying the image in tmux in ms terminal which output using sixel and visa versa I cat'ed out a sixel image in ms term and it displayed in kitty. scrolling the tmux window works but that isn't quite what you were doing.

I will see if i can reproduce what you describe tomorrow. Meanwhile, tell me if I got the quantization bug this time. I feel we're getting closer. Thanks for help testing, very helpful!

@bptato I ran your shell script:

it didn't "move" the icon down, it waited a second then it printed a second icon. Is that what you expected?

@bptato Theicat test was useful for the new Kitty→tmux→SIXEL interoperability because we now deliberately intercept that wrapped placeholder form and convert it into shared tmux image cells.

I pushed another fix to hopefully fix or improve the scrolling in chawan, pleases let me know if it's any better or worse.

After my last fix, your script indeed makes the little icon folder move down (I see only one icon now).

I will see if i can reproduce what you describe tomorrow. Meanwhile, tell me if I got the quantization bug this time.

It seems to be working correctly now, thanks.

I pushed another fix to hopefully fix or improve the scrolling in chawan, pleases let me know if it's any better or worse.

Something is still off. I can't reliably reproduce this, but following script:

#!/bin/sh
reset
tput cup 0 0
printf 'test\n'
tput cup $(($(tput lines) - 1)) 0
printf 'hi world\n'
printf '\033_GC=1,s=26,v=26,p=1,q=2,i=1,a=T,f=100,m=0;iVBORw0KGgoAAAANSUhEUgAAABoAAAAaCAYAAACpSkzOAAACmUlEQVR4XmOgF2CEWVRfX8/w//9/BkZGRgy7YeKNjY1kuwtsam1tLQMTE5MG0MBsoEWW6KYBxf8DxScD6UVNTU1kWcZYU1MD0qjLzMx8EmgYJy5TgJb8/ffvn+2vX7+Od3Z2kmwZC9AnIE0FIEuAhl0H4ulA9k9kk4Bi4UB1TkA8H2iRYXl5+Xd2dna8loGCG2Q2LLgZQXED1PEIaLjs379/g4Ds9VDLwQYBxRiAPpFmZWW9DFQjCDRgLxDfJeQloJpdQHPWA9X9A1kGs+gZ0BBJoKQtUOIIeqSDgheoKR6IF5ASZkAH3gKaCzLzFQsxGltaWhjq6uoWguIJ6hi82oCG8wIV+AEdpga0LBfIrmUh1oWg1AYM5iVA9UuI1JMGVDcTiCNIsghkOChIQ0NDwfasXr0ap33QeL8GUgD0HTeIZmIgAUAtYQZqYYZZSKx2oi2CWaKpqbkIhEm1jOg40tLSAjleEBgUUSAGkJ8PpN8Q6yOCFoF8AvQBTvOAqZHh+vXrDPjijGAcIcXJUqBPQOXda5iNQP5rkBiQv5SYYMTrIxUVFZC5IkDCCk8QWQHVgdS8xBeMeBNDe3s7A7CwfQl0uSmwKDIBYhdQGQbCIDZIDCQHUgNSS7ZFII2gUgEU6UADzwJz+kUgDa6zoOyzIDmoGgaKLAJpBheKkArxC7BIeQDCQPEvIEuJrQyJzkdQA38ADbcHYaBFP0ipcVGqCWDYBwPxOkYs1TkDCeA/pEkQBDRnLZD9GKhVjgUkCGTsBgomAekWIJYG4p8MlAFQrZgJMgJo/m4QzQJ1fT9QIBLI1gTiSRRaAm/gAM38DjSvH2QeuHECLW1xNk7IsRhoyXGgJVOBem+AExPMEHzNLTItYiAlVTJQCwAAJxkMkgbHo2UAAAAASUVORK5CYII=\033\\'
printf '\n'
sleep 1
printf '\033_GC=1,s=26,v=26,p=1,q=2,i=1,a=p;\033\\\n'
sleep 1

sometimes leaves the word "hi" under the icon, like:

Keyword sometimes; in about 50% the cases it displays correctly. Happens in xterm, wezterm, yafy (but not in kitty, so it's probably a sixel output issue.)

Edit: although thinking of it more, this is probably unrelated to the (well, another) problem I still see with cha, as the script depends on scroll. I'll investigate...

Theicat test was useful for the new Kitty→tmux→SIXEL interoperability because we now deliberately intercept that wrapped placeholder form and convert it into shared tmux image cells.

Well, you added that after my post :)
Anyway, what is your rationale for it? Personally I would expect a passthrough sequence to, well, pass things through.
(Such a kludge will also break things that depend on full support for the protocol, which this patch doesn't do.)

Well, you added that after my post :)
Anyway, what is your rationale for it? Personally I would expect a passthrough sequence to, well, pass things through.
(Such a kludge will also break things that depend on full support for the protocol, which this patch doesn't do.)

Ok, i see what you mean, let me reverse that. What do you think should happen on a simultaneously connected sixel compatible terminal (not kitty)? Should we try to passively look at the passthrough image and if possible try to create a sixel shadow image of it for the sixel capable terminal (for example Microsoft Windows Terminal)? Because otherwise, those other terminals will just see a gaping hole. This won't support everything you could throw at kitty like animations but it would support the majority of the static images which seems useful. Not the easiest thing to do but it should be possible. What's your suggestion in this case?

Ok, i see what you mean, let me reverse that. What do you think should happen on a simultaneously connected sixel compatible terminal (not kitty)?

I'd say "nothing". We shouldn't incentivize apps to stick to an obsolete hack by adding another hack on top of it.

The right way to fix this is to patch terminal libraries (that don't already do so) to query for kitty support without passthrough as well. Then, if they get a regular response, they can just send the image as normal, and optionally fall back to passthrough for older tmux versions as long as that's relevant.

@bptato some major changes in the 4902-image-support branch. Would you mind retesting your application and let me know if everything still works? Basically this branch is based on the redraw-damage-rectangles branch which tracks damage to the screen and redraws only the necessary spans. It should substantially reduce the amount of flashing of images.

It seems to work fine, thank you.

(Although if the goal was to eliminate flashing, that still happens with Sixel, except when Sync hides it - in xterm it's visible even when I just dump an image to the shell with img2sixel and scroll by repeatedly hitting return.)

How can I compile this correctly, seems like --with-image-support no longer works:

"configure: WARNING: unrecognized options: --with-image-support"

Currently when I test this PR sixel graphics do not show up at all.

Based on logs the --enable-sixel is deprecated as well.