Add support for pre-multiplied alpha #3391
Description
At the bottom of this pull request (#2601) @benfry mentioned opening an issue about implementing pre-multiplied alpha, which I don't think was ever done? I certainly dropped the ball on following up. It would be great if an official image mode for this was added (I'm currently sub-classing the GL renderer to achieve this). Otherwise multiple layering (as well as convolution, etc) don't work correctly.
For reference, a long and admittedly opinionated extract of an email sent to @JakubValtar last year with a suggestion for handling this internally in the OpenGL renderer - feel free to TL;DR, although the NVIDIA link is worth a read.
OK, you asked me how Processing might use pre-multiplied alpha without
breaking the API. So to start, a possibly controversial statement -
blending in Processing is completely broken. It was one of the
reasons I stopped using Processing the first time around (~2006) - I
like the API but in my opinion there are some issues with the
implementation. Praxis LIVE v1 deliberately made use of aspects of
the Processing API, but re-implemented on top of a custom software and
OpenGL renderer. Fast forward to a year ago, and I decided to stop
trying to re-implement the API - the new Processing OpenGL is too good
to miss! So Praxis LIVE v2 now embeds Processing as a library -
however, there's no way I'm breaking blending in the process - luckily
it isn't that difficult to work around.
There are many sources around about pre-multipled alpha. I found this
page recently looking into GL blend modes
(http://www.slideshare.net/Mark_Kilgard/blend-modes-for-opengl) - it's
about the new NVIDIA blend mode extensions - note that slide 16 talks
about the fact that blend modes assume pre-multiplied alpha. It
mentions one of the key issues (one that affects me) - drawing into
surfaces with alpha channels - sometimes you can get away with it if
the destination is opaque. He also mentions the Porter-Duff model and
how it relies on pre-multiplied alpha - interesting that the default
blending in the Java2D renderer works fine - because the JRE
AlphaComposite pre-multiplies, blends and un-premultiplies.
Incidentally, that slide also touches on another issue -
pre-multiplied alpha is necessary for performing various filtering
effects (eg. blur) on surfaces with alpha. I haven't looked at the
PImage filters but I imagine there might be an issue there (note the
ConvolveOp in the JRE multiplies, filters and unmultiplies).
Now, all that multiplying and unmultiplying is a bit pointless! ;-)
Not to mention it's not easily achievable with OpenGL. So, how to use
pre-multiplied alpha without changing the API?
The important thing in my opinion is separating the external format of
images to the format they're stored internally. Take managed images
in the Java2D backend - cached textures are not stored in the format
of the BufferedImage, but the optimum format for rendering. Just
because the format to the user is un-premultiplied ARGB doesn't mean
that the underlying storage has to be. Processing could easily switch
to storing the underlying pixel data, in all modes, as pre-multiplied.
Or, simpler, though not fixing issues elsewhere, it could just store
pre-multiplied data in the OpenGL textures.
The key things to handle are times when the user is setting colours,
times when the user is retrieving colours, and access to the pixel
array.
Processing already does colour calculations to match a colour format
set by the user and convert into internal data. What I've done in
Praxis LIVE's PGraphics sub-class is override the various colorCalc()
methods to multiply in the alpha. That's pretty much all that I'm
using right now, but then I don't have 100% API coverage! It would be
possible to handle multiply and unpremultiply fairly easily in set()
and get(), which leaves the pixel array - I'd suggest running a
un-premultiply <> premultiply during loadPixels() and updatePixels(),
with an extra hint to leave it premultiplied and save any hit (of
course RGB images wouldn't have any anyway).
I'm not quite sure if there would be any implications to PShape that
would need looking at - I guess I'll find out when I come to
integrating support for them! ;-)
The one aspect of the API that would change would be GLSL - mind you
I'd assert that's fixing it rather than breaking it - however, it's
the one place that might affect the behaviour of user code.
Incidentally, if it's of interest - some alternate software blend
modes that I and another person on the Java Gaming forum worked on a
few years back - all using pre-multiplied alpha, obviously -
http://www.java-gaming.org/topics/some-new-blendmodes-add-multiply-overlay-etc/24529/view.html
Activity
neilcsmith-net commented on Jun 17, 2015
To follow up with a simple, quick sketch that demonstrates the issue - comment out renderer as appropriate.
Result : incorrect OpenGL behaviour top, correct (seamless) Java2D behaviour below.
Incidentally, I believe that pg.clear() shouldn't be required, but P2D gives another incorrect result without it (veering to grey).
JakubValtar commented on Jun 17, 2015
First of all P2D and Java2D use exactly the same blending function. Just initial colors and alpha channels are different, that's why your examples behave the way you observe.
You are correct in that Blend mode being glBlendFunc(SRC_ALPHA, ONE_MINUS_SRC_ALPHA) ignores the destination alpha, so if destination has 0 alpha, the destination color will still bleed into the result because (1 - SRC_ALPHA) is used as the factor. However, Java2D does the same, so it is consistent across renderers. You are also correct that this could be changed by using pre-multiplied alpha and rewriting the Java2D blending functions.
Your example, however, shows different issue. Java2D off-screen buffer is for some reason always opaque (might be a bug), that's why (255, 0, 0, 127) over (0, 0, 0, 255) = (127, 0, 0, 255). When you draw this on primary surface, you get (127, 0, 0, 255) over (0, 0, 0, 255) = (127, 0, 0, 255).
In case of P2D, clear() is just background(0, 0), so you are making the surface transparent (0, 0, 0, 0). (The reason for color drifting towards grey is that P2D has initial color (204, 204, 204, 0) - probably a bug again, I would expect it to be transparent black. Calling background(0, 0) or clear() before drawing should fix it.) Blending goes as follows: (255, 0, 0, 127) over (0, 0, 0, 0) = (127, 0, 0, 127). When drawn to primary surface, (127, 0, 0, 127) over (0, 0, 0, 255) = (63, 0, 0, 255). If you replace clear() with background(0, 255) to make the initial color same as Java2D, you get the same result as Java2D.
P.S. You can use println(hex(pg.get(int x, int y))) to print out the color of any pixel in any PGraphics.
neilcsmith-net commented on Jun 17, 2015
You're right about Java2D surface being opaque in 3.0a10. This is a regression from 2.2.1 where it is transparent. In either version, the output of that sketch in Java2D mode is the same! It does not change the fact that blending in OpenGL is broken - the colour should be seamless in P2D and it is not.
The output of drawing (255,0,0,127) over (0,0,0,0) should be (255,0,0,127) assuming you want the surface to be not pre-multiplied. The entire problem is that OpenGL cannot un-pre-multiply the colour value, so each time you draw it the colour gets multiplied by the alpha again.
JakubValtar commented on Jun 17, 2015
Please don't use 2.2.1 Java2D renderer as a reference. It was broken to such a degree that source and destination colors were swapped and random bits from memory were used as destination alpha.
Java2D and P2D now use the same blending function, so they are both broken. I believe this is the best we can do without pre-multiplied alpha.
To sumarize, we have:
neilcsmith-net commented on Jun 17, 2015
I'm actually using expected behaviour as the reference!
Why do you believe this is the case?
Java2D default blending uses SrcOver, which is equivalent to -
P2D is doing glBlendFunc(SRC_ALPHA, ONE_MINUS_SRC_ALPHA) which effectively rolls together 1 & 2, but completely ignores 3, hence the buggy blending. Why is fixing this a feature request?
Incidentally, Java2D AlphaComposite recommends pre-multiplied alpha also on performance grounds because it removes the need to do 1 & 3, but at least Java2D will have the expected outcome.
JakubValtar commented on Jun 17, 2015
Ah sorry, I forgot we are using SRC_OVER AlphaComposite instead of PImage blending function like the rest of blending modes. It is indeed not the same but still closest we can get with this color model.
It is a feature request because current color model is deliberately used with it's limitations and benefits in mind. You request something else which also has it's pros and cons.
neilcsmith-net commented on Jun 17, 2015
Fair enough! Not my call, but it's frustrating when almost every bug report on here around blending is because of this issue. I don't see what benefit justifies the bug above.
I'll keep maintaining my OpenGL renderer subclass for now to ensure my application doesn't break, but happy to get involved and/or push stuff back upstream when there's an active effort to fix this.
Nice title change! :-P
Best wishes, Neil
tillnagel commented on Jul 2, 2015
@neilcsmith-net Care to share your OpenGL renderer subclass, or some pointers on how to work-around this for now?
neilcsmith-net commented on Jul 2, 2015
@tillnagel It's all up in the Praxis LIVE repositories - https://github.com/praxis-live
In particular look at PGLGraphics and the revised implementations of blendModeImpl() and colorCalc(). Incidentally those methods are forks of the Processing ones which isn't reflected in the header - will correct that as soon as I can.
This will only get you part of the way though. Praxis LIVE has its own software renderer which uses premultiplied alpha already. You will also have to make sure the pixel data in the image is premultiplied before uploading to the texture.
RudnikNelson commented on Mar 11, 2018
Coming on close to 3 years of no support for pre-multiplied alpha in Processing 3. I thought I'd post and be a squeaky wheel!
For those who arrive here still having this issue, there's a work around, kind of. You add a "blendMode(REPLACE)" before drawing shapes within a PGraphics element. If you wish to draw an image within PGraphics, you must revert to blendMode(BLEND). This creates a tinted halo in the lower alpha pixels of the image, however. It's subtle, but the more layering of images there are and interactions with those layers, the more obvious, ugly, and problematic it becomes. It might not bother many of the amateurs that Processing attracts, but for professionals, it remains a no go to upgrade to Processing 3 because of this one issue. I'll continue to work with Processing 2 until there is support for proper blending.
Much respect to the team, by the way. Appreciate you endlessly. And Jakub, you turning me onto InteliJ 2.5 years ago changed my life! Processing 2 is amazing. I'm just sore after checking on this issue, that I can't go and upgrade already and reap the benefits of 3. Best Regards
Example showing the work around and resulting bug with image in PGraphics:
`PGraphics pg;
String renderer = JAVA2D;
//String renderer = P2D;
PImage hair;
void setup() {
size(640, 480, renderer);
hair = loadImage("hair2.png");
pg = createGraphics(320, 480);
pg.beginDraw();
pg.clear();
pg.blendMode(REPLACE);
pg.noStroke();
pg.fill(255,0,0,127);
pg.rect(0,0,320,480);
pg.blendMode(BLEND);
pg.image(hair,0,0);
pg.endDraw();
}
void draw() {
background(0);
noStroke();
fill(255,0,0,127);
rect(0,0,320,480);
pushStyle();
blendMode(BLEND);
image(pg,320,0);
popStyle();
image(hair,0,0);
//save("example.jpg");
}`
neilcsmith-net commented on Mar 11, 2018
@RudnikNelson so are you saying the Java2D renderer is now broken too?! This used to be just about the OpenGL renderers because the underlying JDK code for Java2D does handle un-premultiply correctly. In P2D/P3D land things are somewhat better since a switch to using separate blend functions, but still broken in some areas.
Incidentally, if you need to use Processing 3 libs with correctly functioning alpha, come and use them inside Praxis CORE / LIVE instead. And maybe we can get you hooked on a real Java IDE in the process! 😉
cadin commented on May 27, 2018
+1 squeaky wheel.
I'm running into this while trying to update some sketches from 2.x to 3.x.
Java2D and P2D behave the same for me (in 3.3.7).
Processing 3 is not useable for me without this fix.
neilcsmith-net commented on May 27, 2018
@cadin you might want to open another issue completely. This got marked as an enhancement, and the title changed - simple fact is blending is broken in Processing. This isn't about premultiplied vs non-premultiplied - it's that you can't mix them and expect things to work correctly. When I posted this originally it was only an issue in the OpenGL renderers because the output of GL blending is always premultiplied. That that behaviour seems to have been replicated in the Java2D renderer, which at least theoretically could do this correctly, is an issue and why I think you should report it separately as a bug.
The OpenGL renderer in PraxisLIVE is fixed to work correctly, and I'm quite happy to put together a PR for upstream if this ever gets accepted as an actual bug! Also happy to help with the problem in Java2D, although don't use it.
cadin commented on May 29, 2018
@neilcsmith-net Ah, I see. It was a bit hard to follow everything in this thread, but it sounded like my issue. I'll open a separate bug. Thanks.
github-actions commented on Jul 15, 2026
This issue has been automatically closed and locked.
Processing 3.x is no longer actively developed. Since January 2020, development moved to the Processing 4 repository.
If this issue is still relevant to Processing 4, we'd love to hear from you. Please open a new issue at:
https://github.com/processing/processing4/issues/new/choose
Thank you for helping improve Processing.