Repository navigation
[ENH]: Plot overlay #30953
Description
Problem
One limiting factor of interactivity in Matplotlib is that we draw almost all visual elements via Artists onto the figure. This has poor performance for fast changing elements like crosshair cursors, because we always have to redraw the complete figure (or try to fiddle with blitting).
Proposed solution
The fundamental solution idea here is to pull such elements out of the actual figure and instead overlay them on the not-changing figure image.
This is currently in a very early idea stage.
What can be achieved with this
Interactivity with fast changing elments such as:
- Zoom box (already implemented as a special case)
- Crosshair cursors (see also PoC: GUI-native crosshair cursor #30516).
- Tooltips
- Selectors
What still won't be possible
Dynamically changing plot properties like greying out data depending on a selector like https://docs.bokeh.org/en/latest/docs/examples/interaction/linking/linked_brushing.html.
What would be needed
-
A concept of overlays. This could be:
a) GUI specific, e.g. we define a set of native primitives (lines, rectangles, maybe semantic ones like CrosshairCursor, similar to artists) that are drawn by the canvas on top of the actual figure image. This requires implementing of the primitives in all backends and will provide the fastest rendering.
b) Create the concept of an "overlay figure" that uses our exising rendering mechanisms to create a (mostly transparent) full image that is drawn on top of the existing figure image. -
Some mapping and event logic, so e.g. that the overlay can draw a line across a complete Axes, or get tooltip information for the current coordinate.
Activity
scottshambaugh commented on Feb 3, 2026
Potentially related to #2189, by separating draw cycles for different elements.
aman-coder03 commented on Feb 12, 2026
@timhoffm i saw your note on #30516 that the crosshair work will likely move under this broader overlay concept, so i have been reading through this discussion.
Between the two directions you outlined, GUI-native primitives vs. an “overlay figure” using existing rendering, is there currently a preferred direction, or is this still open for exploration?
It seems like (a) would give the best performance but require backend specific implementations, while (b) would be more consistent with existing Artist infrastructure but potentially heavier.
I would be interested in experimenting with a small prototype(maybe a minimal crosshair overlay) to better understand the trade-offs if that would be helpful.
buddy0452004 commented on Mar 8, 2026
@timhoffm
Hi! I'm Rahul Monani, and I've been following this issue with a lot of interest — the overlay concept is exactly the kind of architectural challenge that excites me.
I've been actively contributing to Optuna's visualization module — I have a merged PR modernizing the codebase and an open PR adding user_attrs support to plot_parallel_coordinate. Working on those gave me a solid feel for how visualization libraries handle rendering and where performance bottlenecks hide.
The problem you've described here — redrawing the entire figure just to move a crosshair — is something I find genuinely worth solving. I've started exploring the backend code and the draw cycle to understand where the overlay would plug in.
Quick question: between approach (a) GUI-native primitives and approach (b) overlay figure — is there a preferred direction for GSoC, or is evaluating the tradeoffs itself part of the exploration?
Excited to dig deeper into this!
— Rahul Monani
timhoffm commented on Mar 8, 2026
@buddy0452004 welcome! So far, there is no preferred direction. We haven't analysed the trade-offs in depth yet. If you want to dig deeper, analyzing both approaches and identifying their benefits, limitations and pain points is a good next step. Showing that you are able to analyze the situation on your own and have well-founded solution ideas is a stong proposal for GSoC.
That is quite a mouth full. I hope you have learned some of that as part of digging into the library. The PRs themselves do not substantially touch these topics. Anyway, please note that advertising yourself with words is not needed. We judge people on what the do (e.g. analyzing topics, writing good issues and PRs, way of communication, ...), not what they claim they can.
buddy0452004 commented on Mar 9, 2026
@timhoffm
I spent some time analyzing both approaches by reading backend_bases.py and backend_tools.py. Here are my findings:
Approach (a) GUI-native primitives:
Each backend would need its own implementation of overlay primitives. Looking at the existing backends (Qt, Tk, wx, GTK, macOS), this means 5+ separate implementations to maintain. The existing RubberbandBase follows this pattern — each backend overrides draw_rubberband independently. Fast, but high maintenance cost.
Approach (b) Overlay figure:
This could reuse FigureCanvasBase.blit() which already exists. The overlay would be a transparent RGBA layer composited on top of the frozen figure image. Only one implementation needed. Slightly slower than native but consistent across all backends.
My initial intuition is that (b) is more maintainable and aligns better with matplotlib's existing architecture, but (a) would give the smoothest UX. Would a hybrid make sense starting with (b) as the base implementation, with optional (a) overrides per backend?
buddy0452004 commented on Mar 9, 2026
@timhoffm
I put together a small proof of concept for approach (b) to get a better feel for the trade-offs.
The idea: cache the background with copy_from_bbox() after each full draw, then on every motion_notify_event just restore that background + draw the crosshair lines + blit() only the axes region. No figure.draw() on mouse move at all.
Running it side by side with a naive draw_idle() crosshair, the difference is noticeable — the blit version stays smooth even as you move the mouse quickly, while the naive version lags because it redraws the full figure each time. On my machine, moving the mouse around produced ~300 blit updates with only ~3 full redraws (startup + resize), vs ~300 full redraws for the naive version.
Gist:
https://gist.github.com/buddy0452004/6e40e5f6c70ccd14d1bceee726ffa6ae
A few open questions from digging into this:
copy_from_bbox / restore_region / blit are already on FigureCanvasBase but feel like internal plumbing. Would the overlay API expose something higher-level on top of these, or build directly on them?
When the user zooms or pans, the background cache goes stale. The draw_event hook handles this, but it means the overlay needs to be aware of figure state changes. Is that the right coupling point?
Approach (a) would avoid this entirely since the overlay lives outside the figure's render tree — but at the cost of per-backend implementations. Happy to sketch that out too if useful.
timhoffm commented on Mar 9, 2026
You could also compare the timing with the native implementation #30516. This will give us an estimate how much slower the blit solution is compared to GUI-native draws.
We will need a dedicated "overlay" architecture anyway. If we go with (b) that could become a more powerful architecture. We are not limited to a single overlay but could potentially introduce multiple layers so that different parts could be drawn and updated independently. That could also cover the current biltting logic of widgets and animations. Even more general, it allows to have fast changing updates in the middle of the draw stack; .e.g. with three layers, you could have static expensive to render background and foreground and something fast-changing in between. You could then just re-render the middle part and compose the three layers together.
This is more flexible than GUI-native overlay. In theory, it may even be possible to decide per layer whether to render GUI-native or with the matplotlib backend. It's worth considering such a generalized layered drawing. That's a substantial architecture investigation, but can yield significant benefits. It's complex though: We need some kind of coordination and common understanding of coordiates so that everything is correctly aligned. Order of updates could also play a role.
buddy0452004 commented on Mar 9, 2026
@timhoffm
I ran a timing comparison between the three approaches on my machine (TkAgg backend, Windows, 500-point sine wave).
Blit averaged around 1.35ms per update (~740 FPS), naive draw_idle() came in at ~143ms per update (~7 FPS), and the native-sim which only measures the geometry calculation from #30516 with no matplotlib rendering — averaged 0.16ms (~6110 FPS).
So blit is roughly 106x faster than naive. The gap between blit and native-sim is about 8x, but since native-sim skips all rendering, the real Qt-native overhead will be higher than that so the actual gap is probably smaller.
At ~740 FPS blit feels more than fast enough for smooth interaction. The backend-agnostic nature of blit seems worth that overhead compared to maintaining per-backend native implementations.
Your point about a generalized layered architecture makes sense. If the overlay is designed as a composable layer stack from the start static background, fast-changing middle, foreground the same mechanism could replace the scattered blitting logic in widgets and animations too, rather than solving just the crosshair case.
Benchmark code:
https://gist.github.com/buddy0452004/6e40e5f6c70ccd14d1bceee726ffa6ae
mp-007 commented on Mar 10, 2026
Hi everyone,
Maybe it can interest you but I use plot overly with kivy and Agg backend for one of my github project. My package kivy_matplotib_widget use kivy canvas to solve performance issue for this kind of elements:
Here an example code (100 000pts) and a video to show the behavior.
from kivy.config import Config #avoid conflict between mouse provider and touch (very important with touch device) Config.set('input', 'mouse', 'mouse,disable_on_activity') Config.set('kivy', 'keyboard_mode', '') #disable keyboard mode from kivy.lang import Builder from kivy.app import App from kivy.core.window import Window import numpy as np from kivy_matplotlib_widget.uix.legend_widget import MatplotlibInteractiveLegend from kivy_matplotlib_widget.uix.minmax_widget import add_minmax from kivy_matplotlib_widget.uix.hover_widget import add_hover, HightChartHover import matplotlib.pyplot as plt KV = ''' Screen figure_wgt:figure_wgt BoxLayout: orientation:'vertical' KivyMatplotNavToolbar: id:nav_bar nav_icon:'all' hover_mode:'desktop' show_cursor_data:True figure_wgt:figure_wgt MatplotFigureSubplot: id:figure_wgt auto_cursor:True interactive_axis:True max_hover_rate: 5 /60 ''' def activate_monitor_mode(): """Activate kivy monitor mode (get fps rate information in top bar)""" from kivy.modules import Modules from kivy.config import Config Config.set('modules', 'monitor', '') Modules.activate_module('monitor',Window) class Test(App): def build(self): self.screen=Builder.load_string(KV) return self.screen def on_start(self, *args): fig, ax1 = plt.subplots(1, 1) nb_pts=50000 ax1.plot(np.random.randn(nb_pts),label='line1') ax1.plot(np.random.randn(nb_pts)+2,label='line2') ax1.legend(loc=4) fig.subplots_adjust(left=0.13,top=0.96,right=0.93,bottom=0.2) self.screen.figure_wgt.figure = fig #add hover option, minmx option and interactive legend add_hover(self.screen.figure_wgt,mode='desktop',hover_widget=HightChartHover()) add_minmax(self.screen.figure_wgt) MatplotlibInteractiveLegend(self.screen.figure_wgt) activate_monitor_mode() Test().run()buddy0452004 commented on Mar 16, 2026
@timhoffm just checking in did the benchmark numbers make sense? I've also been thinking about the coordinate alignment problem for the layered architecture. Happy to dig deeper into that if it would be useful.
timhoffm commented on Mar 16, 2026
The native numbers have only limited value. As you mentioned, they leave out rendering. But the computation part of cross-hairs is negligible. So the only thing we learn from that is that native cannot be much faster than blitting, or the other way round, blitting is not too bad in terms of performance.
However, blitting is quite a hassle in state management. I'm inclined that we rather want a layer approach, still may be fast and also does not put us in a corner on native vs. matplotlib artist drawing.
See #30515, in particular #30515 (comment)
buddy0452004 commented on Mar 16, 2026
@timhoffm thanks for the clarification. The point about each layer deciding its own rendering strategy makes sense. For a crosshair overlay, redrawing the full upper layer is cheap since it only contains a few artists, and having direct access to the cached lower layer background is the real win.
I also noticed you just referenced this on #22183 so the layer model would effectively supersede the explicit BlitManager approach too. That makes the scope clearer: rather than patching blitting in individual widgets, a proper layer architecture would give all of them a clean foundation.
Regarding AI tools: I used them to understand unfamiliar parts of the codebase and as a sounding board for ideas, but I read the relevant source code myself and verified everything before using it in my analysis or prototype.
buddy0452004 commented on Mar 16, 2026
@timhoffm based on your feedback about each layer deciding its own rendering strategy, I put together a draft LayerManager prototype.
The design: each Layer owns its artists, caches its own background via copy_from_bbox, and redraws independently. The LayerManager hooks into draw_event and resize_event to handle cache invalidation automatically.
Quick benchmark on TkAgg, 500-point sine wave:
LayerManager crosshair: avg 5.73ms (~174 FPS)
Naive canvas.draw(): avg 40.59ms (~25 FPS)
Gist: https://gist.github.com/buddy0452004/1fa7c24f60a6a80198a7bcdf7f55a934
Happy to extend this multiple layers compositing, coordinate alignment, or per-backend rendering strategies.
timhoffm commented on Mar 17, 2026
Good PoC to show the performance. Eventually, we'd want to refine the architecture. Just spitballing, but a layer should not store its background but rather the layer below should (optinally) store its output, because in the end the knowledge whether the stored image is still valid depends on the artists of the layer the artists are in.
My idea is that a draw would always be a complete top-level command, and the layers should internally figure out what can be cached and what needs to be redrawn. Something like:
Does that sound reasonable?
buddy0452004 commented on Mar 18, 2026
@timhoffm
Yes, that sounds reasonable.
The way I understood it each layer should own its output, not store what's below it. That way the layer itself knows if its cache is valid based on its own artists, rather than relying on the layer above to figure that out.
So the flow I implemented:
Updated the gist with this.
On performance v1 was 174 FPS, v2 is 109 FPS so v1 is faster. But v1 breaks with 3 layers middle layer changes and the top layer has no idea its background is stale. v2 is slower because of the bottom-up query but at least gets the invalidation right. Feels like the right base to build on even if the performance needs work later.
The explicit cache on/off per layer makes sense for now background always caches, crosshair never does since it redraws every frame anyway.
Updated gist: https://gist.github.com/buddy0452004/1fa7c24f60a6a80198a7bcdf7f55a934
VanTrang159 commented on Mar 28, 2026
Hi,
I’ve been exploring this idea of overlay-based interaction and found this issue very relevant to what I’ve been working on. I’ve implemented a small prototype that introduces a simple overlay structure with:
The goal is to separate fast-changing interactive elements from the static figure and update them independently. From my experiments, this approach seems to improve responsiveness and provides a cleaner structure for managing multiple interactive elements.
Does this kind of overlay abstraction align with the intended direction for this issue? Would it be preferable to explore this at the backend level instead?
buddy0452004 commented on Mar 31, 2026
@timhoffm I noticed #31368 touches cursor blitting which overlaps with the layer system approach. Does that fix change anything about the planned architecture for the overlay layer API?
timhoffm commented on Mar 31, 2026
No.
buddy0452004 commented on Apr 5, 2026
While working on other contributions I have been thinking more about the coordinate alignment problem. One approach that might work is anchoring all layers to the figures pixel coordinate system at render time so that zoom/pan/resize events only invalidate layers whose underlying data actually changed rather than invalidating everything because the view transformed this way a static background layer survives a pan/zoom without rerender. Happy to write this up more formally or prototype it if it would help the design discussion.
timhoffm commented on Apr 6, 2026
Not sure I understand. I'd anticipate that most things will need re-rendering on zoom/pan/resize. What are examples that you think would stay unchanged under these operations?
buddy0452004 commented on Apr 7, 2026
good point, concrete examples would be things like a background fill or a static annotation using transAxes or transFigure. those dont change a pixel on zoom or pan because their transform chain doesnt contain transData at all. so if a layer is anchored in figure or axes space it could survive a zoom or pan without any recomputation since its completely view independent, only layers whose transform actually depends on transData would need full invalidation.
that made me dig a bit deeper into the existing codebase and theres actually something interesting already sitting in transforms.py. the transform tree already has contains_branch which can tell you whether an artists transform depends on the view transform or not. so in theory the layer system wouldnt even need layers to manually declare themselves as view independent, you could just inspect the transform chain automatically and know. seemed like a natural connection between the existing transform infrastructure and the layer invalidation problem but happy to be wrong on this.
timhoffm commented on Apr 7, 2026
Those should be quite rare and not expensive to redraw. I'd not complicate the logic by trying to detect such cases.
buddy0452004 commented on Apr 7, 2026
fair enough on the rare cases, not worth the complexity. but the contains_branch part was actually a separate thought, less about figure space elements specifically and more about whether the layer system could use the existing transform tree to make invalidation decisions automatically rather than redrawing everything on every view change. is that kind of smart invalidation something on the radar for the architecture or is the plan to keep it simple and just redraw all layers on zoom/pan?
timhoffm commented on Apr 7, 2026
For a start, I'd keep it simple. We can always refine the logic later.