I think we can zoom in on colorbar axes now, though, and we can certainly set the x/ylimits. Won't these now dangle in space if we do that?

I was thinking that, but when I tried it with the original example, I was unable to pan or zoom the colorbar. Did it break, or is the test not sufficiently set up to allow it?

I don't see these dangling in space if I test the original example with this...

import matplotlib as mpl
import numpy as np
import matplotlib.pyplot as plt  
from matplotlib.colors import from_levels_and_colors

my_cmap = mpl.cm.viridis
bounds = np.arange(10)
nb_colors = len(bounds) + 1
colors = my_cmap(np.linspace(100, 255, nb_colors).astype(int))
my_cmap, my_norm = from_levels_and_colors(bounds, colors, extend='both')

plt.figure(figsize=(5, 1))
ax = plt.subplot(111)
cbar = mpl.colorbar.ColorbarBase(ax, cmap=my_cmap, norm=my_norm, orientation='horizontal', drawedges=True)
cbar.ax.set_xlim(2, 10)
cbar.ax.set_navigate(True)
plt.subplots_adjust(left=0.05, bottom=0.4, right=0.95, top=0.9)
plt.show()

Because it is a BoundaryNorm we disable interaction:

Going back to what pan/zoom on the Colorbar means, it moves the norm limits, and then moves the axis to match, so the colours and the dividers never move.

Calling set_xlim can break things though:

This now tests with set_xlim as well as the other vertical orientation. It should produce a divider at the right places now even when the limits have been changed.