PR Summary

Fixes #18925. This doesn't reset the original Axes' locator/formatter which requires a much larger change, but at least calling remove() doesn't error with this PR.

PR Checklist

• Has pytest style unit tests (and pytest passes).
• Is Flake 8 compliant (run flake8 on changed files to check).
• [n/a] New features are documented, with examples if plot related.
• [n/a] Documentation is sphinx and numpydoc compliant (the docs should build without error).
• [n/a] Conforms to Matplotlib style conventions (install flake8-docstrings and run flake8 --docstring-convention=all).
• [n/a] New features have an entry in doc/users/next_whats_new/ (follow instructions in README.rst there).
• [n/a] API changes documented in doc/api/next_api_changes/ (follow instructions in README.rst there).