Reorganize dependency installation for better squashing - #1523
develra wants to merge 1 commit into
Conversation
Sorry, something went wrong.
I'll leave it up to y'all to decide if the changes/risks here are worth the reduction in image size. Thanks! Reduced image size ┌─────────────────┬──────────┬─────────┬───────────────┐ │ Metric │ Original │ New │ Reduction │ ├─────────────────┼──────────┼─────────┼───────────────┤ │ Image Size │ 60.1 GB │ 48.2 GB │ 11.9 GB (20%) │ ├─────────────────┼──────────┼─────────┼───────────────┤ │ Filesystem Size │ 49 GB │ 44 GB │ 5 GB (10%) │ └─────────────────┴──────────┴─────────┴───────────────┘ Note: Image size includes all layers; filesystem size is the actual disk usage inside the container. - Added --no-cache to uv pip install (Safe) Cache is only useful for repeated installs in the same environment. In Docker builds, each layer is fresh, so cache provides no benefit. - Removed Intel MKL numpy (Less sure) Removed the Intel MKL numpy install from Intel's Anaconda channel. Intel's channel only has numpy 1.26.4 (numpy 1.x), but the base image has numpy 2.0.2. Installing Intel's numpy would downgrade and break packages compiled against numpy 2.x ABI. The base image's numpy 2.0.2 uses OpenBLAS optimizations and is compatible with all installed packages. - Removed preprocessing package (Less sure) Package is unmaintained (last release 2017) and requires nltk==3.2.4 which is incompatible with Python 3.11 (inspect.formatargspec was removed). Package hasn't been updated in 7+ years and cannot function on Python 3.11. - Updated scikit-learn to 1.5.2 (Less sure) Changed from scikit-learn==1.2.2 to scikit-learn==1.5.2. scikit-learn 1.2.2 binary wheels are incompatible with numpy 2.x ABI, causing "numpy.dtype size changed" errors. scikit-learn 1.5.x maintains API compatibility with 1.2.x. The original pin was for eli5/learntools compatibility, which should work with 1.5.x. - Added uv cache cleanup to clean-layer.sh (safe) Added /root/.cache/uv/* to the cleanup script. The script only cleaned pip cache, not uv cache. Cache cleanup scripts are run after package installs; cache is not needed at runtime.
|
Adding Vincent since they are on docker ops |
Sorry, something went wrong.
rosbo
left a comment
There was a problem hiding this comment.
Given the unsafe nature of the layer squashing operations, let's focus on the safe changes (deleting files from the uv cache) and making sure clean-layer is called at each layer writing to disk.
Sorry, something went wrong.
| && uv pip install --no-cache --system --force-reinstall --no-deps torchtune gensim "scipy<=1.15.3" "huggingface-hub==0.36.0" "google-cloud-translate==3.12.1" "numpy==2.0.2" "pandas==2.2.2" \ | ||
| && uv pip install --no-cache --system --force-reinstall "protobuf==5.29.5" \ | ||
| && uv pip install --no-cache --system --force-reinstall "nltk>=3.9.1" \ | ||
| && rm -rf /root/.cache/uv /root/.cache/pip |
There was a problem hiding this comment.
Can you move the ADD clean-layer.sh /tmp/clean-layer.sh line above and call clean-layer.sh here instead.
Sorry, something went wrong.
| # b/468383498: numpy will auto-upgrade to 2.4.x, which causes issues with numerous packages | ||
| # b/468367647: Unpin protobuf, version greater than v5.29.5 causes issues with numerous packages | ||
| # b/408298750: We reinstall nltk because older versions have: `AttributeError: module 'inspect' has no attribute 'formatargspec'` | ||
| RUN uv pip install --no-cache \ |
There was a problem hiding this comment.
If we are cleaning up the layer at the end, is passing --no-cache redundant?
Sorry, something went wrong.
There was a problem hiding this comment.
Probably - it takes a little while to run each experiment as I need to build the whole container, but I'll try to drop it and just be more specific about the clean-layer.sh
Sorry, something went wrong.
| RUN echo "$GIT_COMMIT" > /etc/git_commit && echo "$BUILD_DATE" > /etc/build_date | ||
|
|
||
| # Final cleanup | ||
| RUN rm -rf /root/.cache/uv /root/.cache/pip /tmp/clean-layer.sh |
There was a problem hiding this comment.
Removing the .cache directory at this point is pointless given it has been saved in previous layers anyway.
Sorry, something went wrong.
| # Stage 2: Squash the base + reinstalls to eliminate layer bloat | ||
| # ============================================================================= | ||
| FROM scratch AS clean-base | ||
| COPY --from=base-reinstalls / / |
There was a problem hiding this comment.
This doesn't copy all the changes to environment variable and whatnot. Causing issues like:
�[0m - InvalidDefaultArgInFrom: Default value for ARG ${BASE_IMAGE}:${BASE_IMAGE_TAG} results in empty or invalid base image name (line 9)
- UndefinedVar: Usage of undefined variable '$PYTHONPATH' (line 153)
ERROR: failed to build: failed to solve: process "/bin/sh -c uv pip install --no-cache --system \"pycuda\"" did not complete successfully: exit code: 1
Sorry, something went wrong.
|
Closed in favor of #1532. We won't do the layer squashing for now because it is not safe: #1523 (comment) |
Sorry, something went wrong.
) Docker layers. We were not clearing the `uv` cache adding extra bloat to each our our docker layer. Added more background on this here: https://b.corp.google.com/issues/277208559#comment12 Reasoning why I preferred to use `--no-cache` rather than the `clean-layer.sh` script: https://b.corp.google.com/issues/277208559#comment13 Inspired by #1523. http://b/277208559
I'll leave it up to y'all to decide if the changes/risks here are worth the reduction in image size. Thanks!
Reduced image size
Note: Image size includes all layers; filesystem size is the actual disk usage inside the container.
Added --no-cache to uv pip install (Safe)
Cache is only useful for repeated installs in the same environment. In Docker builds, each layer is fresh, so cache provides no benefit.
Removed Intel MKL numpy (Less sure)
Removed the Intel MKL numpy install from Intel's Anaconda channel. Intel's channel only has numpy 1.26.4 (numpy 1.x), but the base image has numpy 2.0.2. Installing Intel's numpy would downgrade and break packages compiled against numpy 2.x ABI.
The base image's numpy 2.0.2 uses OpenBLAS optimizations and is compatible with all installed packages.
Removed preprocessing package (Less sure)
Package is unmaintained (last release 2017) and requires nltk==3.2.4 which is incompatible with Python 3.11 (inspect.formatargspec was removed). Package hasn't been updated in 7+ years and cannot function on Python 3.11.
Updated scikit-learn to 1.5.2 (Less sure)
Changed from scikit-learn==1.2.2 to scikit-learn==1.5.2. scikit-learn 1.2.2 binary wheels are incompatible with numpy 2.x ABI, causing "numpy.dtype size changed" errors. scikit-learn 1.5.x maintains API compatibility with 1.2.x. The original pin was for eli5/learntools compatibility, which should work with 1.5.x.
Added uv cache cleanup to clean-layer.sh (safe)
Added /root/.cache/uv/* to the cleanup script. The script only cleaned pip cache, not uv cache. Cache cleanup scripts are run after package installs; cache is not needed at runtime.