Patch .bashrc to set up pip to enable pip install - #243
d1jang merged 2 commits into
Conversation
Sorry, something went wrong.
Create a directory for pip to install modules via users' "pip install" command. Set up PIP_CONFIG_FILE & PYTHONPATH to use those modeuls in a Kernel session. Currently this change is no-op until we populate KAGGLE_WORKING_DIR from the worker.
cchamberlain
left a comment
There was a problem hiding this comment.
LGTM - Left comments but not a python guru (#goals) so I trust your implementation.
Sorry, something went wrong.
| # Set up pip to enable pip install. | ||
| ADD patches/kaggle_bashrc /root/.bashrc | ||
| # Patch the system-wide bashrc file for non-root users. | ||
| RUN cat /root/.bashrc >> /etc/bash.bashrc |
There was a problem hiding this comment.
Do root shells not load /etc/bash.bashrc? If they do load it, why not just patch the system-wide bashrc?
Sorry, something went wrong.
There was a problem hiding this comment.
Done.
Sorry, something went wrong.
| # a user to use his/her installed one. | ||
| # TODO(dsjang): Currently "lib/python3.6/site-packages" is hard-coded | ||
| # throughout Dockerfile. Parameterize it to avoid a version mismatch. | ||
| export PYTHONPATH=${PIP_INSTALL_PREFIX_DIR}/lib/python3.6/site-packages:${PYTHONPATH} |
There was a problem hiding this comment.
+1 on the parameterization in future. Could probably read in python version and interpolate I'm assuming though unsure if we'll support multiple versions or chicken / egg scenario with python command being dependent on PYTHONPATH hah.
Sorry, something went wrong.
There was a problem hiding this comment.
Another thing to possibly investigate in the future (if you haven't already) is leaving the top level implementation alone for reserved read-only control and instead setting PYTHONUSERBASE, then documenting to users that they should use:
!pip install --user ...
Unsure on any gotchas that might be relevant there, just feels nice since they're effectively installing custom user specific packages on top of our base. Also using virtualenv to put users into their own sandbox within the working directory is another option that might be worth investigating at some point.
Sorry, something went wrong.
There was a problem hiding this comment.
Acked. Thanks for the very useful feedback.
Your suggested method was actually what I wanted to use, but unfortunately we're using PYTHONUSERBASE for monkeypatching something while python is first loaded. I stashed your comment in case we want to clean things up better in the future.
Thanks!
Sorry, something went wrong.
| # a user to use his/her installed one. | ||
| # TODO(dsjang): Currently "lib/python3.6/site-packages" is hard-coded | ||
| # throughout Dockerfile. Parameterize it to avoid a version mismatch. | ||
| export PYTHONPATH=${PIP_INSTALL_PREFIX_DIR}/lib/python3.6/site-packages:${PYTHONPATH} |
There was a problem hiding this comment.
Did you confirm that this doesn't have an adverse effect on the existing installed libs?
Sorry, something went wrong.
There was a problem hiding this comment.
Except for excessive disk use by installing extra packages, it's hard to predict any problems here.
Without changing how our docker containers' filesystem is structured, this seems to be the only way to make pip install work AFAIK.
Sorry, something went wrong.
There was a problem hiding this comment.
Was more worried about it interfering in some way without doing a pip install. We should be covered there with run_green.
Sorry, something went wrong.
emzeq
left a comment
There was a problem hiding this comment.
LGTM
Sorry, something went wrong.
Create a directory for pip to install modules via users' "pip install" command.
Set up PIP_CONFIG_FILE & PYTHONPATH to use those modeuls in a Kernel session.
Currently this change is no-op until we populate KAGGLE_WORKING_DIR from the worker.