ENH Add micropip.list() by ryanking13 · Pull Request #2012 · pyodide/pyodide · GitHub

Summary

Closes: #1967

Adds a new API micropip.list() which returns a list of installed packages.

Detail

PackageMetadata: a dataclass object that contains (name, version, source) of a package
PackageDict: a dict-like object that maps package name to PackageMetadata
- has custom __repr__() to print list of packages in tabularized format
micropip.list() returns PackageList instance which contains list of installed packages
open to better names and better container design 😀

>>> import micropip
>>> await micropip.install("black")
>>> await micropip.install("https://files.pythonhosted.org/packages/89/06/2c2d3034b4d6bf22f2a4ae546d16925898658a33b4400cfb7e2c1e2871a3/pytz-2020.5-py2.py3-none-any.whl")
>>> pkgs = micropip.list()
>>> print(pkgs)
| Name              | Version  | Source                                                                                                                                      |
| ----------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| regex             | 2021.7.6 | pyodide                                                                                                                                     |
| click             | 8.0.3    | pypi                                                                                                                                        |
| typing-extensions | 4.0.1    | pypi                                                                                                                                        |
| platformdirs      | 2.4.0    | pypi                                                                                                                                        |
| pathspec          | 0.9.0    | pypi                                                                                                                                        |
| tomli             | 1.2.2    | pypi                                                                                                                                        |
| mypy-extensions   | 0.4.3    | pypi                                                                                                                                        |
| black             | 21.11b1  | pypi                                                                                                                                        |
| pytz              | 2020.5   | https://files.pythonhosted.org/packages/89/06/2c2d3034b4d6bf22f2a4ae546d16925898658a33b4400cfb7e2c1e2871a3/pytz-2020.5-py2.py3-none-any.whl |
>>> "regex" in pkgs
True
>>> "pytz" in pkgs
True
>>> "numpy" in pkgs
False

Checklists

Add a CHANGELOG entry
Add / update tests
WHEEL_BASE path
Update pyodide.loadedPackages docs

This (checking digest) is a hacky way to detect whether the wheel is from PyPI or from a custom URL.
Another way can be adding a key that explicitly indicates the wheel is from a custom URL.

Thanks @ryanking13 , I was hoping you would be interested in doing this :)

PackageMetadata / PackageList could use some tests, or at least a few doctests.

…installation succeed

Applied suggestions:

Make micropip implementation private _micropip.py (This also prevent sphinx from generating duplicated docs)
Rename list() to _list() and use aliasing when importing
Lighter table formatting
Use collections.UserDict, and rename PackageList to PackageDict
- Now PackageDict is a really simple dict-like object with custom __repr__ method.
Update installed_packages after installation succeeds.
Add tests for micropip.list()

Thanks! A few more minor comments below.

A more major point, is that if we want micropip to be the main API for users #1470, micropip.list should also list packages loaded via loadPackage. In particular, for a new page load in console it would currently show that no packages are installed, as far as I understand, while the following were loaded,

>>> micropip.micropip.loadedPackages.to_py()
{'distutils': 'default channel', 'micropip': 'default channel', 'pyparsing': 'default channel', 'packaging': 'default channel'}

So I think we can fix this by merging both sources with something like,

def _list():
    from importlib.metadata import version as get_version
    packages = copy.deepcopy(self.installed_packages)
    for name, pkg_source in micropip.micropip.loadedPackages.to_py().items():
         if name in packages:
             continue    

          version = get_version(name)
          source = 'pyodide'
          if pkg_source != 'default channel':
              # Pyodide package loaded from a custom URL
              source = pkg_source 
          packages[name] = PackageMetadata(name=name, version=version, source=source)
    return packages

Though we might want to add a check in test_import here that getting the version with importlib doesn't fail.

WDYT?

Not sure about this. We don't necessarily want to create a new temp folder each time micropip is imported in a new process (outside of Pyodide), particularly given that this is only going to be used in tests. Or else we might need to clean it up afterward.
Maybe let's revert for now and open an issue about it?

For now, testing micropip (== micropip is imported in a new process (outside of Pyodide)) generates wheels folder in the working directory.

How about renaming it to .wheels and adding it to .gitignore?

Also, let's add a mention to https://pyodide.org/en/stable/usage/api/js-api.html#pyodide.loadedPackages docs that this only concerns packages installed with loadPackage and that for a more general solution it is recommended to use

{func}`micropip.list`

A more major point, is that if we want micropip to be the main API for users #1470, micropip.list should also list packages loaded via loadPackage.

Totally agreed, I'll apply that, thanks!

Co-authored-by: Roman Yurchak <rth.yurchak@gmail.com>

Also, let's add a mention to https://pyodide.org/en/stable/usage/api/js-api.html#pyodide.loadedPackages docs that this only concerns packages installed with loadPackage and that for a more general solution it is recommended to use
{func}`micropip.list`

Actually, pyodide.loadedPackages is updated when micropip.install is called. So there are some duplications...

pyodide/packages/micropip/src/micropip/_micropip.py

Lines 137 to 142 in 337847a

    
           async def _install_wheel(name, fileinfo): 
        
               url = fileinfo["url"] 
        
               wheel = io.BytesIO(fileinfo["wheel_bytes"]) 
        
               _validate_wheel(wheel, fileinfo) 
        
               _extract_wheel(wheel) 
        
               setattr(loadedPackages, name, url)

Thanks LGTM.

Thanks! I'll look into the pyodide.loadedPackages later. Maybe we could make it private after #1470.

Implement micropip.list()

c09b5a7

ryanking13 commented Dec 1, 2021

View reviewed changes

rth reviewed Dec 1, 2021

View reviewed changes

ryanking13 added 12 commits December 2, 2021 16:14

Make micropip implementation private

d309586

Rename list method to prevent overriding builtin type

666fd64

Make tabularize method private, and do not capitalize keys

0669edb

Fix table formatting

7e14a58

Update changelog

b582d97

Rename PackageList to PackageDict and make it inherit UserDict

3a6bed2

Remove __all__ from _micropip.py

3d46df1

Add callback function to ensure installed_packages updates only when …

b83192a

…installation succeed

Add some docstrings

0103d08

Add tests for micropip.list()

113fd4d

Fix doctest

78a054b

Prevent black from complaining doctest

8b70488

rth reviewed Dec 5, 2021

View reviewed changes

Apply suggestions from code review [skip ci]

337847a

Co-authored-by: Roman Yurchak <rth.yurchak@gmail.com>

ryanking13 added 4 commits December 6, 2021 14:38

List packages installed through pyodide.loadPackage

4176104

Merge remote-tracking branch 'upstream/main' into micropip/list

dff8fa8

Mock loadedPackages.to_py() on out of browser environment

f180606

Fix mocking loadedPackages.to_py()

1533bca

rth approved these changes Dec 7, 2021

View reviewed changes

rth merged commit 2764136 into pyodide:main Dec 7, 2021

ryanking13 deleted the micropip/list branch June 13, 2022 05:32

                   WHEEL_BASE = Path(getsitepackages()[0])
               else:
-                  WHEEL_BASE = Path(".") / "wheels"
+                  WHEEL_BASE = Path(tempfile.mkdtemp())

+                          # detect whether the wheel metadata is from PyPI or from custom location
+                          # wheel metadata from PyPI has SHA256 checksum digest.
+                          wheel_source = "pypi" if wheel["digests"] is not None else wheel["url"]

ENH Add micropip.list() by ryanking13 · Pull Request #2012 · pyodide/pyodide · GitHub

Uh oh!

Uh oh!

ENH Add micropip.list() - #2012

ENH Add micropip.list()#2012rth merged 18 commits intopyodide:mainpyodide/pyodide:mainfrom ryanking13:micropip/listryanking13/pyodide:micropip/listCopy head branch name to clipboard

Conversation

ryanking13 commented Dec 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Detail

Checklists

Uh oh!

ryanking13 Dec 1, 2021

Choose a reason for hiding this comment

Uh oh!

rth left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryanking13 commented Dec 3, 2021

Uh oh!

rth left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rth Dec 3, 2021

Choose a reason for hiding this comment

Uh oh!

ryanking13 Dec 6, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rth commented Dec 5, 2021

Uh oh!

ryanking13 commented Dec 6, 2021

Uh oh!

ryanking13 commented Dec 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rth left a comment

Choose a reason for hiding this comment

Uh oh!

ryanking13 commented Dec 7, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ENH Add micropip.list()#2012
rth merged 18 commits into
pyodide:mainfrom
ryanking13:micropip/list

ryanking13 commented Dec 1, 2021 •
edited
Loading

rth left a comment •
edited
Loading

rth left a comment •
edited
Loading

ryanking13 commented Dec 6, 2021 •
edited
Loading