Collection of python code to re-use across python-based scrapers

• This library is meant to be installed via PyPI (zimscraperlib).
• Make sure to reference it using a version code as the API is subject to frequent changes.
• API should remain the same only within the same minor version.

Example usage:

zimscraperlib>=1.1,<1.2

See documentation at Read the Docs for details.

Most dependencies are installed automatically by pip (from PyPI by default). The following system packages may be required depending on which features you use:

• libmagic — required for file type detection (used in most scrapers)
• wget — required only for zimscraperlib.download functions
• FFmpeg — required only for video processing functions
• gifsicle (>=1.92) — required only for GIF optimization
• libcairo — required only for SVG-to-PNG conversion
• libzim — auto-installed via PyPI, not available on Windows
• Pillow — auto-installed via PyPI; pre-built wheels are used by default and no system image libraries are needed. Only if you need to build Pillow from source should you install additional system libraries — see Pillow's build documentation for details.

  Note: To run the full test suite, all system dependencies listed above must be installed.

brew install libmagic wget ffmpeg gifsicle cairo

sudo apt install libmagic1 wget ffmpeg gifsicle libcairo2

apk add ffmpeg gifsicle libmagic wget cairo

This project adheres to openZIM's Contribution Guidelines.

This project has implemented openZIM's Python bootstrap, conventions and policies v2.0.0.

All instructions below must be run from the root of your local clone of this repository.

If you do not already have it on your system, install hatch:

pip install hatch

Start a hatch shell — this will install all dependencies including dev in an isolated virtual environment:

hatch shell

Set up the pre-commit Git hook (runs linters automatically before each commit):

pre-commit install

Run tests with coverage:

invoke coverage

Non-exhaustive list of scrapers using it (check status when updating API):

• openzim/freecodecamp
• openzim/gutenberg
• openzim/ifixit
• openzim/kolibri
• openzim/nautilus
• openzim/nautilus
• openzim/openedx
• openzim/sotoki
• openzim/ted
• openzim/warc2zim
• openzim/wikihow
• openzim/youtube