# What testing coverage approaches are needed?
1. Dependencies: version control (check commit ID dates)
see: [requirements.txt](https://github.com/speechbrain/speechbrain/blob/develop/requirements.txt)
run: `find *txt . | grep extra`
2. Docstring tests: commented function signatures _(of functions intended for outer calls)_
3. [Unittests](https://github.com/speechbrain/speechbrain/tree/develop/tests/unittests) per function-critical code block
4. [Integration tests](https://github.com/speechbrain/speechbrain/tree/develop/tests/integration) for vanilla experiments to cover use-cases on a generic task basis
5. Regression testing: standing interfaces & their refactoring
6. Linters for automated style checks & corrections of python & yaml code
## Where to get things done?
1. Raise your questions & engage in [Discussions](https://github.com/speechbrain/speechbrain/discussions)
2. Report a bug or request a feature, open [Issues](https://github.com/speechbrain/speechbrain/issues/new/choose)
3. Contribute [Pull requests](https://github.com/speechbrain/speechbrain/pulls)
4. Release pretrained models through SpeechBrain
e.g. registering linking HuggingFace account to SpeechBrain for hosting your model card
## GitHub workflow: strategy by configuration
API configurations are located at [.github/workflows](https://github.com/speechbrain/speechbrain/tree/develop/.github/workflows)
_(all creating a one-time ubuntu-latest environment)_
---
Info: although our PyTorch requirements are
```
torch>=1.9.0
torchaudio>=0.9.0
```
our tests cover one PyTorch version only, _the latest_.
### [pre-commit.yml](https://github.com/speechbrain/speechbrain/blob/develop/.github/workflows/pre-commit.yml)
> SpeechBrain pre-commit / pre-commit (pull_request)
* python-version: '3.12'
* run pre-commit action, configured in [.pre-commit-config.yaml](https://github.com/speechbrain/speechbrain/blob/develop/.pre-commit-config.yaml)
* hook: https://github.com/pre-commit/pre-commit-hooks
trailing-whitespace
end-of-file-fixer
requirements-txt-fixer
mixed-line-ending
check-added-large-files
* hook: https://github.com/psf/black
black
click
* hook: https://github.com/astral-sh/ruff-pre-commit
ruff; see: [pyproject.toml](https://github.com/speechbrain/speechbrain/blob/develop/pyproject.toml) for configuration
* hook: https://github.com/adrienverge/yamllint
yamllint; see: [.yamllint.yaml](https://github.com/speechbrain/speechbrain/blob/develop/.yamllint.yaml)
### [pythonapp.yml](https://github.com/speechbrain/speechbrain/blob/develop/.github/workflows/pythonapp.yml)
> SpeechBrain toolkit CI / Tests (3.10) (pull_request)
> SpeechBrain toolkit CI / Tests (3.13) (pull_request)
* python-version: ["3.10", 3.13]
* create fresh environment
```shell
sudo apt-get install -y libsndfile1
pip install -r requirements.txt
pip install --editable .
pip install ctc-segmentation
```
* run PyTest checks
see: [pytest.ini](https://github.com/speechbrain/speechbrain/blob/develop/pytest.ini) - files: `test_*.py`; `check_*.py`; `example_*.py` & norecursedirs
see: [conftest.py](https://github.com/speechbrain/speechbrain/blob/develop/conftest.py) - prepare test item collection & direct discovery
```
# excerpts
parser.addoption("--device", action="store", default="cpu")
...
```
* a. hook: Consistency tests with pytest
`pytest tests/consistency`
* b. hook: Unittests with pytest
`pytest tests/unittests`
* c. hook: Doctests with pytest
`pytest --doctest-modules speechbrain`
* d. hook: Integration tests with pytest
`pytest tests/integration`
### [verify-docs-gen.yml](https://github.com/speechbrain/speechbrain/blob/develop/.github/workflows/verify-docs-gen.yml) [I.2.a]
> Verify docs generation / docs (pull_request)
* python-version: '3.12'
* create fresh environment
```shell
pip install -r requirements.txt
pip install --editable .
pip install -r docs/docs-requirements.txt
```
* generates docs
```shell
cd docs
make html
```
* compare: [.readthedocs.yaml](https://github.com/speechbrain/speechbrain/blob/develop/.readthedocs.yaml) - python version: 3.8
### [newtag.yml](https://github.com/speechbrain/speechbrain/blob/develop/.github/workflows/newtag.yml)
> Draft release when pushing new tag
* tagging of `develop` branch commit ID
* before
* follow through [tests/PRE-RELEASE-TESTS.md](https://github.com/speechbrain/speechbrain/blob/develop/tests/PRE-RELEASE-TESTS.md)
* set-up fresh environment
* run `pytest`
* a. hook: [tests/.run-load-yaml-tests.sh](https://github.com/speechbrain/speechbrain/blob/develop/tests/.run-load-yaml-tests.sh)
* b. hook: [tests/.run-recipe-tests.sh](https://github.com/speechbrain/speechbrain/blob/develop/tests/.run-recipe-tests.sh)
* c. hook: [tests/.run-HF-checks.sh](https://github.com/speechbrain/speechbrain/blob/develop/tests/.run-HF-checks.sh)
* d. hook: [ests/.run-url-checks.sh](https://github.com/speechbrain/speechbrain/blob/develop/tests/.run-url-checks.sh)
* update of [speechbrain/version.txt](https://github.com/speechbrain/speechbrain/blob/develop/speechbrain/version.txt) to the next
* action: draft push to `main` branch
implies pre-push hook, see: [.pre-push-config.yaml](https://github.com/speechbrain/speechbrain/blob/develop/.pre-push-config.yaml) with hooks to:
* e. [tests/.run-linters.sh](https://github.com/speechbrain/speechbrain/blob/develop/tests/.run-linters.sh)
* f. [tests/.run-unittests.sh](https://github.com/speechbrain/speechbrain/blob/develop/tests/.run-unittests.sh)
* g. [tests/.run-doctests.sh](https://github.com/speechbrain/speechbrain/blob/develop/tests/.run-doctests.sh)
### [release.yml](https://github.com/speechbrain/speechbrain/blob/develop/.github/workflows/release.yml)
> Publish to PyPI
* python-version: 3.12
* action: checkout to `main` branch
* creates: `pypa/build` for binary wheel and source tarball
* action: Publish to PyPI via `pypa/gh-action-pypi-publish@master`
implies use of
* [LICENSE](https://github.com/speechbrain/speechbrain/blob/develop/LICENSE)
* [README.md](https://github.com/speechbrain/speechbrain/blob/develop/README.md)
* [pyproject.toml](https://github.com/speechbrain/speechbrain/blob/develop/pyproject.toml) - target-version = ['py38']
* python_requires=">=3.8.1",
* uses: [speechbrain/version.txt](https://github.com/speechbrain/speechbrain/blob/develop/speechbrain/version.txt)
* requires:
```
"hyperpyyaml",
"joblib",
"numpy",
"packaging",
"scipy",
"sentencepiece",
"torch>=1.9",
"torchaudio",
"tqdm",
"huggingface_hub",
```
* points to https://speechbrain.github.io/
The versions of tools used/hooked in these checks are controlled via [lint-requirements.txt](https://github.com/speechbrain/speechbrain/blob/develop/lint-requirements.txt), a nested dependency in [requirements.txt](https://github.com/speechbrain/speechbrain/blob/develop/requirements.txt).
With major version releases of SpeechBrain, the versions of each hook should be updatedalongside requirement consistency in source, testing & builds incl. running spell-checking.
## PyTest for reporting code coverage rates
How to know test coverage changes of Open PRs to be merged?
_(snippet for cpu-only)_
```
# Example: install more dependencies to avoid ignoring modules
sudo apt install -y libsndfile1
pip install ctc_segmentation
# install coverage
pip install pytest-cov
# run the test (w/ duration reporting)
pytest --durations=0 --cov=speechbrain --cov-context=test --doctest-modules speechbrain tests
```
Example: _After collecting 459 testing items, 4481/16782 statements are reported "missing" (73% coverage)._
YETpython code of the core modules is not all to be covered; thus far, only, consistency is ensured..
---
Further reading:
pytest & coverage - https://breadcrumbscollector.tech/how-to-use-code-coverage-in-python-with-pytest/ (pointer by @Adel-Moumen)
---
```
pytest --durations=0 --cov=speechbrain --cov-context=test --doctest-modules speechbrain tests
---------- coverage: platform linux, python 3.9.12-final-0 -----------
Name Stmts Miss Cover
-----------------------------------------------------------------------------
speechbrain/alignment/aligner.py 380 61 84%
speechbrain/alignment/ctc_segmentation.py 189 10 95%
speechbrain/core.py 424 155 63%