Update and diversify TensorFlow optional installations. by deruyter92 · Pull Request #3292 · DeepLabCut/DeepLabCut · GitHub

Summary

The current tensorflow installation via the optional extra tf is designed as a one-solution-for-all, instead of providing a tailored fit for multiple users with different machines. Also, there is divergence between the recommendations in the docs, CI and the current pyproject.toml. This PR implements a more diverse set of tensorflow installations, and aligns the CI so that these versions are actually tested. The documentation is also updated accordingly.

Addresses #3271

Optional dependencies

The new configuration is as follows:

tf: tensorflow>=2.12,<2.18 with all related dependencies -> recommended range that fits the documented last-supported versions and was tested on cuda 11.8 and cuda 12.0. Takes into account the installed python version.
tf-cu11 tensorflow==2.14 with all related dependencies -> for cuda 11.x machines.
tf-cu12 tensorflow==2.18 with all related dependences -> for cuda 12.x machines
tf-latest tensorflow>=2.18): 'best effort' support for users that prefer the latest installation (not tested).

CI

The separate hardcoded installation of TensorFlow is removed from CI and replaced with installing the tf extra.
The intelligent testing workflow now includes the option to specify custom extras for each job (turned out not to be necessary for tf, but still useful for the future).
A basic smoke test is implemented for just testing the tensorflow installation with some basic operations: tests/test_tf_install_smoke
The smoke test is hooked up to a job in intelligent-testing.yaml testing a matrix of os<>python<>extras. The job is purposely implemented in parallel to the existing lanes (e.g. full testing lane), as it doesn't need downloading ffmpeg etc.

Documentation

The documentation changes are now moved to a separate PR #3317.

Looks nice so far; thanks a lot for handling this! t

Two questions regarding the scope :

How should we handle documentation?
Can we have minimal CI testing that covers each of these paths ? What do you think would be the best method there ? Maybe a workflow only on main pushes that installs each and runs one of the lighter TF test script ?

yess working at it! good suggestions

For the docs happy to coordinate and add this to my ongoing updates!

Note: adding some smoke tests in CI, which I will probably move to a separate PR later to keep the discussions separated. But first want to test for compatible versioning.

I will also update the docs once finished and coordinate with you @C-Achard & help with the new docs to avoid parallel work.

@C-Achard, it's ready for review now. Let me know what you think about the CI changes: for now I added a simple smoke test directly in the intelligent testing workflow, which means we can avoid (downloading and) installing ffmpeg for every run, and also doesn't affect the concurrency with the full-matrix tests. If you think it should make use of the python-package workflow, like the other test lanes, I can see arguments for that as well.

Can we cherry-pick/adapt changes to the latest docs audit updates ?

Seems tensorflow-macos is starting to fail in CI, perhaps we should make this high priority

Note: this has already been replaced by #3280 and can be safely discarded I think

- revise tf extra - add tf-cu11, tf-cu12, tf-latest This brings in line the CI testing with the actual extras provided, simplifies the installation, and allows for targeted installs for different users.

Xet is Hugging Face’s new storage backend designed to replace the aging Git LFS (Large File Storage). While Git LFS treats every file as a single "blob," Xet breaks files into small "chunks." In restricted CI environments (like CircleCI, GCP, or Kaggle), the Xet "chunked" download often stalls. Because Xet opens many simultaneous connections to different data chunks, CI firewalls or NAT gateways sometimes flag this as suspicious activity or simply drop the connections when they hit a certain threshold.

Cleaned up this PR and fixed failing tests:

cherry-pick CircleCI: disable hugginface xet #3316 to fix failing CircleCI test (connections issues with new HuggingFace Xet)
install dependency-groups in CI to ensure pip uses dev dependency group
move TF-docs update to separate PR Docs [April 2026] Update TensorFlow documentation #3317 to resolve merge conflicts with the new docs format.

deruyter92 force-pushed the jaap/update_tf_extras branch from 91026e2 to 33126b8 Compare April 22, 2026 08:52

C-Achard self-requested a review April 22, 2026 08:57

C-Achard reviewed Apr 22, 2026

View reviewed changes

deruyter92 force-pushed the jaap/update_tf_extras branch 2 times, most recently from e198b38 to 42b2813 Compare April 22, 2026 15:51

deruyter92 force-pushed the jaap/update_tf_extras branch from 285ba05 to 3f71887 Compare April 22, 2026 16:41

C-Achard linked an issue Apr 22, 2026 that may be closed by this pull request

Diversify optional installs for TensorFlow stack (LTS) #3271

Closed

deruyter92 force-pushed the jaap/update_tf_extras branch 5 times, most recently from b7e27a1 to 527cf2f Compare April 24, 2026 14:17

deruyter92 marked this pull request as ready for review April 28, 2026 08:57

deruyter92 requested a review from C-Achard April 28, 2026 08:57

C-Achard assigned deruyter92 Apr 28, 2026

C-Achard added Installation Tensorflow 2 dependencies Pull requests that update a dependency file labels Apr 28, 2026

deruyter92 force-pushed the jaap/update_tf_extras branch from 2a29eb6 to 527cf2f Compare May 4, 2026 20:24

C-Achard reviewed May 5, 2026

View reviewed changes

C-Achard approved these changes May 5, 2026

View reviewed changes

C-Achard reviewed May 5, 2026

View reviewed changes

C-Achard mentioned this pull request May 5, 2026

update list_videos_in_folder #3303

Merged

8 tasks

C-Achard added the high-priority label May 5, 2026

deruyter92 requested a review from AlexEMG May 6, 2026 08:58

deruyter92 requested a review from MMathisLab May 6, 2026 08:58

MMathisLab approved these changes May 6, 2026

View reviewed changes

deruyter92 added 6 commits May 7, 2026 11:04

update pyproject.toml and uv.lock TF-extras

f9a9579

- revise tf extra - add tf-cu11, tf-cu12, tf-latest This brings in line the CI testing with the actual extras provided, simplifies the installation, and allows for targeted installs for different users.

add basic smoke test for tensorflow installation

db857b6

update CI: add "extras" field to matrix_json

9a78331

update CI: add TF install smoke test (runs on full test lane)

77085d7

skip functional tests for tensorflow and windows with python >=3.11

b5662c4

Update tests that rely on TF: add import-or-skip.

9e7ed58

deruyter92 force-pushed the jaap/update_tf_extras branch from e5cea11 to 8c880cd Compare May 7, 2026 09:14

deruyter92 added 2 commits May 7, 2026 11:27

Update CI: ensure pip with dev dependency groups

f7b4e37

deruyter92 force-pushed the jaap/update_tf_extras branch from 8c880cd to a9f3d51 Compare May 7, 2026 09:28

deruyter92 mentioned this pull request May 7, 2026

Docs [April 2026] Update TensorFlow documentation #3317

Merged

deruyter92 mentioned this pull request May 7, 2026

CircleCI: disable hugginface xet #3316

Merged

MMathisLab merged commit 85569cb into main May 11, 2026
31 checks passed

MMathisLab deleted the jaap/update_tf_extras branch May 11, 2026 20:28

deruyter92 mentioned this pull request May 21, 2026

Add 3.0 changelog #3340

Merged

1 task

Update and diversify TensorFlow optional installations. by deruyter92 · Pull Request #3292 · DeepLabCut/DeepLabCut · GitHub

Uh oh!

Uh oh!

Update and diversify TensorFlow optional installations. - #3292

Update and diversify TensorFlow optional installations.#3292
MMathisLab merged 8 commits into
mainfrom
jaap/update_tf_extras

deruyter92 commented Apr 22, 2026 •
edited
Loading

Uh oh!

Uh oh!

C-Achard left a comment •
edited
Loading

Uh oh!

Uh oh!

deruyter92 commented Apr 22, 2026

Uh oh!

C-Achard commented Apr 22, 2026

Uh oh!

deruyter92 commented Apr 22, 2026

Uh oh!

deruyter92 commented Apr 28, 2026

Uh oh!

C-Achard May 5, 2026

Uh oh!

C-Achard commented May 5, 2026

Uh oh!

C-Achard May 5, 2026

Uh oh!

deruyter92 commented May 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Update and diversify TensorFlow optional installations. by deruyter92 · Pull Request #3292 · DeepLabCut/DeepLabCut · GitHub

Uh oh!

Uh oh!

Update and diversify TensorFlow optional installations. - #3292

Update and diversify TensorFlow optional installations.#3292MMathisLab merged 8 commits intomainDeepLabCut/DeepLabCut:mainfrom jaap/update_tf_extrasDeepLabCut/DeepLabCut:jaap/update_tf_extrasCopy head branch name to clipboard

Conversation

deruyter92 commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Optional dependencies

CI

Documentation

Uh oh!

C-Achard left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deruyter92 commented Apr 22, 2026

Uh oh!

C-Achard commented Apr 22, 2026

Uh oh!

deruyter92 commented Apr 22, 2026

Uh oh!

deruyter92 commented Apr 28, 2026

Uh oh!

C-Achard May 5, 2026

Choose a reason for hiding this comment

Uh oh!

C-Achard commented May 5, 2026

Uh oh!

C-Achard May 5, 2026

Choose a reason for hiding this comment

Uh oh!

deruyter92 commented May 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Update and diversify TensorFlow optional installations.#3292
MMathisLab merged 8 commits into
mainfrom
jaap/update_tf_extras

deruyter92 commented Apr 22, 2026 •
edited
Loading

C-Achard left a comment •
edited
Loading