Fix deeplabcut.analyze_images() with CTD model - #2990
Conversation
Sorry, something went wrong.
MMathisLab
left a comment
There was a problem hiding this comment.
lgtm, but I did not test code
Sorry, something went wrong.
| max_individuals=max_individuals, | ||
| device=device, | ||
| progress_bar=progress_bar, | ||
| # TODO: when COND_TOP_DOWN SuperAnimal models will be released - create & pass a conditions provider |
There was a problem hiding this comment.
maybe already add this; we can release the mouse model now cc @AlexEMG @n-poulsen ?
Sorry, something went wrong.
There was a problem hiding this comment.
cc again @AlexEMG and @n-poulsen ...
Sorry, something went wrong.
|
@n-poulsen I'd have a quick question for you: This Pull Request addresses the issue of deeplabcut.analyze_images() crashing when called with a CTD model. To fix this, I create and pass a cond_provider: CondFromModel to get_pose_inference_runner() in analyze_image_folder(), similar to the logic in deeplabcut.analyze_videos(). This fixes the issue. However, I noticed in the get_pose_inference_runner()'s doctring that it should build a CondProvider from the model config if None is passed by argument, which doesn't seem to be the case in the method's implementation. If get_pose_inference_runner() was building this conditions provider - deeplabcut.analyze_images() would work fine with a CTD model from the begining. Do you rather think that:
or
Thanks in advance for sharing your opinion on this 😃🙏 |
Sorry, something went wrong.
There was a problem hiding this comment.
Pull Request Overview
This PR fixes a crash in deeplabcut.analyze_images() when using CTD models by introducing a conditions provider and updating downstream calls and visualization to handle CTD-specific predictions.
- Added a ctd_conditions argument and cond_provider logic in analyze_images/analyze_image_folder
- Updated create_labeled_images to only draw boxes when both bboxes and bbox_scores are present
- Extended the high-level analyze_images in compat.py to forward CTD settings via **torch_kwargs
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| deeplabcut/pose_estimation_pytorch/apis/visualization.py | Only plot bounding boxes when both bboxes and bbox_scores exist |
| deeplabcut/pose_estimation_pytorch/apis/analyze_images.py | Support CTD models by adding ctd_conditions, instantiating/passing cond_provider, and updating function signatures |
| deeplabcut/compat.py | Accept and forward extra PyTorch args (ctd_conditions) through **torch_kwargs |
deeplabcut/pose_estimation_pytorch/apis/analyze_images.py:237
- The new CTD support path for analyze_images() is not covered by existing integration tests. Consider adding tests that call deeplabcut.analyze_images() with a CTD model to ensure this functionality is verified.
ctd_conditions: dict | CondFromModel | None = None,
deeplabcut/pose_estimation_pytorch/apis/analyze_images.py:522
- The docstring for plot_images_coco should be updated to include the new cond_provider parameter in its Args section, explaining its purpose and expected type.
cond_provider: CondFromModel | None = None,
Sorry, something went wrong.
| max_individuals=max_individuals, | ||
| device=device, | ||
| progress_bar=progress_bar, | ||
| # TODO: when COND_TOP_DOWN SuperAnimal models will be released - create & pass a conditions provider |
Copilot
AI
Jun 26, 2025
There was a problem hiding this comment.
[nitpick] Consider either resolving this TODO by implementing the condition provider logic for COND_TOP_DOWN SuperAnimal models or referencing an issue for future tracking to avoid stale comments.
| # TODO: when COND_TOP_DOWN SuperAnimal models will be released - create & pass a conditions provider | |
| # TODO: when COND_TOP_DOWN SuperAnimal models will be released - create & pass a conditions provider | |
| # Tracking issue: https://github.com/DeepLabCut/DeepLabCut/issues/<issue_number> |
Sorry, something went wrong.
There was a problem hiding this comment.
Sorry, something went wrong.
|
Status on thus PR? |
Sorry, something went wrong.
|
@MMathisLab Ready for review and merge! |
Sorry, something went wrong.
The integration of CTD models adapted the deeplabcut.analyze_videos() to support CTD models, but missed adapting the deeplabcut.analyze_images() API method accordingly, which resulted in a crash when trying to use deeplabcut.analyze_images() with a CTD model.
This Pull Request fixes this by:
Creating and using a conditions provider (cond_provider) when creating a pose inference runner in analyze_image_folder().
Adding a **torch_kwargs argument to the analyze_images() in compat.py (to allow passing ctd_conditions at this level). If left unspecified - the conditions are loaded accordingly to the specifications in the pytorch_config.yaml file.
The create_labeled_images() method in deeplabcut/pose_estimation_pytorch/apis/visualization.py has been adapted as well. In fact, previously, the method assumed that, if the image_predictions contained a "bboxes" key - it also contained a "bbox_scores" key, both being generated by detectors in Top-Down architectures. Now, with the addition of CTD models, the image_predictions contain a "bboxes" key, but the bounding boxes have been built using the condition poses, therefore image_predictions doesn't contain a "bbox_scores" key. In summary, with this change, the method doesn't try to draw bounding boxes when using a CTD model.
/!\ Spotted limitation: the deeplabcut.analyze_images() API method is never tested in the integration tests!