FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Fix CSVLogger: read existing learning_stats.csv file when resuming previous training by deruyter92 · Pull Request #3177 · DeepLabCut/DeepLabCut · GitHub

/ DeepLabCut Public

Fix CSVLogger: read existing learning_stats.csv file when resuming previous training - #3177

Merged
C-Achard merged 9 commits into
DeepLabCut:mainfrom
deruyter92:jaap/fix_csvlogger_resume_training
Feb 13, 2026
ConversationCommits (9)ChecksFiles changed
Merged

Fix CSVLogger: read existing learning_stats.csv file when resuming previous training#3177
C-Achard merged 9 commits into
DeepLabCut:mainfrom
deruyter92:jaap/fix_csvlogger_resume_training

Conversation

deruyter92 commented Jan 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Previously, when resuming training the CSVLogger created a new learning_stats.csv file for the second training run, overwriting previously saved training metrics.

This commit addresses feature request #3176 and changes the behaviour to reading any previously saved log metrics and appending new epochs to the learning_stats.csv file.

  • added _load_existing_data method to CSVLogger
  • added unit test for _load_existing_data

Fix CSVLogger reads existing file for resumed training
a78f75d 
Previously, when resuming training the CSVLogger created a new learning_stats.csv file for the second training run, overwriting previously saved training metrics.

This commit addresses feature request DeepLabCut#3176 and changes the behaviour to reading any previously saved log metrics and appending new epochs to the learning_stats.csv file.
deruyter92 requested a review from Copilot January 13, 2026 09:49
Copilot AI reviewed Jan 13, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

deruyter92 changed the title Fix CSVLogger reads existing file for resumed training Fix CSVLogger: read existing learning_stats.csv file when resuming previous training Jan 13, 2026
deruyter92 requested a review from Copilot January 13, 2026 09:50
Copilot AI reviewed Jan 13, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

MMathisLab closed this Jan 13, 2026

MMathisLab commented Jan 13, 2026

Copy link
Copy Markdown
Member

closed in favor of #3179

Reverting PR DeepLabCut#3179 which breaks csv logging.
ce2e260 
The header and previous rows are appended for every save call. This in unintended behavior.

- changing back to 'write' mode instead of 'append' mode
deruyter92 mentioned this pull request Jan 16, 2026
Merged
deruyter92 reopened this Jan 16, 2026
deruyter92 mentioned this pull request Jan 16, 2026
Merged
deruyter92 marked this pull request as draft January 16, 2026 09:25
deruyter92 marked this pull request as ready for review January 16, 2026 16:03
deruyter92 requested a review from C-Achard January 16, 2026 16:03

C-Achard commented Jan 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Looks good ! Should we consider adding some more validation in case the previous output is somehow broken, interrupted or to try and account for potential formatting changes in the future ? Or do you feel that would be redundant for now/the current solution is enough ?

C-Achard requested a review from Copilot January 16, 2026 17:06
Copilot AI reviewed Jan 16, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/pose_estimation_pytorch/runners/test_logger.py Outdated
Comment thread tests/pose_estimation_pytorch/runners/test_logger.py Outdated
Comment thread deeplabcut/pose_estimation_pytorch/runners/logger.py Outdated
Comment thread deeplabcut/pose_estimation_pytorch/runners/logger.py Outdated
Comment thread deeplabcut/pose_estimation_pytorch/runners/logger.py Outdated
Comment thread deeplabcut/pose_estimation_pytorch/runners/logger.py Outdated
Comment thread deeplabcut/pose_estimation_pytorch/runners/logger.py Outdated
C-Achard added the enhancement New feature or request label Jan 16, 2026

deruyter92 commented Jan 19, 2026

Copy link
Copy Markdown
Collaborator Author

Should we consider adding some more validation in case the previous output is somehow broken, interrupted or to try and account for potential formatting changes in the future ?
Good thought!

  • The new _load_existing_data method is agnostic to the specific columns in the existent CSV. If they are different from the new training run, this doesn't break the CSV file (missing fields are just kept empty)
  • The only required field is step, containing an integer value. There are no further constraints on what the existing data should look like.
  • I have adjusted the behavior now, based on your suggestion: In case any exception occurred during reading (e.g. corrupted CSV, or CSV that misses 'step' column), the reader now starts with an empty CSV instead of keeping the succeeded rows from the existent CSV.

I don't think it would be a good idea to put specific constraints on the values or data types that should be in the existent CSV, except for the 'step' index. The CSVLogger is currently quite flexible and even logs different metrics for different epoch types (e.g. validation epochs). I think the benefits are limited when we drop this flexible/column-agnostic behavior of the CSVLogger.

deruyter92 and others added 2 commits January 19, 2026 12:03
CSVLogger: change _load_existing_data handling of partly valid CSVs
dfa3c3f 
In case an exception occurrs during reading (e.g. corrupted CSV, or CSV that misses 'step' column), the reader now starts with an empty CSV instead of keeping the succeeded rows from the existent CSV.
Merge branch 'main' into jaap/fix_csvlogger_resume_training
bb073b9
deruyter92 requested a review from C-Achard January 19, 2026 14:54

C-Achard commented Jan 20, 2026

Copy link
Copy Markdown
Collaborator

@deruyter92 Sounds good, thanks !

deruyter92 commented Jan 20, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks @C-Achard! Ok, @MMathisLab, please have a look if you agree. In that case these changes can be merged!

deruyter92 added the bug Something isn't working label Jan 21, 2026
deruyter92 requested a review from MMathisLab February 2, 2026 09:56

deruyter92 commented Feb 4, 2026

Copy link
Copy Markdown
Collaborator Author

@MMathisLab let me know what you think! I think it could be merged.

C-Achard merged commit ca9cfd2 into DeepLabCut:main Feb 13, 2026
10 checks passed
deruyter92 mentioned this pull request Feb 28, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
C-Achard C-Achard approved these changes
MMathisLab Awaiting requested review from MMathisLab

Assignees

No one assigned

Labels

bug Something isn't working enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

Back | FazBrowse Home | New Git URL