Store iteration value in callbacks.MinSaver/callbacks.MaxSaver #722
Description
Please add iteration value in the filename of the model saved by callbacks.MinSaver
Today, using MinSaver('validation_loss') we get a files named:
min-validation_loss.*
however, it could be useful to store iteration value (and value of monitored variable) in the filename.
@ppwwyyxx what do you think about ?
thanks
Activity
ppwwyyxx commented on Apr 3, 2018
Adding the iteration number sounds OK. But this makes it harder to track (and overwrite) the last minsaver checkpoint, if you resume a training.
ppwwyyxx commented on Apr 18, 2018
Looks like MinSaver simply can not work with resume now.
The fundamental problem is that callbacks like MinSaver, JSONWriter are stateful. Therefore once the training was interrupted the state is lost. The way JSONWriter tries to recover its state when resuming is already kind of hacky. I will think if there is a better way to support stateful callbacks.