I have been using the min_relative_train_loss_per_iter and min_relative_test_loss_per_iter in my fittings but during the ladder scheme fitting when the early stopping is triggered at the same time the maximum number of iterations have been reached (I set this to 1000), it crashes the run.
kappa = 0.08
nrad_max = [15, 7, 3, 2, 1, 1], l_max = [0, 4, 3, 2, 1, 1] => 374 functions
ladder_type = 'power_order'
ladder_step = [20, 0.1]
batch_size = 1000
early_stopping_patience = 200
max_iter = 1000
--------------------------------------------TEST STATS--------------------------------------------
Iteration: #1000Loss: Total: 2.6584e-05 (100%)
Energy: 1.2040e-05 ( 45%)
Force: 1.3858e-05 ( 52%)
L1: 4.0020e-07 ( 2%)
L2: 2.8561e-07 ( 1%)
Number of params./funcs: 585/100 Avg. time: 0.00 mcs/at
-------------------------------------------------------------------------------------------------
Energy/at, meV/at Energy_low/at, meV/at Force, meV/A Force_low, meV/A
RMSE: 7.70 3.36 31.63 13.22
MAE: 3.90 2.27 10.31 6.08
MAX_AE: 154.11 28.21 1023.20 208.15
-------------------------------------------------------------------------------------------------
2025/02/11 13:15:36 I - Last relative TEST loss change -4.72e-05/iter (averaged over last 50 step(s))
/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/scipy/optimize/_minimize.py:726: OptimizeWarning: Maximum number of iterations has been exceeded.
res = _minimize_bfgs(fun, x0, args, jac, callback, **options)
2025/02/11 13:15:40 I - EARLY STOPPING: Too small or even positive TEST loss change (best=-9.94e-05 / iter, last=+0.00e+00/iter, threshold = -1.00e-04/iter) within last 200 iterations. Stopping
Current function value: 0.000229
Iterations: 1000
Function evaluations: 1028
Gradient evaluations: 1028
Fitting took 1767.17 seconds
Current function value: 0.000065
Iterations: 1000
Function evaluations: 1023
Gradient evaluations: 1023
Fitting took 1998.61 seconds
Current function value: 0.000044
Iterations: 1000
Function evaluations: 1024
Gradient evaluations: 1024
Fitting took 3360.17 seconds
Current function value: 0.000031
Iterations: 1000
Function evaluations: 1010
Gradient evaluations: 1010
Fitting took 2765.05 seconds
Current function value: 0.000024
Iterations: 1000
Function evaluations: 1013
Gradient evaluations: 1013
Traceback (most recent call last):
File "/cmmc/ptmp/hgaafer/mambaforge/bin/pacemaker", line 401, in <module>
main(sys.argv[1:])
File "/cmmc/ptmp/hgaafer/mambaforge/bin/pacemaker", line 248, in main
general_fit.fit()
File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/generalfit.py", line 481, in fit
self.target_bbasisconfig = self.ladder_fitting(self.initial_bbasisconfig, self.target_bbasisconfig)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/generalfit.py", line 509, in ladder_fitting
current_bbasisconfig = self.cycle_fitting(current_bbasisconfig)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/generalfit.py", line 564, in cycle_fitting
current_bbasisconfig = self.fit_backend.fit(
^^^^^^^^^^^^^^^^^^^^^
File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/fitadapter.py", line 129, in fit
raise e
File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/fitadapter.py", line 96, in fit
fit_res = self.run_tensorpot_fit(bbasisconfig, dataframe, loss_spec, fit_config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/fitadapter.py", line 236, in run_tensorpot_fit
self.fitter.fit(dataframe, test_df=test_dataframe, niter=fit_config[FIT_NITER_KW],
File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/tensorpotential/fit.py", line 125, in fit
self.process_test_metric()
File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/tensorpotential/fit.py", line 318, in process_test_metric
self.test_metric_callback(curr_test_metrics_data)
File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/generalfit.py", line 410, in test_metric_callback
self.detect_early_stopping(mode='test')
File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/generalfit.py", line 465, in detect_early_stopping
raise TestLossChangeTooSmallException(msg)
pyace.generalfit.TestLossChangeTooSmallException: EARLY STOPPING: Too small or even positive TEST loss change (best=-9.94e-05 / iter, last=+0.00e+00/iter, threshold = -1.00e-04/iter) within last 200 iterations. Stopping
Exception: Potential file output_potential.yaml doesn't existsLoading B-basis from 'output_potential.yaml'
Traceback (most recent call last):
File "/cmmc/ptmp/hgaafer/mambaforge/bin/pace_yaml2yace", line 28, in <module>
bbasis = ACEBBasisSet(input_yaml_filename)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Potential file output_potential.yaml doesn't exists
I have been using the min_relative_train_loss_per_iter and min_relative_test_loss_per_iter in my fittings but during the ladder scheme fitting when the early stopping is triggered at the same time the maximum number of iterations have been reached (I set this to 1000), it crashes the run.
This may happen when min_relative_test_loss_per_iter is set to 5e-5 or 1e-4
Those are the other parameters I used in my fittings,
This is the error message I get from the log file after the 1000th iteration,
--------------------------------------------TEST STATS-------------------------------------------- Iteration: #1000Loss: Total: 2.6584e-05 (100%) Energy: 1.2040e-05 ( 45%) Force: 1.3858e-05 ( 52%) L1: 4.0020e-07 ( 2%) L2: 2.8561e-07 ( 1%) Number of params./funcs: 585/100 Avg. time: 0.00 mcs/at ------------------------------------------------------------------------------------------------- Energy/at, meV/at Energy_low/at, meV/at Force, meV/A Force_low, meV/A RMSE: 7.70 3.36 31.63 13.22 MAE: 3.90 2.27 10.31 6.08 MAX_AE: 154.11 28.21 1023.20 208.15 ------------------------------------------------------------------------------------------------- 2025/02/11 13:15:36 I - Last relative TEST loss change -4.72e-05/iter (averaged over last 50 step(s)) /cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/scipy/optimize/_minimize.py:726: OptimizeWarning: Maximum number of iterations has been exceeded. res = _minimize_bfgs(fun, x0, args, jac, callback, **options) 2025/02/11 13:15:40 I - EARLY STOPPING: Too small or even positive TEST loss change (best=-9.94e-05 / iter, last=+0.00e+00/iter, threshold = -1.00e-04/iter) within last 200 iterations. Stopping Current function value: 0.000229 Iterations: 1000 Function evaluations: 1028 Gradient evaluations: 1028 Fitting took 1767.17 seconds Current function value: 0.000065 Iterations: 1000 Function evaluations: 1023 Gradient evaluations: 1023 Fitting took 1998.61 seconds Current function value: 0.000044 Iterations: 1000 Function evaluations: 1024 Gradient evaluations: 1024 Fitting took 3360.17 seconds Current function value: 0.000031 Iterations: 1000 Function evaluations: 1010 Gradient evaluations: 1010 Fitting took 2765.05 seconds Current function value: 0.000024 Iterations: 1000 Function evaluations: 1013 Gradient evaluations: 1013 Traceback (most recent call last): File "/cmmc/ptmp/hgaafer/mambaforge/bin/pacemaker", line 401, in <module> main(sys.argv[1:]) File "/cmmc/ptmp/hgaafer/mambaforge/bin/pacemaker", line 248, in main general_fit.fit() File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/generalfit.py", line 481, in fit self.target_bbasisconfig = self.ladder_fitting(self.initial_bbasisconfig, self.target_bbasisconfig) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/generalfit.py", line 509, in ladder_fitting current_bbasisconfig = self.cycle_fitting(current_bbasisconfig) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/generalfit.py", line 564, in cycle_fitting current_bbasisconfig = self.fit_backend.fit( ^^^^^^^^^^^^^^^^^^^^^ File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/fitadapter.py", line 129, in fit raise e File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/fitadapter.py", line 96, in fit fit_res = self.run_tensorpot_fit(bbasisconfig, dataframe, loss_spec, fit_config, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/fitadapter.py", line 236, in run_tensorpot_fit self.fitter.fit(dataframe, test_df=test_dataframe, niter=fit_config[FIT_NITER_KW], File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/tensorpotential/fit.py", line 125, in fit self.process_test_metric() File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/tensorpotential/fit.py", line 318, in process_test_metric self.test_metric_callback(curr_test_metrics_data) File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/generalfit.py", line 410, in test_metric_callback self.detect_early_stopping(mode='test') File "/cmmc/ptmp/hgaafer/mambaforge/lib/python3.11/site-packages/pyace/generalfit.py", line 465, in detect_early_stopping raise TestLossChangeTooSmallException(msg) pyace.generalfit.TestLossChangeTooSmallException: EARLY STOPPING: Too small or even positive TEST loss change (best=-9.94e-05 / iter, last=+0.00e+00/iter, threshold = -1.00e-04/iter) within last 200 iterations. Stopping Exception: Potential file output_potential.yaml doesn't existsLoading B-basis from 'output_potential.yaml' Traceback (most recent call last): File "/cmmc/ptmp/hgaafer/mambaforge/bin/pace_yaml2yace", line 28, in <module> bbasis = ACEBBasisSet(input_yaml_filename) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ValueError: Potential file output_potential.yaml doesn't exists