CalculatedContent/WeightWatcher-Examples: User Examples of the WeightWatcher project · GitHub

D# WeightWatcher-Examples A curated collection of real-world examples, notebooks, and experiments using WeightWatcher, the open‑source tool for analyzing layer-wise spectra, heavy‑tailed behavior, power‑law exponents (α), correlation traps, and model quality throughout training.

These examples span small MLPs, double descent, and billion-parameter LLMs.

Single layer example

• SingleLayerWWExample.ipynb

How to analyze fine-tuned models

• WW_FIneTuned_Alphas.ipynb
• WW_PEFT.ipynb

How varying the batch size and/or learning rates affect convergence

• WW-MLP3-BatchSizes.ipynb
• WW_MLP3_LearningRates.ipynb

Explaining Epoch-wise Double Descent

• WW_MLP3_LearningRates.ipynb

Comparing the inductive biases between AdamW and Muon

• MLP3-MNIST-AdamW.ipynb
• MLP3-MNIST-Muon.ipynb

MLP3 on CIFAR10: Extreme overfitting in the first layer

• MLP3_CIFAR10.ipynb

AdamW vs Muon: Muon alphas converge, but very slowly

• AdamWvsMuon.ipynb

Post Analysis of the paper "Overtrained Language Models Are Harder to Fine-Tune"

• OLMO1B.ipynb
• OLMO1B_Fine_Tuning_Results.csv

The Magic of Mistral Dragon Kings blog

• WW_Mistral_DragonKings.ipynb

Expeperiment Method: SVD Smooting

• WW_SVDSMoothing_TinyLLaMAipynb.ipynb

The original 1989 Double Descent Experiment (https://calculatedcontent.com/2024/03/01/describing-double-descent-with-weightwatcher/)

• F_Vallet_Full.ipynb
• WW_DoubleDesecent.ipynb

Comparing BERT, RoBERTa, XLNet

• WW-BERT-BlogExample.ipynb

ONNX Format

• WW-ONNX.ipynb

Old experiments on random labels

• random_labels/

• How α < 2 identifies overfitting & correlation traps
• Spectral phase transitions during training
• Epoch-wise double descent behavior
• Optimizer differences (Muon vs AdamW vs SGD)
• Fine‑tuning shifts between underfit → well‑fit → overfit
• Diagnostics for memorization and rank collapse

git clone https://github.com/CalculatedContent/WeightWatcher-Examples.git
cd WeightWatcher-Examples
pip install weightwatcher
jupyter notebook

MIT License — see LICENSE