This folder gathers training scripts for the different arch/scaling and engineering experiments. The naming convention is tr<number>-<short-description>. The current baseline that architecture and scaling experiments compare to is tr3d. In order to launch a new experiment, you should probably start from the arch-and-scaling template.

Some tips:

• TFlops optimization: How to make sure that given a set of hardware you optimize the speed at which you train.
• Instrumentation: How to sync with the hub

Location of the checkpoints of the trained models plus logs and anything else of importance - e.g. eval harness results:

• tr1-13B: gs://bigscience-backups/tr1-13B/

• tr3m-1B3-emb-norm-pile: $six_ALL_CCFRSTORE/checkpoints/tr3m-1B3-emb-norm-pile

• tr4-1B3-rotary: `$six_ALL_CCFRSTORE/checkpoints/

• tr4b-350M-rotary: `$six_ALL_CCFRSTORE/checkpoints/

• tr4c-1B3-rotary-oscar: $six_ALL_CCFRSTORE/checkpoints/tr4c-1B3-rotary-oscar

• tr6-1B3-prefix-lm: $six_ALL_CCFRSTORE/checkpoints/tr6-1B3-prefix-lm

• tr6-1B3-prefix-lm-unbiased-loss: $six_ALL_CCFRSTORE/checkpoints/tr6-1B3-prefix-lm-unbiased-loss

• tr6b-350M-prefix-lm: $six_ALL_CCFRSTORE/checkpoints/tr6b-350M-prefix-lm

• tr6b-350M-prefix-lm-PP2: $six_ALL_CCFRSTORE/checkpoints/tr6b-350M-prefix-lm-PP2

• tr6b-350M-prefix-lm-unbiased-loss: $six_ALL_CCFRSTORE/checkpoints/tr6b-350M-prefix-lm-unbiased-loss

• tr6c-350M-prefix-lm-reset-attention-mask: $six_ALL_CCFRSTORE/checkpoints/tr6c-350M-prefix-lm-reset-attention-mask

• tr6c-350M-prefix-lm-reset-attention-mask.backup: $six_ALL_CCFRSTORE/checkpoints/tr6c-350M-prefix-lm-reset-attention-mask.backup

• tr6d-350M-prefix-lm-pile: $six_ALL_CCFRSTORE/checkpoints/tr6d-350M-prefix-lm-pile

• tr6e-1B3-pile: $six_ALL_CCFRSTORE/checkpoints/tr6e-1B3-pile

• tr6f-1B3-oscar-no-loss-on-targets-only: $six_ALL_CCFRSTORE/checkpoints/tr6f-1B3-oscar-no-loss-on-targets-only

• tr6g-1B3-oscar-loss-reweighting: $six_ALL_CCFRSTORE/checkpoints/tr6g-1B3-oscar-loss-reweighting

• tr7a-1B3-alibi (not a real alibi pos embedding experiment - the alibi matrix were not used in this experiment): $six_ALL_CCFRSTORE/checkpoints/tr7a-1B3-alibi

• tr7b-350-alibi (not a real alibi pos embedding experiment - the alibi matrix were not used in this experiment): $six_ALL_CCFRSTORE/checkpoints/tr7b-350M-alibi

• tr7d-1B3-alibi: six_ALL_CCFRSTORE/checkpoints/tr7d-1B3-alibi

• tr9b-350M-swiglu: six_ALL_CCFRSTORE/checkpoints/tr9b-350M-swiglu

• tr9c-1B3-swiglu-pile: six_ALL_CCFRSTORE/checkpoints/tr9b-1B3-swiglu-pile