FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [View Raw Code]   [Original HTTPS Page]

algorithmic-efficiency/algorithms/target_setting_algorithms at dev · mlcommons/algorithmic-efficiency · GitHub

/ algorithmic-efficiency Public

Breadcrumbs

  1. algorithmic-efficiency
  2. algorithms

target_setting_algorithms

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History
 
 

Breadcrumbs

  1. algorithmic-efficiency
  2. algorithms

target_setting_algorithms

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
criteo1tb
criteo1tb
 
 
criteo1tb_embed_init
criteo1tb_embed_init
 
 
criteo1tb_layernorm
criteo1tb_layernorm
 
 
criteo1tb_resnet
criteo1tb_resnet
 
 
fastmri
fastmri
 
 
fastmri_layernorm
fastmri_layernorm
 
 
fastmri_model_size
fastmri_model_size
 
 
fastmri_tanh
fastmri_tanh
 
 
fineweb_edu_lm
fineweb_edu_lm
 
 
imagenet_resnet
imagenet_resnet
 
 
imagenet_resnet_gelu
imagenet_resnet_gelu
 
 
imagenet_resnet_large_bn_init
imagenet_resnet_large_bn_init
 
 
imagenet_resnet_silu
imagenet_resnet_silu
 
 
imagenet_vit
imagenet_vit
 
 
imagenet_vit_glu
imagenet_vit_glu
 
 
imagenet_vit_map
imagenet_vit_map
 
 
imagenet_vit_post_ln
imagenet_vit_post_ln
 
 
librispeech_conformer
librispeech_conformer
 
 
librispeech_conformer_attention_temperature
librispeech_conformer_attention_temperature
 
 
librispeech_conformer_gelu
librispeech_conformer_gelu
 
 
librispeech_conformer_layernorm
librispeech_conformer_layernorm
 
 
librispeech_deepspeech
librispeech_deepspeech
 
 
librispeech_deepspeech_no_resnet
librispeech_deepspeech_no_resnet
 
 
librispeech_deepspeech_norm_and_spec_aug
librispeech_deepspeech_norm_and_spec_aug
 
 
librispeech_deepspeech_tanh
librispeech_deepspeech_tanh
 
 
ogbg
ogbg
 
 
ogbg_gelu
ogbg_gelu
 
 
ogbg_model_size
ogbg_model_size
 
 
ogbg_silu
ogbg_silu
 
 
wmt
wmt
 
 
wmt_attention_temp
wmt_attention_temp
 
 
wmt_glu_tanh
wmt_glu_tanh
 
 
wmt_post_ln
wmt_post_ln
 
 
README.md
README.md
 
 
__init__.py
__init__.py
 
 
cosine_warmup.py
cosine_warmup.py
 
 
data_selection.py
data_selection.py
 
 
get_batch_size.py
get_batch_size.py
 
 
jax_adamw.py
jax_adamw.py
 
 
jax_momentum.py
jax_momentum.py
 
 
jax_nadamw.py
jax_nadamw.py
 
 
jax_nesterov.py
jax_nesterov.py
 
 
jax_submission_base.py
jax_submission_base.py
 
 
pytorch_adamw.py
pytorch_adamw.py
 
 
pytorch_momentum.py
pytorch_momentum.py
 
 
pytorch_nadamw.py
pytorch_nadamw.py
 
 
pytorch_nesterov.py
pytorch_nesterov.py
 
 
pytorch_submission_base.py
pytorch_submission_base.py
 
 

README.md

Target Setting Run replications

Original runs were run on Google TPUv2-8 machines.

These are not valid submissions, because they use a different hyperparameter setting per workload. But we include them in order to reproduce how we set the target metric values.

To simplify directory setting, set:

ROOT_DIR=/home/znado

Criteo

Target was set using NAdamW with a linear warmup cosine decay LR schedule.

python3 submission_runner.py \
    --framework=jax \
    --data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=criteo1tb \
    --submission_path=algorithms/target_setting_algorithms/jax_nadamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/criteo1tb/tuning_search_space.json
torchrun --redirects 1:0,2:0,3:0,4:0,5:0,6:0,7:0 --standalone --nnodes=1 --nproc_per_node=8 submission_runner.py \
    --framework=pytorch \
    --data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=criteo1tb \
    --submission_path=algorithms/target_setting_algorithms/pytorch_nadamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/criteo1tb/tuning_search_space.json

FastMRI

Target was set using Nesterov with a linear warmup and linear decay LR schedule.

python3 submission_runner.py \
    --framework=jax \
    --data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=fastmri \
    --submission_path=algorithms/target_setting_algorithms/jax_nesterov.py \
    --tuning_search_space=algorithms/target_setting_algorithms/fastmri/tuning_search_space.json
torchrun --redirects 1:0,2:0,3:0,4:0,5:0,6:0,7:0 --standalone --nnodes=1 --nproc_per_node=8 submission_runner.py \
    --framework=pytorch \
    --data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=fastmri \
    --submission_path=algorithms/target_setting_algorithms/pytorch_nesterov.py \
    --tuning_search_space=algorithms/target_setting_algorithms/fastmri/tuning_search_space.json

ImageNet-Resnet

Target was set using Heavy-ball Momentum with a linear warmup and linear decay LR schedule.

python3 submission_runner.py \
    --framework=jax \
    --data_dir=$ROOT_DIR \
    --imagenet_v2_data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=imagenet_resnet \
    --submission_path=algorithms/target_setting_algorithms/jax_momentum.py \
    --tuning_search_space=algorithms/target_setting_algorithms/imagenet_resnet/tuning_search_space.json
torchrun --redirects 1:0,2:0,3:0,4:0,5:0,6:0,7:0 --standalone --nnodes=1 --nproc_per_node=8 submission_runner.py \
    --framework=pytorch \
    --data_dir=$ROOT_DIR/imagenet_pytorch \
    --imagenet_v2_data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=imagenet_resnet \
    --submission_path=algorithms/target_setting_algorithms/pytorch_momentum.py \
    --tuning_search_space=algorithms/target_setting_algorithms/imagenet_resnet/tuning_search_space.json

ImageNet-ViT

Target was set using NAdamW with a linear warmup cosine decay LR schedule.

python3 submission_runner.py \
    --framework=jax \
    --data_dir=$ROOT_DIR \
    --imagenet_v2_data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=imagenet_vit \
    --submission_path=algorithms/target_setting_algorithms/jax_nadamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/imagenet_vit/tuning_search_space.json
torchrun --redirects 1:0,2:0,3:0,4:0,5:0,6:0,7:0 --standalone --nnodes=1 --nproc_per_node=8 submission_runner.py \
    --framework=pytorch \
    --data_dir=$ROOT_DIR/imagenet_pytorch \
    --imagenet_v2_data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=imagenet_vit \
    --submission_path=algorithms/target_setting_algorithms/pytorch_nadamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/imagenet_vit/tuning_search_space.json

Librispeech-Conformer

Target was set using NAdamW with a linear warmup cosine decay LR schedule.

python3 submission_runner.py \
    --framework=jax \
    --data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=librispeech_conformer \
    --submission_path=algorithms/target_setting_algorithms/jax_adamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/librispeech_conformer/tuning_search_space.json
torchrun --redirects 1:0,2:0,3:0,4:0,5:0,6:0,7:0 --standalone --nnodes=1 --nproc_per_node=8 submission_runner.py \
    --framework=pytorch \
    --data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=librispeech_conformer \
    --submission_path=algorithms/target_setting_algorithms/pytorch_adamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/librispeech_conformer/tuning_search_space.json

Librispeech-Deepspeech

Target was set using NAdamW with a linear warmup cosine decay LR schedule.

python3 submission_runner.py \
    --framework=jax \
    --data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=librispeech_deepspeech \
    --submission_path=algorithms/target_setting_algorithms/jax_nadamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/librispeech_deepspeech/tuning_search_space.json
torchrun --redirects 1:0,2:0,3:0,4:0,5:0,6:0,7:0 --standalone --nnodes=1 --nproc_per_node=8 submission_runner.py \
    --framework=pytorch \
    --data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=librispeech_deepspeech \
    --submission_path=algorithms/target_setting_algorithms/pytorch_nadamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/librispeech_deepspeech/tuning_search_space.json

OGBG

Target was set using Nesterov with a linear warmup and linear decay LR schedule.

python3 submission_runner.py \
    --framework=jax \
    --data_dir=$ROOT_DIR/tensorflow_datasets \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=ogbg \
    --submission_path=algorithms/target_setting_algorithms/jax_nesterov.py \
    --tuning_search_space=algorithms/target_setting_algorithms/ogbg/tuning_search_space.json
torchrun --redirects 1:0,2:0,3:0,4:0,5:0,6:0,7:0 --standalone --nnodes=1 --nproc_per_node=8 submission_runner.py \
    --framework=pytorch \
    --data_dir=$ROOT_DIR/tensorflow_datasets \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=ogbg \
    --submission_path=algorithms/target_setting_algorithms/pytorch_nesterov.py \
    --tuning_search_space=algorithms/target_setting_algorithms/ogbg/tuning_search_space.json

WMT

Target was set using NAdamW with a linear warmup cosine decay LR schedule.

python3 submission_runner.py \
    --framework=jax \
    --data_dir=$ROOT_DIR/tensorflow_datasets \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=wmt \
    --submission_path=algorithms/target_setting_algorithms/jax_nadamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/wmt/tuning_search_space.json
torchrun --redirects 1:0,2:0,3:0,4:0,5:0,6:0,7:0 --standalone --nnodes=1 --nproc_per_node=8 submission_runner.py \
    --framework=pytorch \
    --data_dir=$ROOT_DIR/tensorflow_datasets \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=wmt \
    --submission_path=algorithms/target_setting_algorithms/pytorch_nadamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/wmt/tuning_search_space.json
Back | FazBrowse Home | New Git URL