feat: openai wandb sync - #64
Conversation
Sorry, something went wrong.
| entity=args.entity, | ||
| force=args.force, | ||
| ) | ||
| print(resp) |
There was a problem hiding this comment.
Why is a print statement required after openai logger sync is called?
Sorry, something went wrong.
There was a problem hiding this comment.
It will show wandb log completed successfully to confirm the command ran successfully.
It was just to follow the pattern of the other commands but it could be done directly in the logger if you prefer.
Sorry, something went wrong.
There was a problem hiding this comment.
Thanks - makes sense as is!
Sorry, something went wrong.
rachellim
left a comment
There was a problem hiding this comment.
Thanks for adding this! LGTM, besides a few nitty comments
Sorry, something went wrong.
| sub.add_argument("-i", "--id", help="The id of the fine-tune job (optional)") | ||
| sub.add_argument( | ||
| "-n", | ||
| "--n_jobs", |
There was a problem hiding this comment.
We generally refer to these in our documentation / everywhere as "fine-tunes" instead of "fine-tune jobs". Can you remove references to jobs everywhere in this PR?
Sorry, something went wrong.
| @@ -0,0 +1,277 @@ | |||
| try: | |||
There was a problem hiding this comment.
should we name this module and class wandb_logger / WandbLogger respectively to make it more distinct from more generic logging lirbaries?
Sorry, something went wrong.
| ] | ||
|
|
||
| if not show_warnings and not any(fine_tune_logged): | ||
| print("No new successful fine-tune were found") |
There was a problem hiding this comment.
fine-tunes
Sorry, something went wrong.
| fine_tunes = fine_tunes["data"][-n_jobs if n_jobs is not None else None :] | ||
|
|
||
| # log starting from oldest fine_tune | ||
| show_warnings = False if id is None and n_jobs is None else True |
There was a problem hiding this comment.
Nit: Either [1] add a comment to briefly explain the show_warnings logic here and in L81 ("Show individual warnings if the user specifies a fine-tune or a specific number of fine-tunes, otherwise only warn if there are no new successful fine-tunes to sync."), or [2] rename this to show_individual_warnings. I think it'll make the warning logic slightly easier to grok quickly.
Sorry, something went wrong.
There was a problem hiding this comment.
Good idea!
Sorry, something went wrong.
|
|
||
| # start a wandb run | ||
| wandb.init( | ||
| job_type="finetune", |
There was a problem hiding this comment.
Nit: Should this be fine_tune or fine-tune to be consistent with our API? (Does finetune mean something special within your system?)
Sorry, something went wrong.
|
Git was able to solve the conflicts without my help! |
Sorry, something went wrong.
* feat: log fine_tune with wandb * feat: ensure we are logged in * feat: cli wandb namespace * feat: add fine_tuned_model to summary * feat: log training & validation files * feat: re-log if was not successful or force * doc: add docstring * feat: set wandb api only when needed * fix: train/validation files are inputs * feat: rename artifact type * feat: improve config logging * feat: log all jobs by default * feat: log job details * feat: log -> sync * feat: cli wandb log -> sync * fix: validation_files not always present * feat: format created_at + style * feat: log number of training/validation samples * feat(wandb): avoid download if file already synced * feat(wandb): add number of items to metadata * fix(wandb): allow force sync * feat(wandb): job -> fine-tune * refactor(wandb): use show_individual_warnings * feat(wandb): Logger -> WandbLogger * feat(wandb): retrive number of items from artifact * doc(wandb): add link to documentation
Usage - CLI
$ openai wandb sync --help usage: openai wandb sync [-h] [-i ID] [-n N_JOBS] [--project PROJECT] [--entity ENTITY] [--force] optional arguments: -h, --help show this help message and exit -i ID, --id ID The id of the fine-tune job -n N_JOBS, --n_jobs N_JOBS Number of most recent fine-tune jobs to log when an id is not provided --project PROJECT Name of the project where you're sending runs. By default, it is "GPT-3". --entity ENTITY Username or team name where you're sending runs. By default, your default entity is used, which is usually your username. --force Forces logging and overwrite existing wandb run of the same finetune job.Usage - Python
Reference