FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Prefix LM Eval by Muennighoff · Pull Request #313 · bigscience-workshop/Megatron-DeepSpeed · GitHub

/ Megatron-DeepSpeed Public

Prefix LM Eval - #313

Open
Muennighoff wants to merge 3 commits into
bseval_harnessfrom
prefixbseval
ConversationCommits (3)ChecksFiles changed
Open

Prefix LM Eval#313
Muennighoff wants to merge 3 commits into
bseval_harnessfrom
prefixbseval

Conversation

Muennighoff commented Jul 16, 2022 •
edited
Loading

Copy link
Copy Markdown
Collaborator

This PR adapts evaluation to work with Prefix LMs, such as used for T0 finetuning experiments.

Using the normal eval harness I get the following results:

Using CHECKPOINT_PATH=$six_ALL_CCFRSCRATCH/checkpoints/tr11f-6B3-ml/checkpoints/main/global_step163750 (CKPT prior to MTF):
copa "acc": 0.58

Using CHECKPOINT_PATH=/gpfsscratch/rech/six/commun/checkpoints/tr13f-6B3-ml-t0/checkpoints/prefix/global_step2000:
copa "acc": 0.7

Using CHECKPOINT_PATH=/gpfsscratch/rech/six/commun/checkpoints/tr13f-6B3-ml-t0/checkpoints/prefix/global_step3100:
copa "acc": 0.67

Using CHECKPOINT_PATH=/gpfsscratch/rech/six/commun/checkpoints/tr13f-6B3-ml-t0/checkpoints/prefix/global_step3100 without --prefix:
copa "acc": 0.73

Muennighoff requested a review from thomasw21 July 16, 2022 14:12

Muennighoff commented Jul 16, 2022

Copy link
Copy Markdown
Collaborator Author

cc @lintangsutawika @haileyschoelkopf - can't add you as reviewers somehow, but would be great if you could take a look.
I'm not 100% sure about the results I got 🧐

lintangsutawika commented Jul 16, 2022

Copy link
Copy Markdown
Contributor

Will take a closer look.

lintangsutawika commented Jul 22, 2022

Copy link
Copy Markdown
Contributor

@Muennighoff so the intended results is suppose to be that with Prefix-LM the performance should be higher, right? However, based on the scores you shared, this does not seem to be the case.

Muennighoff commented Jul 22, 2022

Copy link
Copy Markdown
Collaborator Author

Yeah so according to the current results evaluating the model as a causallm is better than a prefixlm after it was fine-tuned as a prefixlm. Also note:

  • In both cases it is better than prior to fine-tuning.
  • There is no strong performance difference for the CD + CD & CD + ND models in the What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? paper. I.e. for
CD:FLM (219B) + CD:MTF (13B)
CD:FLM (219B) + ND:MTF (13B

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

thomasw21 Awaiting requested review from thomasw21

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

Back | FazBrowse Home | New Git URL