FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

DPMSolverMultistep add `rescale_betas_zero_snr` by Beinsezii · Pull Request #7097 · huggingface/diffusers · GitHub

/ diffusers Public

DPMSolverMultistep add rescale_betas_zero_snr - #7097

Merged
yiyixuxu merged 7 commits into
huggingface:mainfrom
Beinsezii:dpm_zsnr
Feb 27, 2024
ConversationCommits (7)ChecksFiles changed
Merged

DPMSolverMultistep add rescale_betas_zero_snr#7097
yiyixuxu merged 7 commits into
huggingface:mainfrom
Beinsezii:dpm_zsnr

Conversation

Beinsezii commented Feb 26, 2024

Copy link
Copy Markdown
Contributor

Effectively identical to the implementation for EulerDiscrete in #6024

TL;DR:

  • rescale_zero_terminal_snr function copied from DDIM
  • Fudge the last cumprod to 2**-24 from 0 when using zsnr so it doesn't produce an inf sigma
  • Upcast samples to torch.float32 for the duration of step() because the zsnr sigmas are substantially less numerically stable and the performance hit is negligible.
    • Also has the side effect of producing slightly better images overall
  • use_karras_sigmas=True still produces strange results as it does in EulerDiscrete. Might be worth investigating separately?
  • Simple true/false UT for the new config value

Demo images

use_karras_sigmas also seems to just generally produce bad/noisy results even without ZSNR so I might not have the config set up correctly?

yiyixuxu reviewed Feb 26, 2024
View reviewed changes

yiyixuxu left a comment •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

oh thanks
left a question about dtype

Comment thread src/diffusers/schedulers/scheduling_dpmsolver_multistep.py Outdated
self._init_step_index(timestep)

# store old dtype because model_output isn't always the same it seems
return_dtype = sample.dtype

yiyixuxu Feb 26, 2024

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

do you mean model_output.dtype isn't always the same as sample.dtype before the upcast?

Beinsezii Feb 26, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Turns out that was because model_output = self.convert_model_output(model_output, sample=sample) ends up creating a shadowed tensor cast to the sample's dtype. I moved the sample upcast after this call so return_type is no longer needed. Outputs are the same.

yiyixuxu commented Feb 26, 2024

Copy link
Copy Markdown
Collaborator

use_karras_sigmas=True still produces strange results as it does in EulerDiscrete. Might be worth investigating separately?

yes separate issue please:)

yiyixuxu approved these changes Feb 27, 2024
View reviewed changes

yiyixuxu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

thanks!

yiyixuxu commented Feb 27, 2024

Copy link
Copy Markdown
Collaborator

can you fix the quality test?

HuggingFaceDocBuilderDev commented Feb 27, 2024

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Beinsezii commented Feb 27, 2024

Copy link
Copy Markdown
Contributor Author

Alright I added a single newline.

yiyixuxu merged commit 2e31a75 into huggingface:main Feb 27, 2024

yiyixuxu commented Feb 27, 2024

Copy link
Copy Markdown
Collaborator

thank you!

Beinsezii deleted the dpm_zsnr branch February 29, 2024 10:42
Beinsezii mentioned this pull request Apr 3, 2024
Closed
yiyixuxu mentioned this pull request May 8, 2024
Merged

lachlan-nicholson commented Jun 5, 2024

Copy link
Copy Markdown
Contributor

use_karras_sigmas=True still produces strange results as it does in EulerDiscrete. Might be worth investigating separately?

yes separate issue please:)

Have we created a separate issue for ZeroSNR + Karras?
At a glance the max lambda is very large and could be clipped similar to #3314, potentially via config.sigma_max.
This gives viable results but introduces a new parameter.
I'm left wondering what it means to apply both ZeroSNR and Karras modifiers at the same time.

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

yiyixuxu yiyixuxu approved these changes

Assignees

No one assigned

Labels

scheduler

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

Back | FazBrowse Home | New Git URL