| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
|
Hi @charchit7 @staghado @EdoardoBotta. I've opened this PR since I had some free time today and was trying to implement the community version of this as Yiyi suggested. I was only able to get in touch with charchit7 on discord and it seems like he's a little busy currently. If you're still interested, and if anyone else is looking to help out with the implementation, please DM on wandereronarock#0000 (Arrow#1334) so I can add you as a collaborator to my fork and colab notebook. |
Sorry, something went wrong.
|
Sure, go ahead @a-r-r-o-w I'll hope in 5 days. You can add me if that'll not be too late. (Ref PR : #5872) |
Sorry, something went wrong.
|
@a-r-r-o-w I'm interested in being added. I have also been experimenting with the pipelines at https://github.com/EdoardoBotta/diffusers/blob/controlanimate/examples/community/animatediff_controlnet.py. It would be great to merge the changes. |
Sorry, something went wrong.
@charchit7 Sure! Thanks for pointing out the incorrect issue number - fixed.
@EdoardoBotta Sending a collaborator invite to my fork. After taking a quick look, it seems like we've both adopted code from existing pipelines and have similar changes for the most part. I've currently commented out all MultiControlNet related stuff since I wanted a single one to work first. Would be great if you could add in those changes! |
Sorry, something went wrong.
|
@yiyixuxu Seems like it works, however I'm not too sure about the quality. Will run a few experiments to see if this can be improved. Edit: Referencing #5881 here because of the similarity in output quality. Results
|
||||||||
Sorry, something went wrong.
Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com>
There was a problem hiding this comment.
Super cool pipeline!
Could you also add a short section to https://github.com/huggingface/diffusers/blob/main/examples/community/README.md ?
Sorry, something went wrong.
|
Ah, forgot doing that. Doing so now |
Sorry, something went wrong.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
Sorry, something went wrong.
|
How can I make the controlnet only affect the first frame or certain frames? |
Sorry, something went wrong.
There was a problem hiding this comment.
Good to merge for me once all tests are green
Sorry, something went wrong.
wow, awesome ❤️ this is probably the first PR I consider as actual good work from my end haha. Also big thanks to @EdoardoBotta. He had something working as well on his fork which helped with the multicontrolnet support that I couldn't get working initially.
@loboere I definitely think this is possible. Let me try to experiment a bit and I'll add to this pipeline if I'm able to get it to work. |
Sorry, something went wrong.
* begin work on animatediff + controlnet pipeline * complete todos, uncomment multicontrolnet, input checks Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com> * update Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com> * add example * update community README * Update examples/community/README.md --------- Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
|
@a-r-r-o-w, @EdoardoBotta nicely done! |
Sorry, something went wrong.
|
This is incredible work! Well done!!! This is very exciting. Thank you for setting this up for all of us. I was wondering if you guys knew of a strategy to have consistent animations past the 16 frame limit? Say for example, a video of 32 frames, I run this for the first 16, then I would run this again for the next 16 frames, but when I stitch it together, it's obvious when the first 16 frames end. I've tried setting the seed manually, but that doesn't seem to work. The weirdest part is that the same seed works to generate the same 16 frames whether its the first half or the second half. I've tried including the first frame and the last 16 frames to see if it had to do with the interpolation that included the first frame, but that doesn't seem to work. This is for the pose or canny controlnets. I'm on a m2 macbook pro, so I can't really run things in a single python script before I run out of memory. My strategy so far has been running python animatediff.py 0 16 and then python aniamtediff.py 16 32 -- any ideas? Thank you! Edit: I just really have to emphasize, how amazing it is that this was set up by all of you. Thanks again! |
Sorry, something went wrong.
|
Thanks for the kind words @rmasiso! This is indeed possible and I'm experimenting on adding support for arbitrary number of frames to both animatediff and this community pipeline. I have a non-functional notebook at the moment but I'm getting there slowly by reversing this comfy-ui code that does support longer generations. |
Sorry, something went wrong.
…5839) * add poc for benchmarking workflow. * import * fix argument * fix: argument * fix: path * fix * fix * path * output csv files. * workflow cleanup * append token * add utility to push to hf dataset * fix: kw arg * better reporting * fix: headers * better formatting of the numbers. * better type annotation * fix: formatting * moentarily disable check * push results. * remove disable check * introduce base classes. * img2img class * add inpainting pipeline * intoduce base benchmark class. * add img2img and inpainting * feat: utility to compare changes * fix * fix import * add args * basepath * better exception handling * better path handling * fix * fix * remove * ifx * fix * add: support for controlnet. * image_url -> url * move images to huggingface hub * correct urls. * root_ckpt * flush before benchmarking * don't install accelerate from source * add runner * simplify Diffusers Benchmarking step * change runner * fix: subprocess call. * filter percentage values * fix controlnet benchmark * add t2i adapters. * fix filter columns * fix t2i adapter benchmark * fix init. * fix * remove safetensors flag * fix args print * fix * feat: run_command * add adapter resolution mapping * benchmark t2i adapter fix. * fix adapter input * fix * convert to L. * add flush() add appropriate places * better filtering * okay * get env for torch * convert to float * fix * filter out nans. * better coment * sdxl * sdxl for other benchmarks. * fix: condition * fix: condition for inpainting * fix: mapping for resolution * fix * include kandinsky and wuerstchen * fix: Wuerstchen * Empty-Commit * [Community] AnimateDiff + Controlnet Pipeline (#5928) * begin work on animatediff + controlnet pipeline * complete todos, uncomment multicontrolnet, input checks Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com> * update Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com> * add example * update community README * Update examples/community/README.md --------- Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * EulerDiscreteScheduler add `rescale_betas_zero_snr` (#6024) * EulerDiscreteScheduler add `rescale_betas_zero_snr` * Revert "[Community] AnimateDiff + Controlnet Pipeline (#5928)" This reverts commit 821726d. * Revert "EulerDiscreteScheduler add `rescale_betas_zero_snr` (#6024)" This reverts commit 3dc2362. * add SDXL turbo * add lcm lora to the mix as well. * fix * increase steps to 2 when running turbo i2i * debug * debug * debug * fix for good * fix and isolate better * fuse lora so that torch compile works with peft * fix: LCMLoRA * better identification for LCM * change to cron job --------- Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com> Co-authored-by: Aryan V S <contact.aryanvs@gmail.com> Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com> Co-authored-by: Beinsezii <39478211+Beinsezii@users.noreply.github.com>
|
Amazing work @a-r-r-o-w ! I was wondering if the current animatediff pipeline would work with SDXL and its related controlnets and motion module ? Best |
Sorry, something went wrong.
Thanks! It is indeed possible. See here: https://github.com/guoyww/AnimateDiff/blob/sdxl/. I'll try to make a notebook and share it here soon. @yiyixuxu @patrickvonplaten @sayakpaul AnimateDiff with SDXL seems like a great addition as the quality looks really cool. Would you be open to PRs adding support for it? I'm not sure if the current diffusers implementation will work since I see a few implementation differences in the AnimateDiff repo and here. Will test more and get back. |
Sorry, something went wrong.
|
Just a quick notebook update before I go sleep: Colab. I'm yet to replicate the results of the sdxl branch as demonstrated in the repo. This notebook is able to get the inference process up and running. Current Results
Also, it probably makes sense to create a new issue asking for support of these features at this point :) |
||||
Sorry, something went wrong.
This is exciting! Is the hack they did in comfy-ui, the idea that one converts the entire video into latents first, and then sends them through animate-diff for the 16 frame interpolation in a sort of rolling context window? Similar to my hack, but it would end up more consistent, because it's in latent space from the get-go? |
Sorry, something went wrong.
* begin work on animatediff + controlnet pipeline * complete todos, uncomment multicontrolnet, input checks Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com> * update Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com> * add example * update community README * Update examples/community/README.md --------- Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
…uggingface#5839) * add poc for benchmarking workflow. * import * fix argument * fix: argument * fix: path * fix * fix * path * output csv files. * workflow cleanup * append token * add utility to push to hf dataset * fix: kw arg * better reporting * fix: headers * better formatting of the numbers. * better type annotation * fix: formatting * moentarily disable check * push results. * remove disable check * introduce base classes. * img2img class * add inpainting pipeline * intoduce base benchmark class. * add img2img and inpainting * feat: utility to compare changes * fix * fix import * add args * basepath * better exception handling * better path handling * fix * fix * remove * ifx * fix * add: support for controlnet. * image_url -> url * move images to huggingface hub * correct urls. * root_ckpt * flush before benchmarking * don't install accelerate from source * add runner * simplify Diffusers Benchmarking step * change runner * fix: subprocess call. * filter percentage values * fix controlnet benchmark * add t2i adapters. * fix filter columns * fix t2i adapter benchmark * fix init. * fix * remove safetensors flag * fix args print * fix * feat: run_command * add adapter resolution mapping * benchmark t2i adapter fix. * fix adapter input * fix * convert to L. * add flush() add appropriate places * better filtering * okay * get env for torch * convert to float * fix * filter out nans. * better coment * sdxl * sdxl for other benchmarks. * fix: condition * fix: condition for inpainting * fix: mapping for resolution * fix * include kandinsky and wuerstchen * fix: Wuerstchen * Empty-Commit * [Community] AnimateDiff + Controlnet Pipeline (huggingface#5928) * begin work on animatediff + controlnet pipeline * complete todos, uncomment multicontrolnet, input checks Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com> * update Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com> * add example * update community README * Update examples/community/README.md --------- Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * EulerDiscreteScheduler add `rescale_betas_zero_snr` (huggingface#6024) * EulerDiscreteScheduler add `rescale_betas_zero_snr` * Revert "[Community] AnimateDiff + Controlnet Pipeline (huggingface#5928)" This reverts commit 821726d. * Revert "EulerDiscreteScheduler add `rescale_betas_zero_snr` (huggingface#6024)" This reverts commit 3dc2362. * add SDXL turbo * add lcm lora to the mix as well. * fix * increase steps to 2 when running turbo i2i * debug * debug * debug * fix for good * fix and isolate better * fuse lora so that torch compile works with peft * fix: LCMLoRA * better identification for LCM * change to cron job --------- Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com> Co-authored-by: Aryan V S <contact.aryanvs@gmail.com> Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com> Co-authored-by: Beinsezii <39478211+Beinsezii@users.noreply.github.com>
|
Thank you for the pipe.Can't this pipe use multiple ControlNets like other pipes? |
Sorry, something went wrong.
It does support multiple controlnets. Does it error out when you use it with multiple? If so, I'll look into fixing it quickly. |
Sorry, something went wrong.
|
when I do this: self.tile = ControlNetModel.from_pretrained(
"/home/img2video/models/control/control_v11f1e_sd15_tile", torch_dtype=torch.float16
)
self.pose = ControlNetModel.from_pretrained(
"/home/lora_test/model/controlnet_pose", torch_dtype=torch.float16
)
self.pose_processor = OpenposeDetector.from_pretrained('/home/lora_test/model/pose_dete').to('cuda')
self.adapter = MotionAdapter.from_pretrained("/home/AnimateDiff-main/models/Motion_Module",
torch_dtype=torch.float16)
self.pipe = AnimateDiffControlNetPipeline.from_pretrained(
"/home/lora_test/model/AWPainting_v1.2",
motion_adapter=self.adapter,
controlnet=[self.pose, self.tile],
torch_dtype=torch.float16)
I get : Traceback (most recent call last): |
Sorry, something went wrong.
|
Okay, I see the issue. Not sure why/how I removed the line to handle multicontrolnet. In the __init__ function, the following line seems to be missing: if isinstance(controlnet, (list, tuple)):
controlnet = MultiControlNetModel(controlnet)There might be other things I've missed. I'll test and open a PR fixing it in a bit. Thanks for reporting this! |
Sorry, something went wrong.
* begin work on animatediff + controlnet pipeline * complete todos, uncomment multicontrolnet, input checks Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com> * update Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com> * add example * update community README * Update examples/community/README.md --------- Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
…uggingface#5839) * add poc for benchmarking workflow. * import * fix argument * fix: argument * fix: path * fix * fix * path * output csv files. * workflow cleanup * append token * add utility to push to hf dataset * fix: kw arg * better reporting * fix: headers * better formatting of the numbers. * better type annotation * fix: formatting * moentarily disable check * push results. * remove disable check * introduce base classes. * img2img class * add inpainting pipeline * intoduce base benchmark class. * add img2img and inpainting * feat: utility to compare changes * fix * fix import * add args * basepath * better exception handling * better path handling * fix * fix * remove * ifx * fix * add: support for controlnet. * image_url -> url * move images to huggingface hub * correct urls. * root_ckpt * flush before benchmarking * don't install accelerate from source * add runner * simplify Diffusers Benchmarking step * change runner * fix: subprocess call. * filter percentage values * fix controlnet benchmark * add t2i adapters. * fix filter columns * fix t2i adapter benchmark * fix init. * fix * remove safetensors flag * fix args print * fix * feat: run_command * add adapter resolution mapping * benchmark t2i adapter fix. * fix adapter input * fix * convert to L. * add flush() add appropriate places * better filtering * okay * get env for torch * convert to float * fix * filter out nans. * better coment * sdxl * sdxl for other benchmarks. * fix: condition * fix: condition for inpainting * fix: mapping for resolution * fix * include kandinsky and wuerstchen * fix: Wuerstchen * Empty-Commit * [Community] AnimateDiff + Controlnet Pipeline (huggingface#5928) * begin work on animatediff + controlnet pipeline * complete todos, uncomment multicontrolnet, input checks Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com> * update Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com> * add example * update community README * Update examples/community/README.md --------- Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * EulerDiscreteScheduler add `rescale_betas_zero_snr` (huggingface#6024) * EulerDiscreteScheduler add `rescale_betas_zero_snr` * Revert "[Community] AnimateDiff + Controlnet Pipeline (huggingface#5928)" This reverts commit 821726d. * Revert "EulerDiscreteScheduler add `rescale_betas_zero_snr` (huggingface#6024)" This reverts commit 3dc2362. * add SDXL turbo * add lcm lora to the mix as well. * fix * increase steps to 2 when running turbo i2i * debug * debug * debug * fix for good * fix and isolate better * fuse lora so that torch compile works with peft * fix: LCMLoRA * better identification for LCM * change to cron job --------- Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com> Co-authored-by: Aryan V S <contact.aryanvs@gmail.com> Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com> Co-authored-by: Beinsezii <39478211+Beinsezii@users.noreply.github.com>
| Back | FazBrowse Home | New Git URL |
What does this PR do?
Community pipeline version for #5866.
Colab Notebook
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@yiyixuxu @patrickvonplaten