8000 add OptimalSteps by bebebe666 · Pull Request #7584 · comfyanonymous/ComfyUI · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

add OptimalSteps #7584
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged
merged 1 commit into from
Apr 13, 2025
Merged

add OptimalSteps #7584

merged 1 commit into from
Apr 13, 2025
+57 −0

Conversation

bebebe666
Copy link
Contributor

Hello, this is the implementation of the method OptimalSteps, which can accelerate diffusion model sampling significantly, almost without quality loss. The added node "OptimalStepsScheduler" supports the acceleration for text-to-image model FLUX and text-to-video model Wan. Here, we give a demo result of FLUX model with 10 sampling steps:
pr
Here are the example workflows with OptimalSteps for FLUX-workflow and Wan-workflow.
For more information and results, please visit our repo and paper.

@comfyanonymous comfyanonymous merged commit 9ee6ca9 into comfyanonymous:master Apr 13, 2025
5 checks passed
@komikndr
Copy link
komikndr commented Apr 13, 2025 •
edited
Loading

Hi, sorry, I have a few questions and would appreciate some clarification:

  1. Does this work with Wan I2V models?

  2. I downloaded your workflows and noticed that they use 20 steps with the Euler sampler. Is 20 steps in this setup equivalent to 40 steps in a non-OptimalSteps workflow? Also, can this workflow work with other samplers besides Euler?

  3. Following up on point 2, I normally use 19 steps with UniPC in my non-OptimalSteps workflow. Logically, I tried reducing it to 10 steps with UniPC, but this resulted in blotchy artifacts, similar to what happens when the model doesn’t have enough steps.

  4. So I tried using 20 steps as suggested in the example workflow, but it still takes the same time per iteration as my normal workflow, 73s/it, or 43s/it when using SageAttn2.

Thanks for your time

My Specification

  1. Windows 10, 22H2, 19045.5247
  2. ComfyUI latest windows portable
  3. Python 3.11.19
  4. RTX 3090, 64GB RAM
  5. Optimization, SageAttn2 qk_int8_pv_fp16_triton

After a couple of testing:

  1. It works in 10 steps without producing nasty artifact, cheers!!!. I used a wrong sampler(uni_pc) instead of euler

@EricBCoding
Copy link
EricBCoding commented Apr 13, 2025 •
edited
Loading

Very cool. This method appears to work for SDXL too. I normally generate images in ~8 steps using DMD2, but I added the following noise schedule to the OptimalSteps node and am seeing great results in only 4-5 steps:

"SDXL": [12.1769, 9.9182, 7.0887, 4.5944, 2.2473, 0.9020, 0.2872, 0.0738, 0.0197, 0.0020, 0.0]

Pairs well with the LCM or Euler A samplers.

These levels were adapted from the AYS scheduler. Not 100% sure on my math--maybe someone can suggest better values--but so far I'm impressed with the results. And for Flux, it's a total gamechanger. That model needs all the help it can get, as far as inference speed goes.

@asuonpaa asuonpaa mentioned this pull request Apr 13, 2025
Open
@bebebe666
Copy link
Contributor Author

Thank you for your interest and for reaching out here. Here are some clarifications:

  1. So far, we don't support the Wan I2V and samplers like UniPC.
  2. The overhead of our 20 steps is equal to the non-OptimalSteps workflow 20 steps.
  3. We recommend using the default model in our workflow; the optimalsteps always be different for different models.

Although the default workflow is the one we recommend, feel free to try other combinations to find interesting things!

Finally, We are glad our research was able to help or even bring a little excitement to you! Feel free to contact me for any problems or something we can help!

@anduvo
Copy link
anduvo commented Apr 14, 2025

hi there, very excited about this!
I got your sample flow, it works with the prompt of the sports woman but when I change to another prompt that generates more details, I get a very pixelated image.
It seems its very good when there are gradient of colors but not good with fine details. Is it like this ? or am I doing something wrong? please advise.

test1

prompt used:
A rugged off-road vehicle navigates through a perilous jungle path carved along the edge of a towering limestone cliff. Massive tree roots intertwine with ancient stone, creating natural archways draped with vibrant emerald vines and flowering orchids. Dappled sunlight filters through the dense canopy above, creating dramatic shafts of golden light that illuminate suspended particles of mist and pollen. The cliff face plunges dramatically hundreds of feet to a valley floor blanketed in layers of verdant rainforest, where a serpentine river cuts through like a silver ribbon. The car's metallic surface catches and reflects fragments of the surrounding jungle colors, enhancing its integration with the wild environment. Droplets of condensation cling to the vehicle's exterior, suggesting the intense humidity of the tropical climate. The composition balances tension between the engineered precision of the vehicle and the chaotic vitality of untamed nature, shot from a dynamic perspective that emphasizes the precarious position between vertical rock and infinite drop. Hyperrealistic rendering with atmospheric perspective creates depth through layers of misty jungle receding into the distance. Photographic techniques include selective focus with shallow depth of field, cinematic color grading with rich greens and earthy browns, and high dynamic range lighting that captures both shadow detail and highlight brilliance.

@prezofpizza
Copy link

Sorry if this is a silly question, I can't seem to use the scheduler.

I've updated comfyUI, interface and all. Any input is appreciated, thanks!

RTX4090 - Pytorch 12.8.

@prezofpizza
Copy link

Switched to ComfyUI Nightly and it works. Cheers!

@bebebe666
Copy link
Contributor Author

hi there, very excited about this! I got your sample flow, it works with the prompt of the sports woman but when I change to another prompt that generates more details, I get a very pixelated image. It seems its very good when there are gradient of colors but not good with fine details. Is it like this ? or am I doing something wrong? please advise.

Hello, thank you for your interest and testing! We tested our OptimalSteps on your prompt doesn't have an exciting result. But it is at least comparable with the no-OptimalSteps 10 steps, so you can always have a try before adding more resources. Thank you again for providing this valuable prompt for us, this is helpful for our further research.

@pamparamm pamparamm mentioned this pull request Apr 18, 2025
Closed
@Foul-Tarnished
Copy link

Any idea if this work with HiDream-I1 ?
Or LTXV 0.9.6 ?

@EricBCoding
Copy link

@Foul-Tarnished I tested OptimalSteps with LTXV 0.9.6 Distilled yesterday - the WAN preset works great, even down to 4-5 steps sometimes.

@EricBCoding EricBCoding mentioned this pull request Apr 25, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yoland68 yoland68 Awaiting requested review from yoland68 yoland68 is a code owner

@robinjhuang robinjhuang Awaiting requested review from robinjhuang robinjhuang is a code owner

@huchenlei huchenlei Awaiting requested review from huchenlei huchenlei is a code owner

@pythongosssss pythongosssss Awaiting requested review from pythongosssss pythongosssss is a code owner

@ltdrdata ltdrdata Awaiting requested review from ltdrdata ltdrdata is a code owner

@Kosinkadink Kosinkadink Awaiting requested review from Kosinkadink Kosinkadink is a code owner

@webfiltered webfiltered Awaiting requested review from webfiltered webfiltered is a code owner

@comfyanonymous comfyanonymous Awaiting requested review from comfyanonymous comfyanonymous is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@bebebe666 @komikndr @EricBCoding @anduvo @prezofpizza @Foul-Tarnished @comfyanonymous
0