Add activations_shape info in UNet models #7482
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can you do: list(x.shape) so it's consistent with the other shapes in the dict? |
|
Yes, good idea. Done. |
KohakuBlueleaf
added a commit
to KohakuBlueleaf/ComfyUI
that referenced
this pull request
Apr 8, 2025
* Allow disabling pe in flux code for some other models. * Initial Hunyuan3Dv2 implementation. Supports the multiview, mini, turbo models and VAEs. * Fix orientation of hunyuan 3d model. * A few fixes for the hunyuan3d models. * Update frontend to 1.13 (comfyanonymous#7331) * Add backend primitive nodes (comfyanonymous#7328) * Add backend primitive nodes * Add control after generate to int primitive * Nodes to convert images to YUV and back. Can be used to convert an image to black and white. * Update frontend to 1.14 (comfyanonymous#7343) * Native LotusD Implementation (comfyanonymous#7125) * draft pass at a native comfy implementation of Lotus-D depth and normal est * fix model_sampling kludges * fix ruff --------- Co-authored-by: comfyanonymous <121283862+comfyanonymous@users.noreply.github.com> * Automatically set the right sampling type for lotus. * support output normal and lineart once (comfyanonymous#7290) * [nit] Format error strings (comfyanonymous#7345) * ComfyUI version v0.3.27 * Fallback to pytorch attention if sage attention fails. * Add model merging node for WAN 2.1 * Add Hunyuan3D to readme. * Support more float8 types. * Add CFGZeroStar node. Works on all models that use a negative prompt but is meant for rectified flow models. * Support the WAN 2.1 fun control models. Use the new WanFunControlToVideo node. * Add WanFunInpaintToVideo node for the Wan fun inpaint models. * Update frontend to 1.14.6 (comfyanonymous#7416) Cherry-pick the fix: Comfy-Org/ComfyUI_frontend#3252 * Don't error if wan concat image has extra channels. * ltxv: fix preprocessing exception when compression is 0. (comfyanonymous#7431) * Remove useless code. * Fix latent composite node not working when source has alpha. * Fix alpha channel mismatch on destination in ImageCompositeMasked * Add option to store TE in bf16 (comfyanonymous#7461) * User missing (comfyanonymous#7439) * Ensuring a 401 error is returned when user data is not found in multi-user context. * Returning a 401 error when provided comfy-user does not exists on server side. * Fix comment. This function does not support quads. * MLU memory optimization (comfyanonymous#7470) Co-authored-by: huzhan <huzhan@cambricon.com> * Fix alpha image issue in more nodes. * Fix problem. * Disable partial offloading of audio VAE. * Add activations_shape info in UNet models (comfyanonymous#7482) * Add activations_shape info in UNet models * activations_shape should be a list * Support 512 siglip model. * Show a proper error to the user when a vision model file is invalid. * Support the wan fun reward loras. --------- Co-authored-by: comfyanonymous <comfyanonymous@protonmail.com> Co-authored-by: Chenlei Hu <hcl@comfy.org> Co-authored-by: thot experiment <94414189+thot-experiment@users.noreply.github.com> Co-authored-by: comfyanonymous <121283862+comfyanonymous@users.noreply.github.com> Co-authored-by: Terry Jia <terryjia88@gmail.com> Co-authored-by: Michael Kupchick <michael@lightricks.com> Co-authored-by: BVH <82035780+bvhari@users.noreply.github.com> Co-authored-by: Laurent Erignoux <lerignoux@gmail.com> Co-authored-by: BiologicalExplosion <49753622+BiologicalExplosion@users.noreply.github.com> Co-authored-by: huzhan <huzhan@cambricon.com> Co-authored-by: Raphael Walker <slickytail.mc@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, the TransformerBlocks inside the UNet only receive the flattened activations. This makes it impossible for them to know what the actual spatial dimensions of the activations that were passed to them were (because of downsampling, padding, etc. the aspect ratio of the activations is different in different layers, so it can't even be reliably inferred from the original latent shape).
For example, in Cubiq's IPAdapter implementation, an initial guess is made to the mask size/shape, but in some cases we have to give up and pad the mask with zeros, likely creating incorrect results.
Similarly, in the official comfy code for SAG we actually attempt to factorize the attention map shape, and take whichever "possible shape" is closest to the shape of the original latent.
Clearly, the need to implement a heuristic algorithm that guesses what shape the original activations were makes it needlessly complex to create an attention patch. This PR adds a new key to transfomer_options, which is only set inside of SpatialTransformers and SpatialVideoTransformers, that contains the non-flattened, spatial shape of the activations inside that block. This makes it easy for attention patches to resize their masks correctly, compute attention maps, etc.