Skip to content

[LoRA] Only swap lora_B in final_layer scale/shift remaps (Flux Control, HunyuanVideo) - #15307

Open
MaiAditya wants to merge 3 commits into
huggingface:mainfrom
MaiAditya:fix/final-layer-lora-a-swap
Open

MaiAditya wants to merge 3 commits into
huggingface:mainfrom
MaiAditya:fix/final-layer-lora-a-swap

Conversation

@MaiAditya

Copy link
Copy Markdown

What does this PR do?

Fixes #15305

_convert_bfl_flux_control_lora_to_diffusers and the HunyuanVideo original-format converter swapped the scale/shift halves of both lora_A and lora_B for final_layer.adaLN_modulation.1. The halves are the layer's output rows, which belong to lora_B only; lora_A's rows are rank components, so swapping them changes the product B @ A instead of reordering it. Both converters now swap only lora_B (and the Flux Control lora_B.bias), the same as the kohya Flux converter and #15276.

With the reproducer from the issue, the converted delta now matches the expected (scale, shift) result exactly.

Tests: one per converter, modelled on #15276's test_lora_bfl_final_layer_modulation. Each builds a LoRA with distinct lora_A halves, converts it, and checks that lora_A is unchanged and that B @ A equals the original delta with its output halves swapped. Both fail on main and pass with the fix.

Self-review notes

Ran the self-review skill on the diff (rubric: review-rules.md, code_style.md, testing.md, pitfalls.md). Verdict: READY, no blocking issues.

  • pytest tests/models/transformers/test_models_transformer_flux.py tests/models/transformers/test_models_transformer_hunyuan_video.py -k "final_layer_modulation or lora": 32 passed, 13 failed, 14 skipped. The 13 failures also happen on main on macOS (4 test_lora_layerwise_casting_*, 9 Flux test_hotswapping_compile*), the same set Swap shift and scale in Flux2 LoRAs of final_layer.adaLN_modulation.1 #15276 saw.
  • utils/check_copies.py: clean, no # Copied from points at either converter. ruff check / ruff format --check: pass.
  • Dead code: none (the identity maybe_swap is used for lora_A; the bias branch is used by Flux Control LoRAs that ship lora_B.bias).

Left for review:

  1. HunyuanVideo matches "lora_B" in key. The old code swapped every final_layer.adaLN_modulation.1.* key, so a full (non-LoRA) .weight/.bias under that prefix would no longer be swapped. "lora_A" not in key would keep the old behaviour for those; I don't know whether such checkpoints exist, so I can switch if you prefer.
  2. The Flux Control test builds the full 19 double + 38 single block key set with zero tensors, because the converter raises on leftover keys. It runs in under a second; happy to trim it if there's a smaller fixture pattern you'd like.

Before submitting

Who can review?

@sayakpaul

…ol, HunyuanVideo)

The original final layer orders its modulation (shift, scale) and diffusers' norm_out
(scale, shift). The Flux Control and HunyuanVideo converters swapped the halves of
lora_A as well as lora_B, which scrambles lora_A's rank dimension and gives a wrong
delta. Swap only lora_B, as the kohya Flux converter already does.
def test_lora_bfl_control_final_layer_modulation(self):
# The original final layer orders its modulation (shift, scale) and diffusers' `norm_out` (scale, shift). Only
# lora_B's rows are the layer's outputs, so the converted update must be the original one with its halves swapped.
rank, inner_dim, mlp_dim = 1, 3072, 4 * 3072

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can reduce the inner_dim and mlp_dim.

Comment on lines +321 to +325
for i in range(19):
for stream in ("img", "txt"):
modules += [f"double_blocks.{i}.{stream}_{name}" for name in ("mod.lin", "attn.qkv", "attn.proj")]
modules += [f"double_blocks.{i}.{stream}_mlp.{j}" for j in (0, 2)]
for i in range(38):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to iterate over these many layers.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MaiAditya we don't need to iterate over 19 and 38 layers right?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, not anymore. In 3f34c58 _convert_bfl_flux_control_lora_to_diffusers takes the number of double and single blocks from the state dict instead of hard-coding 19/38, the same way _convert_non_diffusers_flux2_lora_to_diffusers and _convert_kohya_flux2_lora_to_diffusers do. The test now calls the converter directly with only the non-block layers plus the final layer, so there's no loop at all. A full 19/38-block state dict still converts as before (checked with the previous fixture), and the test still fails without the lora_A fix.

Comment on lines +341 to +344
delta = (
converted["transformer.norm_out.linear.lora_B.weight"]
@ converted["transformer.norm_out.linear.lora_A.weight"]
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're checking against just norm_out, then why construct this big a state_dict?

@MaiAditya

Copy link
Copy Markdown
Author

Thanks @sayakpaul! Pushed 30ebd63:

  • The LoRA tensors are now minimal: lora_A is 1×1 and lora_B 3×1 everywhere, except single_blocks.*.linear1, which the converter splits into q/k/v/mlp and so keeps its full 21504-row output, as an expanded view of a single element. That takes the fixture from ~24 MB to a few KB.
  • The number of blocks has to stay: _convert_bfl_flux_control_lora_to_diffusers hard-codes 19 double / 38 single blocks and inner_dim=3072, pops every block's keys unconditionally and raises if anything is left over, so a state dict with only the final layer fails with a KeyError before reaching norm_out.

If you'd prefer, I can make the converter infer the block counts from the keys in a follow-up, which would let this test use a single block.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flux Control and HunyuanVideo LoRA converters swap the scale/shift halves of lora_A in final_layer

2 participants