Repository navigation
Conversation
…ol, HunyuanVideo) The original final layer orders its modulation (shift, scale) and diffusers' norm_out (scale, shift). The Flux Control and HunyuanVideo converters swapped the halves of lora_A as well as lora_B, which scrambles lora_A's rank dimension and gives a wrong delta. Swap only lora_B, as the kohya Flux converter already does.
| def test_lora_bfl_control_final_layer_modulation(self): | ||
| # The original final layer orders its modulation (shift, scale) and diffusers' `norm_out` (scale, shift). Only | ||
| # lora_B's rows are the layer's outputs, so the converted update must be the original one with its halves swapped. | ||
| rank, inner_dim, mlp_dim = 1, 3072, 4 * 3072 |
There was a problem hiding this comment.
We can reduce the inner_dim and mlp_dim.
| for i in range(19): | ||
| for stream in ("img", "txt"): | ||
| modules += [f"double_blocks.{i}.{stream}_{name}" for name in ("mod.lin", "attn.qkv", "attn.proj")] | ||
| modules += [f"double_blocks.{i}.{stream}_mlp.{j}" for j in (0, 2)] | ||
| for i in range(38): |
There was a problem hiding this comment.
No need to iterate over these many layers.
There was a problem hiding this comment.
@MaiAditya we don't need to iterate over 19 and 38 layers right?
There was a problem hiding this comment.
Right, not anymore. In 3f34c58 _convert_bfl_flux_control_lora_to_diffusers takes the number of double and single blocks from the state dict instead of hard-coding 19/38, the same way _convert_non_diffusers_flux2_lora_to_diffusers and _convert_kohya_flux2_lora_to_diffusers do. The test now calls the converter directly with only the non-block layers plus the final layer, so there's no loop at all. A full 19/38-block state dict still converts as before (checked with the previous fixture), and the test still fails without the lora_A fix.
| delta = ( | ||
| converted["transformer.norm_out.linear.lora_B.weight"] | ||
| @ converted["transformer.norm_out.linear.lora_A.weight"] | ||
| ) |
There was a problem hiding this comment.
If we're checking against just norm_out, then why construct this big a state_dict?
|
Thanks @sayakpaul! Pushed 30ebd63:
If you'd prefer, I can make the converter infer the block counts from the keys in a follow-up, which would let this test use a single block. |
What does this PR do?
Fixes #15305
_convert_bfl_flux_control_lora_to_diffusersand the HunyuanVideo original-format converter swapped the scale/shift halves of bothlora_Aandlora_Bforfinal_layer.adaLN_modulation.1. The halves are the layer's output rows, which belong tolora_Bonly;lora_A's rows are rank components, so swapping them changes the productB @ Ainstead of reordering it. Both converters now swap onlylora_B(and the Flux Controllora_B.bias), the same as the kohya Flux converter and #15276.With the reproducer from the issue, the converted delta now matches the expected (scale, shift) result exactly.
Tests: one per converter, modelled on #15276's
test_lora_bfl_final_layer_modulation. Each builds a LoRA with distinctlora_Ahalves, converts it, and checks thatlora_Ais unchanged and thatB @ Aequals the original delta with its output halves swapped. Both fail on main and pass with the fix.Self-review notes
Ran the
self-reviewskill on the diff (rubric:review-rules.md,code_style.md,testing.md,pitfalls.md). Verdict: READY, no blocking issues.pytest tests/models/transformers/test_models_transformer_flux.py tests/models/transformers/test_models_transformer_hunyuan_video.py -k "final_layer_modulation or lora": 32 passed, 13 failed, 14 skipped. The 13 failures also happen on main on macOS (4test_lora_layerwise_casting_*, 9 Fluxtest_hotswapping_compile*), the same set Swap shift and scale in Flux2 LoRAs of final_layer.adaLN_modulation.1 #15276 saw.utils/check_copies.py: clean, no# Copied frompoints at either converter.ruff check/ruff format --check: pass.maybe_swapis used forlora_A; the bias branch is used by Flux Control LoRAs that shiplora_B.bias).Left for review:
"lora_B" in key. The old code swapped everyfinal_layer.adaLN_modulation.1.*key, so a full (non-LoRA).weight/.biasunder that prefix would no longer be swapped."lora_A" not in keywould keep the old behaviour for those; I don't know whether such checkpoints exist, so I can switch if you prefer.Before submitting
self-reviewskill on the diff?lora_Ainfinal_layer#15305)Who can review?
@sayakpaul