Skip to content

[docs] Add note for context parallelism: call enable_parallelism on every transformer in the pipeline - #15334

Open
menglcai wants to merge 2 commits into
huggingface:mainfrom
menglcai:docs-cp-multiple-transformers
Open

menglcai wants to merge 2 commits into
huggingface:mainfrom
menglcai:docs-cp-multiple-transformers

Conversation

@menglcai

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes #15317

Adds a short note to the context parallelism guide: enable_parallelism only applies to the model you call it on, so a pipeline with more than one transformer (such as transformer and transformer_2) needs it on each one. Otherwise, the steps the other transformers run aren't split across GPUs.

The note goes right after the Ring Attention example, since every other variant section points back to it. Following @stevhliu's suggestion in the issue, it is general rather than specific to Wan. The Wan 2.2 A14B measurements that motivated it are in the issue.

Self-review notes

Verdict: READY, no blocking issues. Docs-only change, so no dead code and no tests apply. I didn't run make quality locally; CI runs the doc style check.

Left for the reviewer:

  1. The note uses the same one-paragraph > [!NOTE] form as the other notes in docs/source/en/optimization/, and the same doc-builder link to ModelMixin.enable_parallelism as the rest of the page.
  2. "Enable context parallelism while loading" also shows only transformer. The note's rule applies there too, so I didn't add a second note.

Before submitting

  • Did you use an AI agent (Claude Code, Codex, Cursor, etc.) to help with this PR? If so:
    • Did you read the Coding with AI agents guide?
    • Did you run the self-review skill on the diff?
    • Did you share the final self-review notes in the PR description or a comment?
  • Did you read the contributor guideline?
  • Did you read our philosophy doc? (important for complex PRs)
  • Was this discussed/approved via a GitHub issue or GitHub Discussions? Please add a link to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?
  • Are you the author (or part of the team we work with) of the model/pipeline this PR relates to?

Who can review?

@stevhliu @sayakpaul

…lled on

The context parallelism guide only calls `pipeline.transformer.enable_parallelism(...)`. A pipeline with more than one transformer, such as Wan 2.2 A14B with `transformer` and `transformer_2`, needs it on each one, or the steps the other transformers run aren't split across GPUs. Add a short note after the Ring Attention example.

Signed-off-by: menglcai <menglcai@amd.com>
@github-actions github-actions Bot added documentation Improvements or additions to documentation fixes-issue size/S PR with diff < 50 LOC labels Nov 18, 2026
@menglcai
menglcai marked this pull request as ready for review November 18, 2026 07:54

@stevhliu stevhliu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks! maybe move this up to line 17 to put it in front of everyone, and not just readers who are in the Ring Attention section since this applies to every variant on the page

To try the other variants, replace the `enable_parallelism` call in `ring_attention.py` with the one from their section.

> [!NOTE]
> [`~ModelMixin.enable_parallelism`] only applies to the model you call it on. If a pipeline has more than one transformer, such as `transformer` and `transformer_2`, call it on each one. Otherwise, the steps the other transformers run aren't split across GPUs.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
> [`~ModelMixin.enable_parallelism`] only applies to the model you call it on. If a pipeline has more than one transformer, such as `transformer` and `transformer_2`, call it on each one. Otherwise, the steps the other transformers run aren't split across GPUs.
> [`~ModelMixin.enable_parallelism`] only applies to the model you call it on. If a pipeline has more than one transformer, such as `transformer` and `transformer_2`, call it on each one. Otherwise, the denoising steps that use the other transformers aren't split across GPUs.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the review and the suggestions! I've moved the note up to right after the attention backend paragraph so readers see it before any of the variants, and applied your wording. Please let me know if anything else needs adjusting. Thanks again!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation fixes-issue size/S PR with diff < 50 LOC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[docs] Context parallelism guide only parallelizes transformer; Wan 2.2 A14B's transformer_2 stays unsplit

2 participants