Fix additive attention mask handling in the native NPU attention backend. - #14396
Fix additive attention mask handling in the native NPU attention backend.#14396mengchengTang wants to merge 1 commit into
Conversation
e4a82e0 to
e1962a8
Compare
e1962a8 to
c9ba560
Compare
|
@yiyixuxu @sayakpaul @zhtmike These two Ascend NPU fixes (#14288 and #14396) are issues I ran into while adapting verl-omni — could you take a look when you get a chance? Thanks! |
|
It is just my personal opinion, seems it is little bit strange of this fixing, it is supposed to fix mask in npu attention instead of routing to sdpa? And kindly ping @sayakpaul, can you take a look of this PR if you have time? Seems there is some mask handling probelm in NPU attention, and this fixing is limited in NPU scope. Thanks! |
| if attn_mask is not None and attn_mask.is_floating_point(): | ||
| # Preserve additive mask values by letting native SDPA select the appropriate NPU implementation. | ||
| return _native_attention( | ||
| query, | ||
| key, | ||
| value, | ||
| attn_mask, | ||
| dropout_p, | ||
| scale=scale, | ||
| _parallel_config=_parallel_config, | ||
| ) |
There was a problem hiding this comment.
If it's NPU specific then we should add a conditional to handle it. @zhtmike could you confirm if it works as expected on CUDA?
There was a problem hiding this comment.
Hi @sayakpaul, I believe the code modification is within npu scope.
the changes only takes effective only when
transformer.set_attention_backend("_native_npu") -> routing to _native_npu_attention -> fix the mask handling in NPU.
So it should be safe for CUDA.
There was a problem hiding this comment.
But a kind of strange why routing to spda instead fixing the mask directly? @mengchengTang
There was a problem hiding this comment.
@zhtmike The npu_fusion_attention called by _native_npu only supports boolean masks. Routing additive masks to SDPA prevents logic errors from forcibly converting them to boolean masks. SDPA can correctly handle additive masks.
What does this PR do?
Fixes attention mask handling in the native NPU attention backend.
PyTorch SDPA defines two mask formats:
Truemeans attend.Previously,
_native_npu_attentionpassed every mask to_maybe_modify_attn_mask_npu, which casts the mask to boolean and inverts it fornpu_fusion_attention. This works for boolean masks but loses the values and semantics of floating-point additive masks.This PR makes the mask handling explicit:
npu_fusion_attention.No new dependencies are introduced.
Fixes #14397
Test results
dg845/LTX-2.3-Diffuserswith verl-omni NPU traininglog_prob_diffwas around1e-3during the first 5 training stepslog_prob_diffdecreased to around1e-5during the first 5 training stepsBefore submitting
self-reviewskill on the diff?documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.