PyTorch Profiling Series Turns to Attention Kernels and SDPA Backends
Hugging Face
- Research
- Developer tools
- #pytorch
- #sdpa
- #profiling
- #attention
A Hugging Face blog post, the third in a PyTorch profiling series, steps through naive causal attention, an in-place masking variant, and SDPA backends, reading each in the profiler. The authors report the out-of-place masked_fill triggers a Memcpy kernel, that the in-place version removes it, and that the SDPA math backend ran 3.7x slower than their naive in-place attention on an A100.
What the post demonstrates
The authors continue a profiling series by building causal attention from primitives: matmul of queries and transposed keys, scaling, masked_fill, softmax, then a second matmul with values. Reading the trace, they spot an unexpected Memcpy kernel alongside the five expected kernels. They attribute it to out-of-place masking, since PyTorch copies tensors for ordinary operations, and show that switching to masked_fill_ drops the Memcpy from the GPU lane. They note in-place work is safe under torch.no_grad but risky for autograd. Replacing the module with F.scaled_dot_product_attention lets them pin backends through sdpa_kernel; with the math backend they report 20 GPU kernels and 7.239 ms average CUDA time versus 1.955 ms for naive in-place attention. They trace this to kernel names: the naive matmul carries a bfloat16 Tensor Core signature while the math path does not.
Try it in a merge review
A performance engineer reviewing an attention-heavy model could use this as a checklist during a merge request that introduces or changes an attention path. Rather than trusting that a one-line SDPA call is automatically faster, they could capture traces on the target GPU for the old and new code and count kernels per forward step, noting any Memcpy entries or fused-versus-unfused matmul names. This is a plausible use of the authors' method, not a claim that their numbers transfer to other models.
A small, measurable trial
A team could try profiling one attention layer under torch.no_grad in three variants: out-of-place masking, in-place masking, and SDPA with a pinned backend. Success would be a lower per-forward kernel count and CUDA time against the naive baseline, checked in the profiler table. As failure checks, confirm gradients still behave if any variant runs outside no_grad, and verify the pinned backend is actually the one dispatched. A human should review traces before any claim of improvement, since a single-layer result may not hold at full model scale.
Trade-offs and open questions
In-place masking saves a copy but conflicts with autograd, so it is not a blanket recommendation for training loops that need backward passes. Pinning an SDPA backend trades automatic selection for predictability, and the post's backends and results may depend on dtype, head dimension, mask and GPU. The excerpt does not cover the other backends' traces or production-scale effects, so a team should not assume the math backend is slower in general or that these A100 figures apply elsewhere; measure on the intended hardware first.
Context and limitations
- Based on a short excerpt of the post; later sections on other SDPA backends and kernels were not reviewed.
- All figures come from the authors' single A100-SXM4-80GB setup and were not independently reproduced here.
- The slower math backend is one reported measurement, not a general benchmark of SDPA across hardware or dtypes.
- Based on bounded source text, not independent testing. Practical suggestions are editorial proposals.