Differential Transformer: Noise-Canceling Attention by Subtraction

Every softmax attention head has a dirty secret: it can never assign exactly zero probability to a token. The exponential in softmax is strictly positive, so even the most irrelevant token in a 100K-token context gets a sliver of attention weight. Sum those slivers across tens of thousands of tokens and you get a persistent floor of attention noise — probability mass smeared across context that the model does not actually care about. As context grows, this floor rises, drowning the signal from the handful of tokens that matter.

The Differential Transformer (Ye et al., Microsoft Research, ICLR 2025 Oral) attacks this with an idea lifted straight from analog circuit design. A differential amplifier rejects common-mode noise by measuring the difference between two input lines that share the same interference. Diff Transformer does the same thing to attention: compute two separate softmax maps and subtract one from the other. Whatever noise the two maps share cancels; the genuine signal survives.

The mechanism

Standard attention projects the input X into a single query, key, and value, then applies one softmax:

# Standard single-head attention
Q, K, V = X @ Wq, X @ Wk, X @ Wv        # each [n, d]
attn = softmax(Q @ K.T / sqrt(d)) @ V   # [n, d]

Differential attention splits the query and key projections into two halves each, producing two independent attention maps, then subtracts the second (scaled by a learnable λ) from the first:

import torch, torch.nn.functional as F

def diff_attn(X, Wq, Wk, Wv, lam, d):
    # Wq, Wk project to 2*d so we can split into two query/key groups
    Q = (X @ Wq).view(X.size(0), 2, d)   # [n, 2, d]
    K = (X @ Wk).view(X.size(0), 2, d)
    V = X @ Wv                           # [n, d_v]

    Q1, Q2 = Q[:, 0], Q[:, 1]
    K1, K2 = K[:, 0], K[:, 1]

    A1 = F.softmax(Q1 @ K1.T / d**0.5, dim=-1)
    A2 = F.softmax(Q2 @ K2.T / d**0.5, dim=-1)

    return (A1 - lam * A2) @ V           # noise cancels in (A1 - lam*A2)

The intuition is that A1 and A2, both trained on the same input, learn to place similar amounts of weight on the irrelevant "background" tokens. That background is the common-mode noise. When you compute A1 - λ·A2, the shared background subtracts away, leaving a sparse, high-contrast attention pattern focused on relevant tokens. Crucially, the resulting weights are no longer a probability distribution — they can be negative — which is exactly what lets a token be actively suppressed rather than merely assigned tiny positive mass.

Making λ learnable and stable

A naive fixed λ would be fragile. The paper makes λ a learned scalar per head, but reparameterizes it through dot products of learnable vectors so gradients stay well-behaved:

λ = exp(λ_q1 · λ_k1) − exp(λ_q2 · λ_k2) + λ_init

where λ_q1, λ_k1, λ_q2, λ_k2 are learnable vectors and λ_init is a fixed per-layer constant that sets the starting point of the subtraction. The schedule they use grows the initial cancellation strength with depth:

λ_init(l) = 0.8 − 0.6 · exp(−0.3 · (l − 1))

for layer index l (1-indexed). Early layers start near 0.2 (gentle subtraction, preserving raw signal), deeper layers approach 0.8 (aggressive noise cancellation once representations are refined). In practice the exact schedule is not sensitive — the authors note a constant λ_init = 0.8 works comparably.

Two more details make it train cleanly:

1. Per-head GroupNorm (RMSNorm). Because the subtraction can produce different statistics across heads, each head's output is independently normalized before concatenation. This keeps the scale of differential heads comparable to normal heads.

2. Fixed output rescale. The normalized output is multiplied by (1 − λ_init) to align its gradient magnitude with a standard Transformer, so you can reuse the same learning rate and warmup without retuning.

def diff_attn_head(X, ..., lam, lam_init, d):
    out = (A1 - lam * A2) @ V
    out = group_rms_norm(out)            # per-head normalization
    return out * (1.0 - lam_init)        # match baseline gradient scale

Keeping the parameter count honest

Splitting queries and keys into two groups looks like it doubles the projection cost. The paper keeps FLOPs and parameters matched to the baseline by halving the number of heads while giving each head twice the width. A baseline with h heads of dimension d becomes h/2 differential heads, each using 2d for its query/key groups. Net result: same parameter budget, same arithmetic, so any quality gain is a genuine architectural win rather than a capacity increase.

What the subtraction buys you

The headline result is scaling efficiency. Across model sizes and token budgets, Diff Transformer matches a standard Transformer's language-modeling loss using roughly 65% of the parameters or 65% of the training tokens. The gap widens on tasks where attention noise is most damaging:

• Long-context key retrieval. In needle-in-a-haystack style evaluations, the sparser attention pattern locks onto the relevant span far more reliably as context length and distractor count grow. The model allocates a much larger fraction of its total attention mass to the answer span.

• Hallucination. In question answering and summarization, hallucination often traces back to the model attending to irrelevant context and fabricating from it. By suppressing that context, Diff Transformer measurably reduces hallucinated content.

• In-context learning. Few-shot accuracy improves, and — importantly — the model becomes more robust to permutation of the demonstration examples, a notorious source of variance in standard Transformers. Less noise means order matters less.

• Quantization. This one is the sleeper. Standard Transformers accumulate large activation outliers that force wide dynamic ranges and wreck low-bit quantization. Because differential attention produces less extreme activations, Diff Transformer's activations quantize more gracefully — the paper reports usable 4-bit attention where the baseline degrades sharply. Cleaner activations are a direct downstream consequence of canceling the noise floor.

Why this is more than a trick

It is tempting to file this under "clever reparameterization," but the deeper point is about what softmax fundamentally cannot do. A single softmax can only add attention — it has no mechanism to say "ignore this." The best it can manage is a vanishingly small weight, and small-times-many-tokens is still a meaningful leak. Subtracting two softmaxes creates a signed attention operator: it can push weight below zero and genuinely veto a token. That expressive jump — from non-negative to signed attention weights — is the real innovation, and the differential-amplifier framing is just the cleanest way to make it trainable.

The cost is modest: two attention maps instead of one (though FLOP-matched via fewer, wider heads), plus a handful of scalar parameters per head. For long-context workloads, retrieval-heavy pipelines, or anywhere you plan to quantize aggressively, that is a favorable trade. Sometimes the best way to hear the signal is to stop amplifying it and start subtracting the noise.