Critical flaws in `Avoid Catastrophic Forgetting with Rank-1 Fisher from Diffusion Models’

Avoid Catastrophic Forgetting with Rank-1 Fisher from Diffusion Models forms the second moment \(F_{\mathrm E}=\mathbb E[gg^\top ]\) of per-example diffusion-loss gradients and uses a rank-one approximation to \(F_{\mathrm E}\) in Elastic Weight Consolidation (EWC). The proposed justification has three separate problems: the linear proof contains an invalid matrix identity, the leading low-SNR term is independent of the old task, and \(F_{\mathrm E}\) is not generally Fisher information.

What EWC must preserve

EWC adds a local old-task penalty to the new-task loss,

\[ L_B(\theta )+\frac {\lambda }{2}(\theta -\theta _A^\star )^\top F_A(\theta -\theta _A^\star ). \]

In its Bayesian derivation, \(F_A\) approximates the local curvature of the negative log posterior for task \(A\) near \(\theta _A^\star \) ( Kirkpatrick et al.). If \(F_A=\lambda _1vv^\top \), the penalty constrains only the scalar displacement \(v^\top (\theta -\theta _A^\star )\). Interpreting this penalty as old-task preservation therefore requires \(v\) to identify parameter changes to which task \(A\) is sensitive.

The linear proof is incorrect

The paper assumes a linear score model \(s_A(x)=Ax\) with target \(\gamma x\). Even when \(A\) itself is the parameter, the per-example loss and its gradient, up to a positive scalar \(c\), are

\[ \ell (A;x)=\frac {c}{2}\lVert (A-\gamma I)x\rVert ^2, \qquad \nabla _A\ell (A;x)=c(A-\gamma I)xx^\top . \]

The proof replaces the last expression by \(c\lVert x\rVert ^2(A-\gamma I)\), which would require \(xx^\top =\lVert x\rVert ^2I\). This identity is false for \(d>1\). Except in one-dimensional or otherwise degenerate cases, the orientation of \(xx^\top \) varies across examples, so the vectorized gradients need not be collinear. If \(A=A_\theta \), the chain rule instead gives \(\nabla _\theta \ell =cJ_\theta ^\top \operatorname {vec}[(A_\theta -\gamma I)xx^\top ]\), where \(J_\theta =\partial \operatorname {vec}(A_\theta )/\partial \theta \); no input-independent direction follows without an additional restriction on this Jacobian. The rank-one form of \(F_{\mathrm E}\) therefore does not follow even for the stated linear surrogate. Measurements on a nonlinear U-Net may still show a dominant empirical eigenvector, but they do not repair this derivation.

Low SNR removes task identity

For the variance-preserving forward process

\[ x_t=\sqrt {\bar \alpha _t}x_0+\sqrt {1-\bar \alpha _t}\epsilon , \qquad \epsilon \sim \mathcal N(0,I), \]

Tweedie’s identity gives

\[ s_t^*(x_t) =\frac {\sqrt {\bar \alpha _t}}{1-\bar \alpha _t} \mathbb E[x_0\mid x_t] -\frac {x_t}{1-\bar \alpha _t}. \]

For data distributions with finite second moment, the first term vanishes in mean square as \(\bar \alpha _t\to 0\), while the noisy marginal converges to \(\mathcal N(0,I)\). Thus the leading score is \(-x_t/(1-\bar \alpha _t)\), and the corresponding population noise predictor \(\mathbb E[\epsilon \mid x_t]=-\sqrt {1-\bar \alpha _t}\,s_t^*(x_t)\) is also a schedule-dependent scaling of \(x_t\). Neither leading map depends on the old data distribution, so the limiting argument does not establish that its gradient direction measures old-task sensitivity.

The implemented rank-one direction instead averages gradients over both data and sampled timesteps, \(\mathbb E_{x_0,t}[g(\theta ;x_t,t)]\), and the continual-learning objective also includes generative distillation. Finite-SNR gradients and distillation can contain old-task information, but neither is identified by the task-independent limiting argument.

A gradient second moment is not Fisher information

The spectrum of \(F_{\mathrm E}=\mathbb E[\nabla \ell \nabla \ell ^\top ]\) does not in general equal the Fisher-information spectrum of a probabilistic model. For the conditional linear-Gaussian model

\[ x\sim \mathcal N(0,\Sigma _x), \qquad y\mid x,A\sim \mathcal N(Ax,\sigma ^2I_m), \]

the likelihood Fisher for \(A\) is

\[ F_{\mathrm {true}}=\frac {1}{\sigma ^2}\Sigma _x\otimes I_m. \]

If \(x\) is isotropic, this matrix is proportional to the identity rather than rank one. This calculation does not imply that the diffusion-loss gradient moment is useless; it shows that a dominant eigenvector of that moment does not establish a rank-one local KL geometry. Kunstner, Balles, and Hennig analyze the distinction between gradient second moments and Fisher information more generally.

What would establish the mechanism

A theoretical repair must retain the \(xx^\top \) factor or state conditions under which the resulting gradients are approximately collinear. Timestep-restricted EWC matrices, compared under the same replay and distillation procedure, would then show whether old-task preservation comes from low-, intermediate-, or high-SNR gradients. Until that link is established, the penalty should be described as empirically motivated by a gradient second moment, not derived from the stated low-SNR Fisher argument. IEWC illustrates the distinct alternative of deriving an EWC penalty from preservation of earlier optimization updates.