Writing
2026
Irregular parameter sharing in Transformers
Exisiting literatuer explores instantiations of parameter sharing in a highly regular way. For example, HRM can be interpreted as sharing parameters across depth/time. It is therefore warranted to ask whether irrelular instantiations work as well. The immediate benefit of irregularity is a less restricted modeling capacity of the network, while fully retaining the upside of reducing memory requirements.
Critical flaws in ‘Avoid Catastrophic Forgetting with Rank-1 Fisher from Diffusion Models’
Avoid Catastrophic Forgetting with Rank-1 Fisher from Diffusion Models starts from an attractive problem: Elastic Weight Consolidation (EWC) needs a tractable local metric for old-task sensitivity, but the full Fisher is too large to use directly. The proposed solution is to exploit a near rank-1 empirical Fisher that appears in diffusion models at low SNR. If that direction were an old-task sensitivity direction, rank-1 EWC would replace diagonal EWC with a low-dimensional constraint that still retains cross-parameter structure.
pptrain, a ready-to-use pre-pretraining library
Pre-pretraining is a good idea trapped in an awkward workflow. Before training a language model on natural text, first train it on synthetic tasks that may teach useful structure. Cellular automata, symbolic transformations, mathematical primitives all fit this pattern. One hurdle toward shipping this in production is that most of the work lives in paper-specific code, hidden data generators, and one-off training scripts. Another one is verifying the prepretrainer was instantiated correctly and actually has a positive effect. The package pptrain can help practitioners with that:
Chemokinesis and the Curse of Dimensionality
Chemokinesis is movement whose speed depends on a local chemical signal, without any directional sensing.
Three Underrated Theorems in Probability
Probability theory has a few results that receive most of the attention: the central limit theorem, Bayes’ rule, the law of large numbers. These are foundational and rightly so. But there are other theorems that quietly shape how statisticians and machine learning researchers think about convergence, uncertainty, and estimation, yet rarely appear in standard courses. Here are three that I have found particularly useful.