Davide Wiest

Davide Wiest

Computer Science Student aiming to pursue a career in Machine Learning

Posts by Tags

continual-learning

Critical flaws in ‘Avoid Catastrophic Forgetting with Rank-1 Fisher from Diffusion Models’

9 minute read

Published:

Avoid Catastrophic Forgetting with Rank-1 Fisher from Diffusion Models starts from an attractive problem: Elastic Weight Consolidation (EWC) needs a tractable local metric for old-task sensitivity, but the full Fisher is too large to use directly. The proposed solution is to exploit a near rank-1 empirical Fisher that appears in diffusion models at low SNR. If that direction were an old-task sensitivity direction, rank-1 EWC would replace diagonal EWC with a low-dimensional constraint that still retains cross-parameter structure.

diffusion-models

Critical flaws in ‘Avoid Catastrophic Forgetting with Rank-1 Fisher from Diffusion Models’

9 minute read

Published:

Avoid Catastrophic Forgetting with Rank-1 Fisher from Diffusion Models starts from an attractive problem: Elastic Weight Consolidation (EWC) needs a tractable local metric for old-task sensitivity, but the full Fisher is too large to use directly. The proposed solution is to exploit a near rank-1 empirical Fisher that appears in diffusion models at low SNR. If that direction were an old-task sensitivity direction, rank-1 EWC would replace diagonal EWC with a low-dimensional constraint that still retains cross-parameter structure.

experiments

fisher-information

Critical flaws in ‘Avoid Catastrophic Forgetting with Rank-1 Fisher from Diffusion Models’

9 minute read

Published:

Avoid Catastrophic Forgetting with Rank-1 Fisher from Diffusion Models starts from an attractive problem: Elastic Weight Consolidation (EWC) needs a tractable local metric for old-task sensitivity, but the full Fisher is too large to use directly. The proposed solution is to exploit a near rank-1 empirical Fisher that appears in diffusion models at low SNR. If that direction were an old-task sensitivity direction, rank-1 EWC would replace diagonal EWC with a low-dimensional constraint that still retains cross-parameter structure.

geometry

high-dimensional-statistics

intrinsic-dimension

language-models

Irregular parameter sharing in Transformers

5 minute read

Published:

Exisiting literatuer explores instantiations of parameter sharing in a highly regular way. For example, HRM can be interpreted as sharing parameters across depth/time. It is therefore warranted to ask whether irrelular instantiations work as well. The immediate benefit of irregularity is a less restricted modeling capacity of the network, while fully retaining the upside of reducing memory requirements.

pptrain, a ready-to-use pre-pretraining library

2 minute read

Published:

Pre-pretraining is a good idea trapped in an awkward workflow. Before training a language model on natural text, first train it on synthetic tasks that may teach useful structure. Cellular automata, symbolic transformations, mathematical primitives all fit this pattern. One hurdle toward shipping this in production is that most of the work lives in paper-specific code, hidden data generators, and one-off training scripts. Another one is verifying the prepretrainer was instantiated correctly and actually has a positive effect. The package pptrain can help practitioners with that:

machine-learning

Irregular parameter sharing in Transformers

5 minute read

Published:

Exisiting literatuer explores instantiations of parameter sharing in a highly regular way. For example, HRM can be interpreted as sharing parameters across depth/time. It is therefore warranted to ask whether irrelular instantiations work as well. The immediate benefit of irregularity is a less restricted modeling capacity of the network, while fully retaining the upside of reducing memory requirements.

Critical flaws in ‘Avoid Catastrophic Forgetting with Rank-1 Fisher from Diffusion Models’

9 minute read

Published:

Avoid Catastrophic Forgetting with Rank-1 Fisher from Diffusion Models starts from an attractive problem: Elastic Weight Consolidation (EWC) needs a tractable local metric for old-task sensitivity, but the full Fisher is too large to use directly. The proposed solution is to exploit a near rank-1 empirical Fisher that appears in diffusion models at low SNR. If that direction were an old-task sensitivity direction, rank-1 EWC would replace diagonal EWC with a low-dimensional constraint that still retains cross-parameter structure.

pptrain, a ready-to-use pre-pretraining library

2 minute read

Published:

Pre-pretraining is a good idea trapped in an awkward workflow. Before training a language model on natural text, first train it on synthetic tasks that may teach useful structure. Cellular automata, symbolic transformations, mathematical primitives all fit this pattern. One hurdle toward shipping this in production is that most of the work lives in paper-specific code, hidden data generators, and one-off training scripts. Another one is verifying the prepretrainer was instantiated correctly and actually has a positive effect. The package pptrain can help practitioners with that:

Three Underrated Theorems in Probability

3 minute read

Published:

Probability theory has a few results that receive most of the attention: the central limit theorem, Bayes’ rule, the law of large numbers. These are foundational and rightly so. But there are other theorems that quietly shape how statisticians and machine learning researchers think about convergence, uncertainty, and estimation, yet rarely appear in standard courses. Here are three that I have found particularly useful.

natural-language-processing

optimization

parameter-sharing

Irregular parameter sharing in Transformers

5 minute read

Published:

Exisiting literatuer explores instantiations of parameter sharing in a highly regular way. For example, HRM can be interpreted as sharing parameters across depth/time. It is therefore warranted to ask whether irrelular instantiations work as well. The immediate benefit of irregularity is a less restricted modeling capacity of the network, while fully retaining the upside of reducing memory requirements.

pretraining

pptrain, a ready-to-use pre-pretraining library

2 minute read

Published:

Pre-pretraining is a good idea trapped in an awkward workflow. Before training a language model on natural text, first train it on synthetic tasks that may teach useful structure. Cellular automata, symbolic transformations, mathematical primitives all fit this pattern. One hurdle toward shipping this in production is that most of the work lives in paper-specific code, hidden data generators, and one-off training scripts. Another one is verifying the prepretrainer was instantiated correctly and actually has a positive effect. The package pptrain can help practitioners with that:

probability

Three Underrated Theorems in Probability

3 minute read

Published:

Probability theory has a few results that receive most of the attention: the central limit theorem, Bayes’ rule, the law of large numbers. These are foundational and rightly so. But there are other theorems that quietly shape how statisticians and machine learning researchers think about convergence, uncertainty, and estimation, yet rarely appear in standard courses. Here are three that I have found particularly useful.

software

pptrain, a ready-to-use pre-pretraining library

2 minute read

Published:

Pre-pretraining is a good idea trapped in an awkward workflow. Before training a language model on natural text, first train it on synthetic tasks that may teach useful structure. Cellular automata, symbolic transformations, mathematical primitives all fit this pattern. One hurdle toward shipping this in production is that most of the work lives in paper-specific code, hidden data generators, and one-off training scripts. Another one is verifying the prepretrainer was instantiated correctly and actually has a positive effect. The package pptrain can help practitioners with that:

statistics

Three Underrated Theorems in Probability

3 minute read

Published:

Probability theory has a few results that receive most of the attention: the central limit theorem, Bayes’ rule, the law of large numbers. These are foundational and rightly so. But there are other theorems that quietly shape how statisticians and machine learning researchers think about convergence, uncertainty, and estimation, yet rarely appear in standard courses. Here are three that I have found particularly useful.

transformers

Irregular parameter sharing in Transformers

5 minute read

Published:

Exisiting literatuer explores instantiations of parameter sharing in a highly regular way. For example, HRM can be interpreted as sharing parameters across depth/time. It is therefore warranted to ask whether irrelular instantiations work as well. The immediate benefit of irregularity is a less restricted modeling capacity of the network, while fully retaining the upside of reducing memory requirements.