Sitemap
A list of all the posts and pages found on the site. For you robots out there, there is an XML version available for digesting as well.
Pages
Posts
Prepretraining tested: it doesn’t work
Published:
Prepretraining denotes a two-stage training regime: a language-model architecture is first trained on synthetic or abstract data, and only then trained on natural language. This setting is relevant because language-model training is increasingly constrained by data quality, compute allocation, and the finite supply of public human-written text. In that context, scaling-law work established the empirical role of model, data, and compute scale, compute-optimal training sharpened the importance of token budgets, and data-supply analyses motivate methods that improve sample efficiency (Kaplan et al. 2020, Hoffmann et al. 2022, Villalobos et al. 2022).
Delay Flow Matching learns the coupling
Published:
Summary
Recursive Union-of-Manifolds Diagnostics Beyond Images
Published:
Abstract
Parameter sharing works best across width
Published:
Abstract
Critical flaws in ‘Avoid Catastrophic Forgetting with Rank-1 Fisher from Diffusion Models’
Published:
Avoid Catastrophic Forgetting with Rank-1 Fisher from Diffusion Models starts from an attractive problem: Elastic Weight Consolidation (EWC) needs a tractable local metric for old-task sensitivity, but the full Fisher is too large to use directly. The proposed solution is to exploit a near rank-1 empirical Fisher that appears in diffusion models at low SNR. If that direction were an old-task sensitivity direction, rank-1 EWC would replace diagonal EWC with a low-dimensional constraint that still retains cross-parameter structure.
pptrain, a library for prepretraining
Published:
Pre-pretraining is a good idea trapped in an awkward workflow. Before training a language model on natural text, first train it on synthetic tasks that may teach useful structure. Cellular automata, symbolic transformations, mathematical primitives all fit this pattern. One hurdle toward shipping this in production is that most of the work lives in paper-specific code, hidden data generators, and one-off training scripts. Another one is verifying the prepretrainer was instantiated correctly and actually has a positive effect. The package pptrain can help practitioners with that:
Chemokinesis and the Curse of Dimensionality
Published:
Chemokinesis is movement whose speed depends on a local chemical signal, without any directional sensing.
Three Underrated Theorems in Probability
Published:
Probability theory has a few results that receive most of the attention: the central limit theorem, Bayes’ rule, the law of large numbers. These are foundational and rightly so. But there are other theorems that quietly shape how statisticians and machine learning researchers think about convergence, uncertainty, and estimation, yet rarely appear in standard courses. Here are three that I have found particularly useful.
portfolio
Clac Permalink
Published:
A series of programming language projects exploring pattern matching, macro expansion, and interpreter design
ContextFlow Permalink
Published:
A .NET library for building structured LLM interactions through dependency injection and fluent interfaces
Embyte Permalink
Published:
A customizable embed generator for websites, inspired by Discord’s rich embeds
Finsights Permalink
Published:
A newsletter service that sends automated performance updates for selected stocks
GPTVault Permalink
Published:
A recursive knowledge accumulation system using large language models to build structured concept graphs
Instadata Permalink
Published:
A scalable Instagram scraper built with Django for data science pipelines
LetoReader Permalink
Published:
A minimalistic, highly customizable speed reader built as an open-source alternative to paid tools
PV Growth Forecasting Permalink
Published:
Multivariate time series forecasting of photovoltaic electricity production in Germany
Soccer Match Prediction Permalink
Published:
A comparison of goal-count regression versus win/loss/draw classification for football match forecasting
Temporal Regularized Learning Permalink
Published:
A highly local, self-supervised learning procedure that optimizes each neuron individually
TimeWise Permalink
Published:
A workforce scheduler using integer linear programming to generate optimal shift assignments
Gradient Routing for Continual Learning Permalink
Published:
Evaluating sparsity and gradient-reweighted updates as a mechanism for continual learning
Closed-Form Initialization Permalink
Published:
An analytical approach to neural network initialization using covariance and eigendecomposition
pptrain Permalink
Published:
A PyTorch-native library for pre-pretraining language models on synthetic tasks
publications
talks
teaching
Teaching experience 1
Undergraduate course, University 1, Department, 2014
This is a description of a teaching experience. You can use markdown like any other post.
Teaching experience 2
Workshop, University 1, Department, 2015
This is a description of a teaching experience. You can use markdown like any other post.