Prepretraining tested: it doesn’t work

Prepretraining denotes a two-stage training regime: a language-model architecture is first trained on synthetic or abstract data, and only then trained on natural language. This setting is relevant because language-model training is increasingly constrained by data quality, compute allocation, and the finite supply of public human-written text. In that context, scaling-law work established the empirical role of model, data, and compute scale, compute-optimal training sharpened the importance of token budgets, and data-supply analyses motivate methods that improve sample efficiency (Kaplan et al. 2020, Hoffmann et al. 2022, Villalobos et al. 2022).

The central empirical question is whether synthetic upstream training gives the later natural-text phase a better initialization. Recent work instantiates this question with NCA-generated sequences, artificial languages, procedural data, simpler synthetic tasks, LIME-style reasoning tasks, and synthetic summarization tasks (Lee et al. 2026, Ri and Tsuruoka 2022, Jiang et al. 2026, Wu et al. 2022, Wu et al. 2021, Krishna et al. 2021). Across these variants, the transfer claim is stronger than cheap data generation: synthetic data must induce structure that remains useful after the transition to natural text.

This evaluation tests that claim with pptrain, using the Pythia-410M architecture and tokenizer, a 12,288-token context length, three seeds, public Hugging Face text datasets, and six synthetic task families. To keep the comparison explicit, the baseline is random initialization followed by downstream text training, while the natural-text warmup is treated as a separate method, called NLP prepretraining, rather than as the baseline.

Under this protocol, the results do not support a general synthetic-prepretraining benefit. Synthetic prepretraining improves over random initialization in only one of six task families, and that case is small: Simpler tasks improves final downstream eval loss by 0.16%. At the same time, NLP prepretraining outperforms every synthetic method, and the reasoning and algorithmic probes show no transfer gain.

Experimental setup

The benchmark was executed with pptrain replicate; the completed run corresponds to:

pptrain replicate \
  --profile paper_proxy_2048 \
  --model-name-or-path EleutherAI/pythia-410m-deduped \
  --context-length 12288 \
  --output-dir internal/runs/paper_proxy_pythia410m_12k_20260703_1552 \
  --resume

Although the profile is named paper_proxy_2048, the run analyzed here overrides the context length to 12,288 tokens. Because the model is initialized from the Pythia config and tokenizer rather than from the released Pythia weights, the Pythia suite supplies the architectural and tokenizer reference, while the training baseline remains random initialization (Biderman et al. 2023).

Each task used three conditions per seed:

1. Random-init baseline: train only on the downstream text slice.
2. Synthetic transfer: train on a synthetic task, transfer the learned weights into the downstream model, then train on the downstream text slice.
3. NLP prepretraining: spend the upstream budget on natural text from the same text family, then continue on the downstream text slice.

The public text data were loaded through Hugging Face Datasets (Lhoest et al. 2021). General-text tasks used HuggingFaceFW/fineweb-edu, sample-10BT, which is part of FineWeb-Edu (Penedo et al. 2024, dataset card); the math task used HuggingFaceTB/finemath, finemath-4plus (dataset card); and the summarization task used vblagoje/cc_news (dataset card). In all three cases, the datasets were streamed with separate skip offsets for warmup, train, and eval slices.

The six synthetic task families were NCA, LIME, Simpler tasks, Procedural, Dyck, and Summarization. Since the protocol uses public datasets and a shared bounded setup, it is a proxy study rather than an exact reproduction of every original corpus, schedule, or training budget. The scope is therefore to test whether the same synthetic methods improve later NLP training under one shared, public, multi-seed protocol.

Results

Taken together, Figure 1 and Table 1 summarize the final outcomes. NCA, LIME, Procedural, Dyck, and Summarization did not beat random initialization, while all six synthetic methods underperformed NLP prepretraining. The auxiliary probes agree with the loss results: reasoning accuracy for NCA and LIME was zero for both scratch and transferred models, and the algorithmic probes for Procedural and Dyck showed no gain.

Table 1. Main downstream losses and relative gaps. Lower loss and lower perplexity are better. Positive gap values mean synthetic transfer improved over the comparator.

The loss curves show the same ordering over training rather than only at the final step. In most panels, NLP prepretraining ends with the lowest loss, the random-init baseline remains between NLP prepretraining and synthetic transfer, and synthetic transfer starts at higher loss and remains higher.

NCA also failed at the upstream task itself: held-out synthetic next-patch token accuracy was essentially zero, averaging 0.0024% across seeds. Because the implementation had reference-parity checks against paper code fixtures, this result is informative about the training mechanism rather than just the generator implementation. In this run, a validated task generator did not produce a useful upstream learning problem for the model, so the failure cannot be reduced to a downstream evaluation artifact.

Diagnostics

Although the representation diagnostics are descriptive rather than primary success criteria, they match the outcome metrics. Relative to scratch, synthetic transfer has lower CKA to the NLP-prepretrained model. Simpler tasks is the mildest case, combining high activation CKA to NLP prepretraining with the only positive scratch gap, whereas NCA shows the opposite pattern: low CKA, low effective rank, high divergence, and the worst direct upstream synthetic accuracy.

Interpretation

The measured result is bounded but consistent. Under this setup, synthetic prepretraining worsens downstream NLP training in five of six task families. The two controls locate the failure: random initialization tests whether synthetic transfer helps at all, while NLP prepretraining tests whether the same upstream budget is better spent on natural text. Synthetic transfer fails the first comparison in five of six tasks and the second comparison in all six.

A possible mechanism is stiffening plus covariate shift. If synthetic prepretraining moves the network into a less plastic state before the transition to natural text, and if the synthetic and downstream distributions are far apart, then the second stage becomes a difficult warm-start problem rather than a useful initialization. This interpretation is consistent with evidence that warm-starting a trained network can generalize worse than retraining from scratch, that deep networks can lose plasticity across task sequences, and that fine-tuning under distribution shift can underperform even when the pretrained representation is good (Ash and Adams 2020, Dohare et al. 2023, Kumar et al. 2022). In this run, five of the six synthetic tasks produced a worse starting point than random initialization.

The NCA condition illustrates this failure mode because its intended mechanism is that rich non-linguistic dynamics provide useful structure before natural text. Here, however, the NCA-prepretrained model did not learn the held-out synthetic prediction task and then transferred poorly to natural text. For this condition, the upstream curriculum failed before the downstream transfer question became relevant.

Conclusion

Under the tested Pythia-410M, 12k-context, public-dataset protocol, synthetic prepretraining does not provide a useful initialization for downstream NLP training: NLP prepretraining achieves lower final evaluation loss than every synthetic method, and random initialization achieves lower final evaluation loss than five of the six synthetic methods.

The evaluation also identifies the necessary controls for future prepretraining claims. A synthetic upstream method should beat random initialization, beat a natural-text upstream budget, and show that the synthetic task was learned; without those checks, a synthetic curriculum can move the model into a less useful region of parameter space while still appearing plausible as a pretraining method.

Appendix

References

1. Jordan T. Ash and Ryan P. Adams. On Warm-Starting Neural Network Training. NeurIPS 2020.
2. Stella Biderman et al. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. ICML 2023.
3. Shibhansh Dohare et al. Maintaining Plasticity in Deep Continual Learning. 2023.
4. Jordan Hoffmann et al. Training Compute-Optimal Large Language Models. NeurIPS 2022.
5. Liangze Jiang et al. Procedural Pretraining: Warming Up Language Models with Abstract Data. ICML 2026.
6. Jared Kaplan et al. Scaling Laws for Neural Language Models. 2020.
7. Kalpesh Krishna et al. Does Pretraining for Summarization Require Knowledge Transfer?. Findings of EMNLP 2021.
8. Ananya Kumar et al. Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution. ICLR 2022.
9. Dan Lee et al. Training Language Models via Neural Cellular Automata. 2026.
10. Quentin Lhoest et al. Datasets: A Community Library for Natural Language Processing. EMNLP Demo 2021.
11. Alexander Mordvintsev et al. Growing Neural Cellular Automata. Distill 2020.
12. Guilherme Penedo et al. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. 2024.
13. Ryokan Ri and Yoshimasa Tsuruoka. Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models. ACL 2022.
14. Pablo Villalobos et al. Will we run out of data? Limits of LLM scaling based on human-generated data. 2022.
15. Yuhuai Wu et al. LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning. ICML 2021.
16. Yuhuai Wu, Felix Li, and Percy Liang. Insights into Pre-training via Simpler Synthetic Tasks. 2022.
17. Hugging Face. HuggingFaceFW/fineweb-edu dataset card.
18. Hugging Face. HuggingFaceTB/finemath dataset card.
19. Vladimir Blagojevic. vblagoje/cc_news dataset card.