pptrain: a common interface for prepretraining experiments

Prepretraining initializes a language model on synthetic or abstract sequences before continuing training on natural text. pptrain exposes paper-derived task generators through common interfaces for upstream training, weight transfer, and evaluation.

Run a task

pip install pptrain

from pptrain import PrePreTrainer, RunConfig, create_task
from pptrain.integrations import HFCausalLMAdapter, HFModelConfig

trainer = PrePreTrainer(
    task=create_task("simpler_tasks", {
        "preset": "paper_binary_1m",
        "sequence_count": 256,
        "eval_sequence_count": 64,
        "max_length": 128,
    }),
    model_adapter=HFCausalLMAdapter(
        HFModelConfig(model_name_or_path="sshleifer/tiny-gpt2")
    ),
    run_config=RunConfig(output_dir="runs/initial", max_steps=20),
)

bundle = trainer.fit().load_transfer_bundle()

The built-in families generate neural cellular automata rollouts (nca), balanced-bracket sequences (dyck), short programs (procedural), copy, set, and query transformations (simpler_tasks), induction, deduction, and abduction problems (lime), and synthetic document transformations (summarization). Paper-derived presets expose configurable generation and dataset fields.

Verify and evaluate

Reference-parity tests compare normalized examples or token sequences with fixtures produced by the original LIME, synthetic-summarization, procedural-pretraining, and NCA implementations. The replication interface compares downstream training after synthetic initialization with both random initialization and a natural-text warmup, then exports the resulting metrics, plots, tables, and reports.

A custom symbolic generator implements SymbolicTaskFamily by defining how an example is sampled, executed, serialized, and tokenized. A method that requires a different dataset construction or loss implements Task directly.