Prelude experiments
This is an archival repository of experimental checkpoints for the OpenEuroLLM Prelude model. It is mostly intended for internal use and for transparency.
Micro-annnealing checkpoints
Branches with the names starting with anneal100b and anneal300b are the checkpoints of micro-annealing the base Prelude model (step 953312)
on 100B and 300B tokens correspondingly. The model before micro-annealing was trained on 8T tokens. Data mixtures used for micro-annealing are:
E.g., anneal100b_multilingual2_iter_0957600 is a checkpoint saved at step 957600 while micro-annealing on the multilingual2 data mixture for 100B tokens.
Branches without a data mixture name (anneal300b_iter_0960000, etc) contain checkpoints of the baseline annealing on the same dataset as the main Prelude training run.
See more details on micro-annealing mixtures and their properties here.
As a result of these experiments, the multilingual1_dominant1 mixture was chosen for the final 2T tokens Prelude annealing run.