OpenEuroLLM logo

Prelude experiments

This is an archival repository of experimental checkpoints for the OpenEuroLLM Prelude model. It is mostly intended for internal use and for transparency.

Micro-annnealing checkpoints

Branches with the names starting with anneal100b and anneal300b are the checkpoints of micro-annealing the base Prelude model (step 953312) on 100B and 300B tokens correspondingly. The model before micro-annealing was trained on 8T tokens. Data mixtures used for micro-annealing are:

E.g., anneal100b_multilingual2_iter_0957600 is a checkpoint saved at step 957600 while micro-annealing on the multilingual2 data mixture for 100B tokens. Branches without a data mixture name (anneal300b_iter_0960000, etc) contain checkpoints of the baseline annealing on the same dataset as the main Prelude training run.

See more details on micro-annealing mixtures and their properties here.

As a result of these experiments, the multilingual1_dominant1 mixture was chosen for the final 2T tokens Prelude annealing run.

Micro-annealign experiments

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support