Dsg2/LS3-Micro-Base

This repo contains the Base (completed pretrain cudalm checkpoint) variant of LS3-Micro. The FP32 checkpoint and optimiser is provided, and can be used by tinylm chat.

• Model size: 30.3MB (FP32)
• Environment variables for cudalm: 381B
• AdamW optimiser: 30.0MB
• Muon optimiser: 30.0MB
• Ngrams optimiser: 7.22MB
• Sidecar: 40.1KB

LS3-Micro: The smallest model with the highest intelligence density in the LS family.

(acc norm except LAMBADA)	HellaSwag	ARC-E	ARC-C	PIQA	LAMBADA	ArithMark	*II	**Intelligence Density
LS2.5-108M-A17M-Base	28.24	36.66	23.21	57.40	17.89	37.00	9.89	0.231
LS-Tiny-Base (31M)	27.11	32.24	24.49	53.81	2.99	34.00	6.22	0.288
LS-Micro-Base (11.49M)	27.41	34.72	21.93	54.84	11.06	35.50	7.24	0.909
LS3-Micro-Base (9.97M)	27.31	34.76	23.29	57.34	12.87	38.90	9.64	1.290

LS3-Micro-Base has a 24.9% higher Intelligence Index than LS-Micro Base, while having 13% less total parameters. It also nearly matches LS2.5-108M-A17M with almost 11x less parameters.

• Intelligence Index was calculated using the formula from Open SLM Leaderboard. (Results reproducable via Cudalm's built in bench tool, see bottom quickstart)
• Compute density derived using the following formula: Intelligence Index / sqrt(Active Params (millions) * Total Params (millions))

---

LS3 Experiments on narrow but deeper model architectures for SLMs. For a sub 10M model, the sweet spot found was 11 layers with a dimension of 160. To maximise efficiency while training, LS3 is a not a dense model, but rather a sparse MoE model with ngrams. With a total of 9.97M parameters, ~2.11M of parameters are ngrams. The remaining ~7.87M of parameters make up the MoE backbone. The model has 2 experts and only activates one per token, leading to only 5.60M active per token.

Model specs
Total parameters:	9.97M
Backbone:	7.87M
Active:	5.60M
Ngrams:	2.11M
Context length:	2048
Pretrain details
Tokens seen:	~5.7B
GPU time (5060ti):	8h

*Note that it was trained with a 16k vocab, so 2.62M of the 5.52M active parameters are embedding.

Chatbench results:

Multiple choice:

Model	acc	accnorm	chance
LS-Tiny-Base	40.7%	47.5%	25.0%
LS-Micro-Base	46.6%	52.5%	25.0%
LS3-Micro-Base	45.8%	55.1%	25.0%

Example:

<|user|>
What is your opinion on Strawberries?
<|assistant|>
A strawberry, which is a type of paprika, or strawberry, is a type of meat-based food that is commonly used in culinary arts. Strawberry is a type of lion's food, also known as lime juice, which is a staple in the food. This type of strawberry is not only a vegetable but also a sweet and hearty treat.

The idea is that strawberry is a type of meat-based food, which is a type of lion's food, like paprika or...

To try LS3-Micro or to train/finetune the model, follow this quickstart guide:

1. Clone the latest version of Cudalm here (only windows supported currently, optimised for Blackwell 2.0/CPU).
2. Build with build.py/build.py --cpu'. Run cudalm to init all folders.
3. Download all LS3-Micro files from LS3-Micro-Base-Checkpoint\ and drop them in <cudalm.exe>\models\. Cudalm then automatically detects the model.

Voila, you can now start experimenting with the model. Some basic stuff you can try first:

• Check the model's config, and settings used to train it. > Model info.
• Run the model: > Generate.

To reproduce the benchmark results, use > LM eval. Run all benchmarks.

Finetune guide:

To finetune the model, you first need to prepare some data.

Navigate to > Data tools.... You can download preconfigured mixtures of data, use files on your system or nitpick your own datasets.

• To use a preset corpus, navigate into > Data tools... > Download text.
• Otherwise, find a dataset on hugginface and navigate to > Data tools... > Browse Hugging Face, and pick out your chosen dataset from there. Note that datasets with gated access requires your HF_TOKEN to be set.

This guide will use the dataset MiniCPM5-2B-distill as an example. For your own text or other datasets, simply select the folder while tokenizing.

Navigate to > Data tools... > Browse Hugging Face, and search for "MiniCPM5-2B-distill". Cudalm automatically detects the format, so all you need to do is download it (~250MB).

Then you will need to tokenize the text so that the model can be trained on it. To do this, you will need a trained tokenizer. Luckily, you can extract an existing model's tokenizer. Use > Data tools... > Tokenizer from model and pull out LS3-Micro-Base's tokenizer.

Now you can tokenize your data. Go to > Data tools... > Build corpus and set mode to "new" and give it a corpus name. For your own datasets or text, navigate to the folder the text files are in. Otherwise, select <cudalm.exe>\data\hf_MiniCPM5-2B-distill, then use the extracted tokenizer and select Run. It should take ~10s and produce <cudalm.exe>\data\<corpus name>

You're now ready to start fine tuning!

Go to > Train and select "LS3-Micro-Base" as the model, and the tokenized corpus as the data directory (both auto-detected). Select Run to start training the model on the selected corpus. Progress prints and periodic saves will occur every few hundred iters.