Merlin-9M

Merlin-9M is an 8.85M-parameter decoder-only language model, trained from scratch on Stew-10B (Wikipedia / Ultra-FineWeb / DCLM) with no benchmark data of any kind in the training mix. It is the checkpoint from step 12,502 of training, seen ~811M tokens.

Architecture

Property	Value
Parameters	8,854,146
d_model	256
Layers	12 unique
Heads	8 (head_dim 32)
FFN	SwiGLU(384)
Vocabulary	8,192 (GPT-2's first 8192 merges)
Context	1024
RoPE	yes
Normalization	RMSNorm + QK-norm
Activation	SwiGLU
Output	logits softcap 15
Optimizer	Muon (2D weights) + AdamW (embeddings)

Depth-recurrent core

Layers 3–8 are applied twice, giving 18 effective layers from 12 unique ones at zero extra parameter cost. The first pass uses a learned residual scale; the second pass uses a smaller one (scale_loop).

Multi-token prediction (MTP-2)

The training objective adds an auxiliary head predicting the token two positions ahead, tied to the same embedding matrix. This does not affect inference — the model is used as a standard causal LM.

How to use

from huggingface_hub import hf_hub_download
import torch

# download checkpoint
path = hf_hub_download(repo_id="Monster-Code/Merlin-9M",
                       filename="brasa9m_last.pt", repo_type="model")

# The checkpoint dict contains:
#   ck["model"]   -> state_dict
#   ck["cfg"]     -> architecture config
#   ck["step"]    -> training step (12502)

# See merlin9m.py in this repo for the model definition + a load_model() helper.

The merlin9m.py file in this repository is a self-contained inference module:

from merlin9m import load_model, generate

model, cfg, tok, step = load_model("Monster-Code/Merlin-9M")
print(generate(model, cfg, tok, "The capital of France is"))

Tokenizer

GPT-2 BPE. The model was trained on the first 8,192 token IDs of GPT-2's 50,257-token vocabulary — these are the most frequent merges in English. Input token IDs outside [0, 8191] are clamped during inference.

Training data

Monster-Code/Stew-10B-tokenized — a 10B-token pretraining corpus:

Source	Share	Tokens
Wikipedia (20231101.en)	25%	2.5B
Ultra-FineWeb (English)	50%	5.0B
DCLM-baseline 1.0	25%	2.5B

No ARC, HellaSwag, PIQA, WinoGrande, or any other benchmark data was used in training.

License

Mixed upstream licenses (CC BY-SA 3.0, GFDL, see upstream dataset cards). No additional restrictions are imposed by this repository.