8BitStudio/Sekai-1

Sekai-1 is the first model in the Sekai series: a small language model trained from scratch on a single RTX 5060 Ti.

The model has 95.6M parameters, with 23 layers, a hidden size of 576 and a 2,048-token context window. It uses a Qwen3-style decoder-only architecture (grouped-query attention with 9 query heads and 3 key-value heads, a SwiGLU MLP, RMSNorm, RoPE, tied embeddings) and its own 24,576-token BPE tokenizer, trained on Sekai’s data mix, that splits numbers into single digits.

Sekai-1 is pretrained on 58.5B tokens of web, educational, math, code and conversation data, using the Muon and AdamW optimizers with a warmup-stable-decay learning-rate schedule.

Evaluation

The model scores number two in the under 100M parameters, just behind KLM-1.5 on the SLM leaderboard as seen in the image above. All scores are zero-shot. Sekai-1's scores were measured locally on the same tasks the leaderboard uses; the other models' scores are taken from the leaderboard (November 7, 2026).

Training

Pretraining: 58.5B tokens of web, educational, math, code and conversation data, with a warmup-stable-decay schedule. The final ~12B tokens were a decay phase on a higher-quality mix. Chat fine-tuning (SFT): about 400M tokens of instruction following, everyday chat, short and long answers, summarizing and rewriting, code writing and code fixing, plus a small set of identity examples, so the model knows it is Sekai-1. It does not use chain-of-thought (a model this size doesn't benefit from it), and refusals were filtered out of the data.

How to use

pip install transformers torch

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "8BitStudio/Sekai-1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [{"role": "user", "content": "Who are you?"}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_dict=True, return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

The model is small enough to run on a CPU. The default sampling settings (temperature 0.5, top-p 0.9, top-k 40, repetition penalty 1.1) are stored in generation_config.json; pass do_sample=False to generate() for deterministic answers.

Sekai-1 uses the ChatML format:

<|im_start|>user
What is the capital of France?<|im_end|>
<|im_start|>assistant
Paris<|im_end|>

Limitations

Sekai-1 is a 95.6M-parameter model. It makes factual mistakes, loses track in long conversations, and is weak at multi-step reasoning and math beyond simple arithmetic. It has no safety tuning or refusal behavior, so don't use it anywhere that needs either. It understands English only. Check responses for the things that matter.