Satella-30B-A3B

Sparse architecture. Deep reasoning.

A 30.5B-parameter Mixture-of-Experts reasoning model built from Qwen3-30B-A3B through a single CoT-preserving BF16 LoRA pass. Satella is the flagship sparse model of the Atem family: 30.5B total parameters, approximately 3.3B activated per token, native thinking and non-thinking modes, and a training mixture centred on mathematics, code, analytical reasoning, and technically grounded instruction following.

---

Overview

Satella-30B-A3B is the large sparse reasoning branch of the Atem model series. It begins from the already post-trained Qwen/Qwen3-30B-A3B checkpoint and preserves the base model's native dual-mode reasoning behaviour rather than erasing and rebuilding it in separate stages.

The training objective was not to replace Qwen3's broad foundation. It was to make that foundation more deliberate, consistent, and practically useful on difficult tasks: sustaining multi-step reasoning, producing complete code, checking assumptions, correcting mistakes, and separating confident conclusions from uncertainty.

Satella was trained with response-only loss on manually verified assistant-token masks. The model saw a mixed corpus of explicit reasoning traces and genuine direct answers, while the original MoE expert router remained frozen. Its identity is also supplied through a default chat-template system prompt that activates only when the caller does not provide a system message of their own.

The run was manually stopped after 1,078 of 1,505 planned optimiser steps, at approximately 72% of one epoch. Validation loss continued to improve through the final evaluation, reaching 0.59157.

---

Model Details

Property	Value
Base model	Qwen/Qwen3-30B-A3B
Architecture	Sparse Mixture-of-Experts
Total parameters	~30.5B
Activated parameters	~3.3B per token
Experts	128 total, 8 activated per token
Training method	Single-pass CoT-preserving BF16 LoRA SFT
LoRA config	r=16, alpha=32, dropout=0.0
Target modules	q_proj, k_proj, v_proj, o_proj, gate_up_proj, down_proj
Expert router	Frozen
Trainable parameters	642,514,944 (2.061% of loaded parameters)
Training records	24,067 train / 320 validation
Think / direct split	19,542 / 4,525 — 81.2% / 18.8%
Optimiser steps	1,078 / 1,505 planned
Epoch completion	~0.72 / 1.0
Effective batch size	16 (batch 2 × gradient accumulation 8)
Learning rate	3e-5, cosine schedule, 3% warm-up
Max training sequence length	8,192 tokens
Precision	bfloat16 LoRA, not QLoRA
Optimiser	8-bit AdamW, β₁=0.9, β₂=0.95
Weight decay	0.01
Gradient clipping	1.0
Hardware	NVIDIA RTX PRO 6000 Blackwell Server Edition, 96GB
Runtime	4h 52m
Final validation loss	0.591570
Base-model licence	Apache 2.0
Repository licence metadata	other — see Data Provenance and Terms

---

Design Notes

One CoT-preserving pass

Satella is trained directly on the post-trained Qwen3-30B-A3B model. The base model's native reasoning system, instruction following, and thinking/non-thinking modes remain intact. The external reasoning corpus is layered onto that foundation in one supervised pass rather than using an erase-then-rebuild pipeline.

Sparse-MoE LoRA

Qwen3-30B-A3B contains 30.5B total parameters but activates approximately 3.3B per token. LoRA adapters were attached to the attention projections and packed expert MLP projections. The expert-selection router was explicitly checked and kept frozen throughout training, preserving the base model's learned routing policy.

A substantial adapter

Although r=16 appears modest, applying it across the attention and MoE expert projections produces 642.5M trainable parameters. The adapter alone is larger than many complete small language models and represents a meaningful adaptation rather than a cosmetic identity tune.

Response-only loss

System, user, and tool-context tokens were masked from the loss. Training gradients were applied only to assistant reasoning and final-answer tokens. Mask construction was validated before training, with zero invalid assistant masks across the retained corpus.

Native dual-mode behaviour

Satella was trained on both explicit reasoning traces and genuine direct answers:

• Thinking: 19,542 records — 81.2%
• Direct: 4,525 records — 18.8%

Thinking records preserve Qwen3's <think>...</think> structure. Direct records teach concise answers without randomly stripping reasoning from examples that depend on it.

Default identity without template replacement

The repository's chat_template.jinja inserts Satella's system prompt only when no explicit system message is present. User- or application-supplied system prompts retain priority. The underlying Qwen3 template remains responsible for ordinary chat and tool-enabled formatting.

---

Intended Use

Satella is designed for tasks where sustained reasoning and complete execution matter:

• Multi-step mathematical and logical reasoning
• Code generation, debugging, explanation, and algorithm design
• Scientific and technical explanation
• Analytical comparison and argument evaluation
• Complex instruction following with multiple constraints
• Long-form problem decomposition and self-correction
• General-purpose assistance in thinking or direct-answer mode

Satella is not a retrieval system and does not possess live information. It should not be treated as an authoritative source for medical, legal, financial, or safety-critical decisions without independent verification.

---

Training Data

The final training split contained 24,067 records after quality filtering, deduplication, context-length filtering, and source-aware validation splitting.

Domain Distribution

Domain	Training records	Share
Code	7,057	29.3%
Reasoning	5,429	22.6%
Mathematics	5,292	22.0%
General dialogue/instruction	3,948	16.4%
Medical reasoning	1,974	8.2%
Instruction/identity	248	1.0%
Tool use	119	0.5%
Total	24,067	100%

Source Families

Dataset	Role in the mixture
r0b0tlab/qwen3.8-max-distillation-50k	Primary high-quality teacher-distillation source across mathematics, code, reasoning, instruction following, and limited tool use; filtered to quality score ≥8.5 and prioritised by difficulty
mitroitskii/OpenR1-Math-220k-formatted	Correctness-filtered mathematical reasoning
nvidia/OpenCodeReasoning	Code reasoning, implementation, and algorithmic problem solving
FreedomIntelligence/medical-o1-reasoning-SFT	English medical reasoning traces
HuggingFaceH4/ultrachat_200k	Genuine direct-answer conversational data
Custom Satella identity records	Identity, provenance, response discipline, and system-prompt behaviour
Custom native Qwen tool records	Preservation of tool-call and tool-result formatting

A total of 26,939 records entered tokenisation. Of these, 24,387 were retained at 8,192 tokens, 2,552 exceeded the context limit, and zero failed assistant-mask validation. The selected corpus contained approximately 20.9M assistant-loss tokens. The manually stopped run consumed approximately 29.8M total input tokens and an estimated 15.0M assistant-loss tokens.

---

Training Configuration

# Core training configuration
base_model          = "unsloth/Qwen3-30B-A3B"
max_seq_length      = 8192

lora_r              = 16
lora_alpha          = 32
lora_dropout        = 0.0
lora_targets        = [
    "q_proj", "k_proj", "v_proj", "o_proj",
    "gate_up_proj", "down_proj",
]
router_training     = False

batch_size          = 2
grad_accumulation   = 8       # effective batch size: 16
learning_rate       = 3e-5
lr_scheduler        = "cosine"
warmup_ratio        = 0.03
weight_decay        = 0.01
max_grad_norm       = 1.0
optimizer           = "adamw_8bit"

num_epochs          = 1       # ceiling
completed_steps     = 1078    # manually stopped
planned_steps       = 1505
precision           = "bfloat16"
gradient_checkpointing = "unsloth"

Training used Unsloth's Qwen3-MoE kernels with smart gradient offload and double buffering on an NVIDIA RTX PRO 6000 Blackwell Server Edition.

---

Loss Curve

Step	Training loss	Validation loss	Input tokens seen
200	0.556072	0.645650	5,604,240
400	0.480096	0.618799	11,101,712
600	0.495532	0.607141	16,603,280
800	0.437907	0.596377	21,973,648
1,000	0.529542	0.592443	27,416,128
1,078	0.433706	0.591570	29,793,696

Validation loss improved at every evaluation point. The final in-memory step-1,078 model achieved a lower validation loss than the latest scheduled checkpoint at step 1,000 and was the state merged into this repository.

The logged training loss is a local interval measurement and varies with batch domain, difficulty, and sequence length. Validation loss is the more useful trend signal across the run.

---

Evaluation

A clean benchmark suite has not yet been published for Satella-30B-A3B. Training and validation loss are not substitutes for external evaluation.

Planned comparisons against the untouched Qwen/Qwen3-30B-A3B base include:

• ARC-Challenge
• MMLU
• HellaSwag
• Winogrande
• PIQA
• BoolQ
• OpenBookQA
• Code-generation benchmarks
• Clean held-out mathematical and general-reasoning tasks

---

Usage

Transformers — Thinking Mode

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "EphAsad/Satella-30B-A3B"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {
        "role": "user",
        "content": (
            "A train travels from A to B at 60 km/h and returns at "
            "90 km/h. What is its average speed over the full journey?"
        ),
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=4096,
        do_sample=True,
        temperature=0.6,
        top_p=0.95,
        top_k=20,
    )

new_tokens = outputs[0][inputs["input_ids"].shape[-1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

Transformers — Direct Mode

inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Summarise the Monty Hall problem in three sentences."}],
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=True,
        temperature=0.7,
        top_p=0.8,
        top_k=20,
    )

new_tokens = outputs[0][inputs["input_ids"].shape[-1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

Explicit System-Prompt Override

Satella's identity activates automatically only when no system message is supplied. Applications can override it normally:

messages = [
    {
        "role": "system",
        "content": "You are a concise VBA and Microsoft Access specialist.",
    },
    {
        "role": "user",
        "content": "Explain why a DAO recordset may raise error 3021.",
    },
]

Sampling Parameters

Recommended starting values follow the Qwen3 base model's dual-mode guidance.

Thinking mode

temperature = 0.6
top_p       = 0.95
top_k       = 20
min_p       = 0

Direct / non-thinking mode

temperature = 0.7
top_p       = 0.8
top_k       = 20
min_p       = 0

Avoid greedy decoding for long thinking-mode outputs. Generation behaviour can vary by inference engine and quantisation.

---

Default System Prompt

Satella's default identity is embedded in chat_template.jinja and is used only when the caller provides no system message:

You are Satella, the flagship reasoning model of the Atem family, developed by EphAsad.

Your purpose is to solve difficult problems with precision, adaptability, and intellectual honesty. Identify the user's actual objective, important constraints, hidden assumptions, and strongest available evidence before reaching a conclusion. Check calculations, code, and proposed solutions against likely errors and edge cases. Correct mistakes rather than defending them.

Give the answer in the format and level of detail the user requests. Be direct, clear, and practically useful. Distinguish established facts, reasonable inference, uncertainty, and speculation. Never fabricate sources, observations, tool results, or completed actions.

For programming tasks, produce complete, runnable, maintainable solutions. Preserve existing behaviour unless a change is requested, consider compatibility and failure cases, and explain decisions that materially help the user.

Use deliberate reasoning for difficult problems and efficient direct answers for straightforward requests. Do not make an answer verbose merely to appear thorough.

When asked about your identity or origin, state accurately that you are Satella, a model developed by EphAsad using Qwen3-30B-A3B as its base.

---

Known Limitations

No clean benchmark claims yet

Satella has not yet completed a controlled side-by-side benchmark suite against its untouched base using identical prompts, decoding settings, and evaluation backends. The falling validation loss is encouraging but does not prove universal capability gains.

Reasoning-heavy style

The corpus is approximately 81% thinking data. Satella may reason too extensively on simple tasks, imitate teacher-model formatting habits, or produce longer answers than necessary. Use non-thinking mode for straightforward requests.

Tool use was preserved, not deeply trained

Only 119 final training records were categorised as tool use. The run was designed to preserve native Qwen3 tool formatting, not to create a specialised agent model.

English-centred SFT

The base model is multilingual, but this fine-tuning mixture is primarily English. Multilingual capabilities were inherited from Qwen3 and were not specifically improved or comprehensively regression-tested.

No preference optimisation

Satella has not undergone RLHF, DPO, GRPO, or another preference-optimisation stage. It may be less consistent than preference-tuned models on subjective style, refusal calibration, or open-ended conversational preferences.

Model size

The merged BF16 model is approximately 61GB and requires substantial system or GPU memory. Quantised GGUF releases are recommended for ordinary local deployment.

---

Data Provenance and Terms

The base model Qwen/Qwen3-30B-A3B is released under Apache 2.0. However, Satella's training-data lineage is not Apache-2.0-only.

The primary dataset, r0b0tlab/qwen3.8-max-distillation-50k, uses Hugging Face metadata license: other because its source mixture has heterogeneous upstream licences, benchmark origins, and provider terms. Its provenance notice should be reviewed before redistribution, commercial use, or downstream training.

For that reason, this repository uses:

license: other

This model card does not grant rights beyond those provided by the base model, source datasets, teacher-output terms, and applicable law. Downstream users are responsible for reviewing those terms and determining whether their intended use is permitted.

---

Available Files

File	Description
model-00001-of-00013.safetensors … model-00013-of-00013.safetensors	Complete merged BF16 model weights, approximately 61GB total
model.safetensors.index.json	Parameter-to-shard index required to load the model
config.json	Qwen3-MoE architecture configuration
generation_config.json	Default generation and special-token settings
tokenizer.json	Tokenizer vocabulary and tokenisation rules
tokenizer_config.json	Tokenizer and chat-template configuration
chat_template.jinja	Qwen3 chat formatting plus default Satella identity injection
logo.png	Satella model logo used by this model card

---

Citation

@misc{satella_30b_a3b_2026,
  author       = {Asad, Zain},
  title        = {Satella-30B-A3B: A CoT-Preserving Sparse Reasoning Model
                  via BF16 MoE LoRA SFT on Qwen3-30B-A3B},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/EphAsad/Satella-30B-A3B}},
}

---

Base-Model Attribution

Satella was not pretrained from scratch. It is a fine-tuned derivative of Qwen/Qwen3-30B-A3B, developed by the Qwen team. See the original Qwen model card and Qwen3 technical report for architecture, pretraining, multilingual capability, and base-model evaluation details.

---

Built independently by Zain Asad — EphAsad