license: apache-2.0
language: en
library_name: mlx
pipeline_tag: text-generation
base_model: mlx-community/SmolLM-135M-Instruct-4bit
tags:
- mlx
- 4-bit
- apple-silicon
- text-generation
tiny
A four-bit quantised conversion of mlx-community/SmolLM-135M-Instruct-4bit, produced with mlx-lm 0.30.4 and intended for Apple Silicon. The architecture is a Llama variant (thirty layers, 576 hidden dimensions, nine attention heads, three key-value heads, 49152-token vocabulary) with group-size 64 four-bit weights, occupying about 76 MB.
The model.safetensors.index.json alongside the weights is not a sharded checkpoint here; it maps all 694 tensors onto the single model.safetensors file that sits beside it.
Because the weights are quantised, transformers cannot load this checkpoint; it needs mlx-lm, which applies the group quantisation at load time. The instruction tuning lives upstream in SmolLM; this repository adds a format, nothing else. Given the size, expect it to be fast and largely incoherent on factual questions.
Usage
from mlx_lm import load, generate
model, tokenizer = load("harpertoken/tiny")
prompt = "hello"
if tokenizer.chat_template is not None:
prompt = tokenizer.apply_chat_template(
[{"role": "user", "content": prompt}],
add_generation_prompt=True,
return_dict=False,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=100))
Install with pip install mlx-lm. A chat_template.jinja is present, so the template branch above will be taken.
Limitations
This is a redistribution of someone else's work in a different numeric format. There is no training, evaluation or adaptation to report, and if you want SmolLM-135M-Instruct you should use the upstream repository, which is not tied to MLX and can be converted to other formats. Quantisation to four bits costs some accuracy even before the underlying 135M-parameter capacity does; I have not measured the loss and so make no claim about it.
Attribution
SmolLM follows Allouji et al., SmolLM: blazingly fast and remarkably powerful (2024).