JimHue/TASTE-S-SLM

TASTE-S stage-1 speech LM — model release bundle

Everything needed to run inference and evaluation with our pretrained stage-1 speech LM. Total ≈ 24 GB.

taste_s_slm_release/
├── slm/                        ⭐ OUR PRETRAINED MODEL                       3.8 GB
│   ├── training_config.yaml       the architecture spec -- DO NOT hand-edit
│   └── valid_best/
│       ├── model.safetensors      weights (327 tensors: Qwen3.5 backbone + speech proj + speech head)
│       └── acc.txt                validation metrics at the selected step
├── taste_s_tokenizer/          TASTE-S tokenizer `streaming_causal_v2`        11 GB
│                                  CTC head + aggregator + FSQ quantizer + unit decoder + vocoder
├── lm_decoder_base/            base weights for the unit decoder's LM         3.0 GB
├── cosyvoice/                  CosyVoice2-0.5B: vocoder + campplus.onnx       4.6 GB
├── qwen3.5-0.8b/               the SLM's backbone (config + weights)          1.7 GB
└── text_tokenizer/             Qwen3.5 text tokenizer (tokenizer files only)   22 MB

Why all six pieces are required

A reasonable assumption is that for TTS with ground-truth text you would only need the SLM. That is not the case:

piece	why you cannot drop it
slm/	the model under test
taste_s_tokenizer/	⭐ the SLM cannot even be constructed without it. model.py:135 loads the frozen FSQ quantizer from here, and reads the codebook geometry (16 codebooks × 6561, code_dim 128) off it. It also holds the unit decoder + vocoder that turn predicted taste tokens into audio, and the CTC head that produces conditioning text
lm_decoder_base/	⚠️ the tokenizer's config.json has an absolute path to this directory baked in (unit_decoder__lm_decoder_pretrained_model_name_or_path). The loader rewrites it to --lm_backbone_dir, but the directory must exist
cosyvoice/	the vocoder, plus campplus.onnx which computes the speaker embedding from the audio
qwen3.5-0.8b/	the backbone architecture is instantiated from this config before our weights are loaded over it
text_tokenizer/	text → token ids. Only the tokenizer files are shipped, not the 19 GB of 9B weights

⚠️ qwen3.5-0.8b/ weights are technically redundant — our checkpoint contains all 321 lm.* tensors including embed_tokens and lm_head, so they are overwritten immediately. They are included so the normal load path works without special flags.

⚠️ If you are the one publishing this bundle

The directory as built is a symlink tree — do not upload it directly. Measured on huggingface_hub 1.26.0 / Python 3.12: upload_folder enumerates with Path.glob("**/*"), which includes symlinked files but does not recurse into symlinked directories. A direct upload therefore pushes 7 files / 23 MB instead of 24 GB, silently.

bash materialize.sh /some/local/disk/taste_s_slm_release_real   # dereferences + verifies sizes
cd /some/local/disk/taste_s_slm_release_real
hf upload <repo-id> . --repo-type model

---

Where to put it

# 1. download / unpack anywhere, e.g.
#    ~/models/taste_s_slm_release
# 2. point the repo at it (one symlink, no code changes):
cd <repo>/egs/taste_s/ljh_exp/finetune_qwen3_5/slm
ln -sfn ~/models/taste_s_slm_release  pretrained
# 3. sanity check
ls pretrained/slm/valid_best/model.safetensors pretrained/taste_s_tokenizer/config.json

Every default path in evaluate.py resolves inside pretrained/, so after that symlink the whole evaluation is one command:

bash run_eval.sh                 # ceiling / harness check
ARM=slm bash run_eval.sh         # our model

See the repo's slm/README.md for what the numbers mean and how to point it at your own benchmark.

Not included

	why
LibriSpeech test-clean (349 MB)	licensing; download it separately. The repo expects it at slm/librispeech_test_clean/ with a 2-column TSV. slm/README.md documents the format
HuBERT ASR + UTMOS	pulled from HuggingFace on first use (facebook/hubert-large-ls960-ft, tarepan/SpeechMOS)
training data (taste tokens)	not needed for evaluation; ~37 GB per corpus
resume checkpoints	checkpoints/*.ckpt are 9 GB each and only useful for continuing our exact run

Provenance

SLM run	stage1_qwen35_0p8b_v2_d5, 0.8B, 8 epochs, valid_best = step 7118
SLM validation	val_loss_speech 6.5807 (unigram baseline 8.6367, uniform 8.7889); group acc 0.3697 (chance 0.1111)
tokenizer	streaming_causal_v2: LibriSpeech resynthesis WER 4.01 % / UTMOS 4.0818 vs the offline bidirectional reference 4.15 % / 4.083
T1 end-to-end	greedy 5.182 % vs the GT-taste ceiling 2.852 % (200 LibriSpeech utts, CTC conditioning text)
built	2026-08-17

⚠️ Tokenizer generations are not interchangeable: taste codes from streaming_causal (v1) and streaming_causal_v2 agree bitwise 0.0000 of the time. This SLM only works with the taste_s_tokenizer/ shipped here.