SIMB

Weights for SIMB: Self-Improving Cross-Modal Distillation via a Mixture of Bridges without Forgetting. Code and usage: https://github.com/KU-AGI/SIMB

SIMB runs a frozen Janus-Pro-7B (VL expert) side by side with a speech expert (first 30 layers of Qwen2.5-3B over 12.5 Hz DualCodec tokens). Pathway-specific key/value bridges between the experts give speech-to-image generation, image-to-speech and spoken visual question answering, while the VL expert stays bit-identical to Janus-Pro-7B.

This repo is self-contained: the inference code loads everything from it and downloads no other model.

Files Contents dtype
model-0000*-of-00005.safetensors, config.json VL expert (Janus-Pro-7B LLM), SigLIP encoder + aligner, VQ image decoder, image embedding / generation head, speech expert, S2I / I2S / VQA bridges, speech embeddings and heads (11.8B parameters) bf16
tokenizer*.json, special_tokens_map.json Janus-Pro tokenizer
codec/ DualCodec 12.5 Hz codec, w2v-BERT 2.0 semantic encoder, feature normalization statistics fp32
from simb import SIMB   # pip install -e . in https://github.com/KU-AGI/SIMB
model = SIMB.from_pretrained("KU-AGI/SIMB")
image, _ = model.generate_image("prompt.wav", seed=0)

License

These weights combine components under different licenses; every use must comply with all of them (see NOTICE and licenses/):

  • Janus-Pro-7B components: DeepSeek License Agreement, including its use-based restrictions (licenses/Janus-Pro-7B_DEEPSEEK_LICENSE_AGREEMENT.txt).
  • Speech expert, initialized from Qwen2.5-3B: Qwen Research License Agreement, research and evaluation use only (licenses/Qwen2.5-3B_QWEN_RESEARCH_LICENSE_AGREEMENT.txt). Qwen is licensed under the Qwen RESEARCH LICENSE AGREEMENT, Copyright (c) Alibaba Cloud. All Rights Reserved. Built with Qwen.
  • DualCodec (amphion/dualcodec): licenses/DualCodec_MIT_LICENSE.txt.
  • w2v-BERT 2.0 (facebook/w2v-bert-2.0): MIT (licenses/w2v-bert-2.0_MIT_LICENSE.txt).
Downloads last month
12
Safetensors
Model size
12B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for KU-AGI/SIMB

Base model

Qwen/Qwen2.5-3B
Finetuned
(613)
this model