SIMB
Weights for SIMB: Self-Improving Cross-Modal Distillation via a Mixture of Bridges without Forgetting. Code and usage: https://github.com/KU-AGI/SIMB
SIMB runs a frozen Janus-Pro-7B (VL expert) side by side with a speech expert (first 30 layers of Qwen2.5-3B over 12.5 Hz DualCodec tokens). Pathway-specific key/value bridges between the experts give speech-to-image generation, image-to-speech and spoken visual question answering, while the VL expert stays bit-identical to Janus-Pro-7B.
This repo is self-contained: the inference code loads everything from it and downloads no other model.
| Files | Contents | dtype |
|---|---|---|
model-0000*-of-00005.safetensors, config.json |
VL expert (Janus-Pro-7B LLM), SigLIP encoder + aligner, VQ image decoder, image embedding / generation head, speech expert, S2I / I2S / VQA bridges, speech embeddings and heads (11.8B parameters) | bf16 |
tokenizer*.json, special_tokens_map.json |
Janus-Pro tokenizer | |
codec/ |
DualCodec 12.5 Hz codec, w2v-BERT 2.0 semantic encoder, feature normalization statistics | fp32 |
from simb import SIMB # pip install -e . in https://github.com/KU-AGI/SIMB
model = SIMB.from_pretrained("KU-AGI/SIMB")
image, _ = model.generate_image("prompt.wav", seed=0)
License
These weights combine components under different licenses; every use must comply with all of them (see NOTICE and licenses/):
- Janus-Pro-7B components: DeepSeek License Agreement, including its use-based restrictions (
licenses/Janus-Pro-7B_DEEPSEEK_LICENSE_AGREEMENT.txt). - Speech expert, initialized from Qwen2.5-3B: Qwen Research License Agreement, research and evaluation use only (
licenses/Qwen2.5-3B_QWEN_RESEARCH_LICENSE_AGREEMENT.txt). Qwen is licensed under the Qwen RESEARCH LICENSE AGREEMENT, Copyright (c) Alibaba Cloud. All Rights Reserved. Built with Qwen. - DualCodec (
amphion/dualcodec):licenses/DualCodec_MIT_LICENSE.txt. - w2v-BERT 2.0 (
facebook/w2v-bert-2.0): MIT (licenses/w2v-bert-2.0_MIT_LICENSE.txt).
- Downloads last month
- 12
Model tree for KU-AGI/SIMB
Base model
Qwen/Qwen2.5-3B