higgs-audio-v2-tokenizer
This repository hosts the higgs-audio-v2-tokenizer, the audio codec of
Reverb-TTS, for Reverb's
runtimes: ExecuTorch .pte programs for the
React Native ExecuTorch
library, ready to run on device, and quantized safetensors.
Upstream model: Higgs Audio v2 tokenizer
Inputs and outputs
The Higgs Audio v2 tokenizer turns 24 kHz mono audio into 8 codebooks of 1024 codes at 25 frames per second (960 samples per frame), and back.
Each file has one method, forward.
Decoder: forward(codes) -> audio
codesint64[8, T]: row i is codebook i of the residual quantizer (0 the coarsest), each value in 0-1023. T runs from 4 to 1024 frames (0.16 to 41 s).audiofloat32[T * 960]: 24 kHz mono samples, not clipped to [-1, 1].
Encoder: forward(wav) -> codes
wavfloat32[frames, 960]: 24 kHz mono in [-1, 1], cut to whole frames of 960 samples and reshaped row by row. frames runs from 25 to 500 (1 to 20 s). Resampling to 24 kHz and downmixing to mono happen outside the model.codesint64[8, frames]: as the decoder takes them.
Variants
| Path | Component | Backend | Precision |
|---|---|---|---|
decoder/coreml/fp16/codec_decoder_coreml_fp16.pte |
decoder | coreml | fp16 |
decoder/coreml/int8/codec_decoder_coreml_int8.pte |
decoder | coreml | int8 |
encoder/coreml/int4/codec_encoder_coreml_int4.pte |
encoder | coreml | int4 |
encoder/coreml/int8/codec_encoder_coreml_int8.pte |
encoder | coreml | int8 |
Repository structure
config.json 46 B
decoder/coreml/fp16/codec_decoder_coreml_fp16.pte 43.6 MB
decoder/coreml/int8/codec_decoder_coreml_int8.pte 23.3 MB
encoder/coreml/int4/codec_encoder_coreml_int4.pte 136 MB
encoder/coreml/int8/codec_encoder_coreml_int8.pte 176 MB
Quantized weights
The quantized weights the int8 and int4 exports were made from are stored as
safetensors, each with its quantization.json, in decoder/safetensors/int8,
encoder/safetensors/int8 and encoder/safetensors/int4.
Compatibility
These files are published for the ExecuTorch v1.4.1 runtime. ExecuTorch gives no forward compatibility guarantee, so an older runtime may fail to load them.
To use them in React Native ExecuTorch, pass the model constant shipped in the library's model registry to the corresponding task pipeline. See the documentation.
To load these files in your own ExecuTorch runtime, read the compatibility note first.
Model details
Variants
- Decoder: Core ML on the CPU and GPU, fp16 compute.
fp16carries the codec's own weights;int8carries int8 weights from GPTQ and quantization-aware training of the scales, one scale per 8 input channels. - Encoder: Core ML on the CPU only, fp32 compute, as the encoder's
activations need fp32. Both variants carry quantization-aware-trained weights:
int8everywhere,int4in HuBERT's transformer layers and int8 elsewhere.
Measured on an iPhone 17 (iOS 26.6) with React Native ExecuTorch's native libraries v0.11.1. Quality is against the fp32 PyTorch codec: the decoder's waveform SNR on 14 test clips, the encoder's share of equal codes (codebook 0 / 1 / 2-7) on 57 test clips in 7 languages.
| File | Time | Peak memory | Quality vs fp32 |
|---|---|---|---|
| decoder fp16 | 53 / 107 ms per 5 / 10 s | ~200 MB at 10 s | 54.8 dB |
| decoder int8 | 53 / 109 ms per 5 / 10 s | ~690 MB at 10 s | 44.2 dB |
| encoder int8 | 0.38 / 0.84 / 1.97 s per 5 / 10 / 20 s | 420 / 829 MB at 10 / 20 s | 99.51 / 99.10 / 96.63% |
| encoder int4 | 0.32 / 0.75 / 1.86 s per 5 / 10 / 20 s | 420 / 860 MB at 10 / 20 s | 99.07 / 98.31 / 94.57% |
The decoder's int8 file is half the size of fp16 but takes more memory once loaded, as Core ML expands its weights on the GPU. The encoder runs on the CPU for the same reason: on the GPU, Core ML holds 1.4 to 2.6 GB of expanded fp32 weights, while on the CPU it maps them from the file.
- Downloads last month
- 42
Model tree for software-mansion/higgs-audio-v2-tokenizer-Reverb
Base model
bosonai/higgs-audio-v2-tokenizer