higgs-audio-v2-tokenizer

This repository hosts the higgs-audio-v2-tokenizer, the audio codec of Reverb-TTS, for Reverb's runtimes: ExecuTorch .pte programs for the React Native ExecuTorch library, ready to run on device, and quantized safetensors.

Upstream model: Higgs Audio v2 tokenizer

Inputs and outputs

The Higgs Audio v2 tokenizer turns 24 kHz mono audio into 8 codebooks of 1024 codes at 25 frames per second (960 samples per frame), and back.

Each file has one method, forward.

Decoder: forward(codes) -> audio

  • codes int64 [8, T]: row i is codebook i of the residual quantizer (0 the coarsest), each value in 0-1023. T runs from 4 to 1024 frames (0.16 to 41 s).
  • audio float32 [T * 960]: 24 kHz mono samples, not clipped to [-1, 1].

Encoder: forward(wav) -> codes

  • wav float32 [frames, 960]: 24 kHz mono in [-1, 1], cut to whole frames of 960 samples and reshaped row by row. frames runs from 25 to 500 (1 to 20 s). Resampling to 24 kHz and downmixing to mono happen outside the model.
  • codes int64 [8, frames]: as the decoder takes them.

Variants

Path Component Backend Precision
decoder/coreml/fp16/codec_decoder_coreml_fp16.pte decoder coreml fp16
decoder/coreml/int8/codec_decoder_coreml_int8.pte decoder coreml int8
encoder/coreml/int4/codec_encoder_coreml_int4.pte encoder coreml int4
encoder/coreml/int8/codec_encoder_coreml_int8.pte encoder coreml int8

Repository structure

config.json                                          46 B
decoder/coreml/fp16/codec_decoder_coreml_fp16.pte    43.6 MB
decoder/coreml/int8/codec_decoder_coreml_int8.pte    23.3 MB
encoder/coreml/int4/codec_encoder_coreml_int4.pte    136 MB
encoder/coreml/int8/codec_encoder_coreml_int8.pte    176 MB

Quantized weights

The quantized weights the int8 and int4 exports were made from are stored as safetensors, each with its quantization.json, in decoder/safetensors/int8, encoder/safetensors/int8 and encoder/safetensors/int4.

Compatibility

These files are published for the ExecuTorch v1.4.1 runtime. ExecuTorch gives no forward compatibility guarantee, so an older runtime may fail to load them.

To use them in React Native ExecuTorch, pass the model constant shipped in the library's model registry to the corresponding task pipeline. See the documentation.

To load these files in your own ExecuTorch runtime, read the compatibility note first.

Model details

Variants

  • Decoder: Core ML on the CPU and GPU, fp16 compute. fp16 carries the codec's own weights; int8 carries int8 weights from GPTQ and quantization-aware training of the scales, one scale per 8 input channels.
  • Encoder: Core ML on the CPU only, fp32 compute, as the encoder's activations need fp32. Both variants carry quantization-aware-trained weights: int8 everywhere, int4 in HuBERT's transformer layers and int8 elsewhere.

Measured on an iPhone 17 (iOS 26.6) with React Native ExecuTorch's native libraries v0.11.1. Quality is against the fp32 PyTorch codec: the decoder's waveform SNR on 14 test clips, the encoder's share of equal codes (codebook 0 / 1 / 2-7) on 57 test clips in 7 languages.

File Time Peak memory Quality vs fp32
decoder fp16 53 / 107 ms per 5 / 10 s ~200 MB at 10 s 54.8 dB
decoder int8 53 / 109 ms per 5 / 10 s ~690 MB at 10 s 44.2 dB
encoder int8 0.38 / 0.84 / 1.97 s per 5 / 10 / 20 s 420 / 829 MB at 10 / 20 s 99.51 / 99.10 / 96.63%
encoder int4 0.32 / 0.75 / 1.86 s per 5 / 10 / 20 s 420 / 860 MB at 10 / 20 s 99.07 / 98.31 / 94.57%

The decoder's int8 file is half the size of fp16 but takes more memory once loaded, as Core ML expands its weights on the GPU. The encoder runs on the CPU for the same reason: on the GPU, Core ML holds 1.4 to 2.6 GB of expanded fp32 weights, while on the CPU it maps them from the file.

Downloads last month
42
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for software-mansion/higgs-audio-v2-tokenizer-Reverb

Quantized
(3)
this model

Space using software-mansion/higgs-audio-v2-tokenizer-Reverb 1