litert-community/Basic-Pitch-LiteRT

Basic Pitch — LiteRT (CompiledModel GPU) music transcription

Audio input → note posteriorgram (MIDI pitch × time) from the on-device model.

Basic Pitch (Spotify, ICASSP 2022, Apache-2.0) re-authored to a GPU-native LiteRT .tflite — including the conv-based CQT front-end (9 octaves, shared 36×256 kernel banks, lowpass downsample chain, no FFT). Bit-exact torch re-implementation of the official ONNX (corr 1.000000). FP32, 0.84 MB.

Pixel 8a (Tensor G3): 241/241 nodes LITERT_CL (1 partition), ~4.4 ms per 2 s window; note-event F1@0.5 0.98 vs the official model, per-frame argmax agreement 98%.

I/O

• Input [1, 43844] float32 — 2 s @ 22 050 Hz mono, [-1, 1] (official window; overlap windows by 7 680 samples and keep center frames when stitching).
• Outputs: contour [1,172,264], note [1,172,88], onset [1,172,88] — sigmoid posteriorgrams (~11.6 ms frames; note/onset bins = MIDI 21–108).

Minimal usage

import numpy as np, soundfile as sf
from ai_edge_litert.interpreter import Interpreter

wav, _ = sf.read("audio_22050.wav", dtype="float32")   # mono 22.05 kHz
x = np.zeros(43844, np.float32); n = min(len(wav), 43844); x[:n] = wav[:n]
it = Interpreter(model_path="basicpitch.tflite"); it.allocate_tensors()
it.set_tensor(it.get_input_details()[0]["index"], x[None]); it.invoke()
contour, note, onset = (it.get_tensor(o["index"])[0] for o in
                        sorted(it.get_output_details(), key=lambda o: o["index"]))
active = np.argwhere(note > 0.5)          # (frame, key); midi = key + 21, t = frame * 256/22050

Kotlin (Android, LiteRT CompiledModel GPU)

// implementation("com.google.ai.edge.litert:litert:2.1.5")
val model = CompiledModel.create(File(ctx.filesDir, "basicpitch.tflite").absolutePath,
    CompiledModel.Options(Accelerator.GPU), null)
val inBuf = model.createInputBuffers(); val outBuf = model.createOutputBuffers()
inBuf[0].writeFloat(window43844)          // 2 s @ 22.05 kHz mono
model.run(inBuf, outBuf)
val note = outBuf[1].readFloat()          // [172 * 88], frame-major; midi = bin + 21
val onset = outBuf[2].readFloat()         // onset-triggered decoding -> note events

Conversion

Extracted from the official nmp.onnx (102 constants; no TensorFlow). Reflect-pad → anti-diagonal-constant FULLY_CONNECTED; PACK → concat + static slices. Two fp16-on-GPU fixes, both exact: post-log clamp clamp(10·log10(p+1e-10), min=-100) (recovers log(0) from the fp16-flushed floor; desktop no-op) and the per-bin CQT norm folded into per-octave kernel copies (magnitude is linear in kernel scale). FP32 flatbuffer (fp16 weights cost ~0.005 corr on the tiny CQT kernels).

Upstream

spotify/basic-pitch (Apache-2.0). Please cite Bittner et al., A Lightweight Instrument-Agnostic Model for Polyphonic Note Transcription and Multipitch Estimation (ICASSP 2022).

Performance

Measured on a Pixel 8a (Tensor G3, Android 16) with the standard TFLite benchmark_model tool — 10 warm-up runs then 50 timed runs, reported as the tool's mean.

Runtime	Backend	Graph on GPU	Latency
LiteRT CompiledModel (LITERT_CL)	GPU	241 / 241	~4.4 ms
TFLite benchmark_model (TfLiteGpuDelegateV2)	GPU (OpenCL)	241 / 241	did not run

The two GPU rows are different runtimes, not a contradiction. The LITERT_CL figure is the one recorded when this model shipped, taken through LiteRT's own CompiledModel accelerator — the path the Kotlin sample app and the LiteRT API use. The TfLiteGpuDelegateV2 figure is the classic TFLite OpenCL delegate, measured with a tool anyone can download and re-run. They agree on how much of the graph the GPU takes; they disagree on speed, and the classic delegate is the slower of the two here. Read the TfLiteGpuDelegateV2 row as a reproducible floor, not as this model's speed on LiteRT.

Snapdragon NPU (Hexagon)

The GPU is faster: 248.2 ms against 4544.4 ms on the NPU, a factor of 18.31. The NPU still loads 17.65x faster (124 ms against 2194 ms).

backend	compiled	inference (median / min)	load
NPU (Hexagon v81)	on-device JIT	4544.4 ms / 4527.4 ms	124 ms
GPU (Adreno)	—	248.2 ms / 247.2 ms	2194 ms

Measured on a Samsung Galaxy S26 (Snapdragon 8 Elite Gen 5 / SM8850, Hexagon v81, Android 16) with LiteRT CompiledModel 2.2.0, one accelerator per process, 5 warm-up runs then N=50 timed runs, median reported. Every run held thermal status NONE throughout. Headroom 0.65–0.70, where 1.0 is the throttling threshold.

The NPU rows ran the published file unchanged. LiteRT compiled it for the Hexagon on the device at first load. That first compile took 34 s here. The load column above is the cached load every later run pays. Recipe and the runtime libraries it needs: NPU guide.

GPU wiring: GPU guide.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with the LiteRT benchmark_model tool from litert-cli-nightly 0.2.0.dev20260805: CPU inference (XNNPACK, 4 threads), 3 invocations per file of 10 warm-up plus 50 timed runs (the tool caps a phase at 150 s, so very slow graphs run fewer — the Runs column is the actual timed total). The latency is the median across invocations; the spread is the min–max over all timed runs. No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0).

File	Inference (median)	Spread (min–max)	Runs	Peak memory
basicpitch.tflite	24.0 ms	23.7–24.8 ms	150	182 MB