pipecat-ai/voice-reader

Pipecat Voice Reader (rc3)

⚠️ Experimental. This is a release candidate, published for trying out and feedback. Its outputs, names and calibration may change between releases, and its readings are cues for an LLM to weigh, not verdicts to act on alone. Pin a revision, and read the limitations below before relying on it.

A small CPU model that hears how a user sounds: tone of voice, sarcasm in the delivery, energy, laughter and speech rate. It is the voice half of a pair: it is made to run beside Pipecat's Tone Reader (PipecatToneReader), which reads the words. It reads the last 8 s of each user turn, once, when the user stops speaking, and Pipecat's VoiceReaderProcessor writes the reading as RCS perception at the head of the user's message, after the Tone Reader's:

([emotion happy=0.62, neutral=0.30],[sarcasm value=0.08],[tone frustrated=0.71, neutral=0.22],[arousal value=0.34],[laughter value=0.02],[speech_rate value=3.1],[sarcasm_voice value=0.63]) Oh, great.

What changed since rc2

• A steadier base. Two trainings of one recipe can differ a lot: rc2's own recipe at a second seed scored tone macro-F1 0.626 against rc2's 0.647 over four test sets. The rc3 encoder is the average of four trainings, with every head then retrained on that fixed encoder, so its quality does not rest on one good draw.
• A targeted round of sadness labels. Both judges rated 10,000 more Emolia clips, the ones the sad output ranked highest among clips no judge had heard. Both called 1,561 of them sad, and at least one called 3,886. These clips train the heads only; the four encoder trainings came before them.
• Better in conversation (base, rc2 → rc3). Sarcasm AUROC 0.921 → 0.924, and 0.862 → 0.870 on the second conversational test, with F1 at the threshold 0.571 → 0.579 and 0.169 → 0.200. The sad output's AUROC 0.755 → 0.761 in conversation and 0.928 → 0.933 on Emolia. Upset 0.869 → 0.875.
• Sad and frustrated read more readily, by default. Each variant's config.json now carries offsets that the runtime adds to those two tones (see tone_bias under Outputs). With them, base finds more of the sadness in conversation than rc2 did: sad recall 0.74 → 0.80 and 0.49 → 0.60 on the two conversational tests, at lower precision (0.52 → 0.48, 0.38 → 0.31). Tone macro-F1 there moves 0.754 → 0.741 and 0.623 → 0.616. Read without the offsets, rc3 is level with or above rc2 on all four: F1 0.757 and 0.626, sad recall 0.76 and 0.54.
• Acted and synthetic sadness close to rc2 again (base). Acted CREMA-D sad clips read as sad 27 of 34 times (rc2 30, rc3 without the offsets 24). Sad recall on EmoNet-Voice's synthetic clips is 0.43 (0.44, 0.35), and the mean over four sets 0.54 (0.54, 0.47).
• Slightly more sarcasm false alarms (base): 1.0 % → 1.3 % of sincere conversation turns at the threshold, with recall 0.47 → 0.51.
• tiny keeps rc2's weights. Its model files are identical to rc2's, and its config.json gains the default offsets (sad +0.5, frustrated +0.5). They lift its sad recall in conversation 0.65 → 0.70 and 0.29 → 0.40, with tone macro-F1 0.739 → 0.730 and 0.581 → 0.598. Two new tiny models trained this round came out level with it on the test sets and no better overall on our own recordings, so it stays.

Made to pair with the Tone Reader

The two readers split the work, and each declares its own signals so the LLM sees both side by side:

	Tone Reader (the words)	Voice Reader (the voice)
Reads	the final transcript, with the bot's previous line	the last 8 s of the user's audio
Writes	emotion (nine labels), sarcasm	tone (four labels), sarcasm_voice, arousal, laughter, speech_rate
Strong on	what the words say: emotion, and sarcasm the words give away	what the words don't: a flat or frustrated delivery, sadness behind neutral words, sarcasm said with sincere words

The voice reader never writes emotion or sarcasm. Its sarcasm_voice is a weak cue on its own, and its declared description tells the LLM to weigh it with the Tone Reader's sarcasm: the case it earns its place in is words that read sincere, said in a voice that isn't. For example, from our test clips, "He's the greatest, I'd say." said sarcastically reads as happy and 2 % sarcastic from the words, and 20 % (above the 6.6 % threshold) from the voice.

It does run without the Tone Reader: the LLM then gets tone instead of emotion, and no sarcasm from the words to weigh sarcasm_voice against, so we recommend running both.

Variants

This release carries two sizes of the same model, one folder each, read by the same runtime:

Folder	variant	Params	int8 ONNX	Read time, p50 (M1 Max, 4 / 2 / 1 threads)	1 thread, x86 server	Choose it for
voice-reader-base/	"base" (default)	27M	28.6 MB	28 / 49 / 92 ms	~90–170 ms	the best tone reading, and sadness in conversation
voice-reader-tiny/	"tiny"	13M	14.1 MB	13 / 23 / 42 ms	~45–70 ms	one-vCPU deployments: sarcasm about as well, half the CPU

The read starts when the VAD says the user stopped and runs while the speech-to-text finalises, so it rarely delays the transcription; the cost to watch is CPU at the end of each turn, beside the VAD, the turn detector and the reply. Give base two threads where the machine allows.

Each folder has model.int8.onnx (the default), model.onnx (fp32), config.json (calibration, threshold, output names) and MANIFEST.json (SHA-256 of every file, checked by the runtime on download). The config.json at the root is the release index: the variants, their folders and files. The runtime reads it first on every load.

Use

With the Tone Reader (pipecat-ai[tone]) before it, as intended:

from pipecat.processors.tone.reader import PipecatToneReader
from pipecat_voice.reader import VoiceReaderParams, VoiceReaderProcessor

pipeline = Pipeline([
    transport.input(),
    stt,
    PipecatToneReader(),  # the words: emotion, sarcasm
    VoiceReaderProcessor(params=VoiceReaderParams(revision="rc3", variant="base", threads=2)),  # the voice
    user_aggregator,
    llm,
    tts,
    transport.output(),
])

It goes after the STT (it needs the transcriptions, and the STT passes the audio on) and after the Tone Reader, whose signal declarations it merges with its own, so an LLM that keeps only the latest declaration still sees both. The runtime is numpy and onnxruntime; the model downloads from this repository at the pinned revision the first time.

Outputs

Signal	Published to the LLM	Meaning
tone	yes, top two	how the voice sounds: positive, neutral, sad, frustrated (temperature-calibrated, with the release's offsets on sad and frustrated)
sarcasm_voice	yes	the chance the delivery alone sounds sarcastic, Platt-calibrated on real conversation; below the release threshold (0.066 for base, 0.063 for tiny) it is published under 0.5, as a guess; dropped under laughter
arousal	yes	vocal energy, 0–1
laughter	yes	0–1
speech_rate	yes	syllables per second
upset	no, raw reading only	how clearly the speaker is upset, 0–1
sad	no, raw reading only	how likely listeners would hear sadness, 0–1

Nothing is written for a turn with under 0.8 s of speech. The raw reading, including upset and sad, reaches the application through the processor's on_reading event.

Default offsets and tone_bias. tone is calibrated on real conversation, where most turns are neutral, so on its own it leans neutral on a borderline turn: a sad or frustrated delivery a listener would notice can come out as neutral 0.45, sad 0.40. So each variant's config.json carries default_tone_bias, offsets the runtime adds to those tones' logits before the softmax: sad +0.5 and frustrated +0.25 for base, sad +0.5 and frustrated +0.5 for tiny. An offset of +0.5 multiplies a tone's odds by about 1.65. The published tone probabilities include them; the graded sad and upset readings do not. VoiceReaderParams(tone_bias={...}) replaces them, and tone_bias={} reads the calibrated probabilities. An older pipecat_voice runtime ignores the key and reads without them.

What the base offsets buy and cost on our test sets:

base	default offsets	tone_bias={}
Sad recall / precision, conversation (two tests)	0.80 / 0.48, 0.60 / 0.31	0.76 / 0.54, 0.54 / 0.40
Frustrated recall, second conversational test	0.59	0.54
Tone macro-F1, conversation (two tests)	0.741, 0.616	0.757, 0.626
Tone macro-F1, mean over four sets	0.659	0.651
Sad recall, mean over four sets	0.54	0.47
Acted CREMA-D sad clips read as sad	27 of 34	24 of 34
Open Yap turns whose top tone is sad / frustrated	1.1 % / 10.6 %	0.7 % / 9.0 %

Evaluation

Labels on real speech come from two audio LLM judges (Gemini 3.1 Pro preview and OpenAI gpt-audio-1.5) hearing each clip; a turn counts as sarcastic only when both say so (judge agreement on sarcasm: kappa 0.42). Test speakers never appear in training.

	base	tiny
Sarcasm, everyday conversation (3,945 turns both judges rated, 203 sarcastic; Open Yap, otoSpeech, HarperValley), AUROC	0.924	0.908
— at the release threshold: F1 / recall / false alarms on sincere turns	0.579 / 0.51 / 1.3 %	0.548 / 0.45 / 1.1 %
— second conversational test, held out from the rc2 judging round (5,037 turns, 272 sarcastic), AUROC	0.870	0.853
— mean AUROC over four test sets (two conversational, Emolia, MUStARD++)	0.803	0.797
— MUStARD++ (TV, audio only, evaluation only), AUROC	0.682	0.662
Tone macro-F1, with the default offsets: conversation / Emolia / CREMA-D (acted) / EmoNet-Voice (synthetic)	0.741 / 0.676 / 0.909 / 0.527	0.730 / 0.638 / 0.867 / 0.479
— without the offsets (tone_bias={})	0.757 / 0.683 / 0.894 / 0.512	0.739 / 0.647 / 0.873 / 0.477
Sadness: tone recall for sad, mean of four sets, with the offsets / without	0.54 / 0.47	0.48 / 0.45
— sad output, AUROC: conversation (any judge heard sadness) / Emolia	0.761 / 0.933	0.740 / 0.923
Upset, mean AUROC (three sets)	0.875	0.857
Arousal, CCC against the training target	0.90	0.88
Laughter, AUROC / F1	0.993 / 0.66	0.991 / 0.64
int8 vs fp32: tone agreement (without the offsets) / sarcasm p95 |Δp|	0.988 / 0.007	0.983 / 0.009

Limitations

• Sarcasm from the voice alone is a weak cue. At the threshold it catches about half of the sarcasm both judges heard, at about 1 false alarm in 80 sincere turns. Weigh it with the words, never alone. On synthetic (TTS) sarcasm it is weak (AUROC 0.62 for base, 0.56 for tiny, on our 20 demo clips).
• Sadness in conversation is rare (2–6 % of turns), so even a good detector's per-turn flags are mostly wrong (14–26 % precision at a 5 % false-alarm rate). Treat sad as a graded cue or average it over a conversation. It is not a measure of depression or of anyone's mental health, and a single turn cannot show either.
• Borderline tones still lean neutral. Calibrated on conversation, the tone head would publish neutral on many turns a listener might call sad or frustrated. The default offsets pull those two tones forward, at the cost of more false calls: in conversation half or more of base's sad calls are turns the judges agreed on as another tone. tone_bias above is the knob both ways. Acted sadness still reads a little less readily than in rc2: 27 of 34 CREMA-D sad clips by base, 28 by tiny, against rc2's 30. base's misses read as neutral or frustrated.
• Laughter ranks well but its published value is conservative: F1 0.66 at the calibrated 0.5.
• English only; trained on conversational and podcast speech, 8 kHz telephone audio included.
• Tone labels are LLM judgments of the audio, not human ratings.

Training

• Encoder (base): the average of four Whisper encoders, each initialised from LAION's affect-tuned laion/whisper-tiny-base-emotion-voice-burst (CC BY 4.0) and cut to an 8 s window. The four trainings use one recipe and differ in seed or pseudo-labelled set. Only the encoders are averaged, since each training's heads start from different random weights.
• Heads: masked statistics pooling and one small head per output, every head retrained for 3 epochs on the fixed averaged encoder.
• Data for the heads (about 296k training windows): about 137k judged windows from Open Yap 1K and otoSpeech conversations (two judging rounds), HarperValleyBank calls (CC BY 4.0), the English YODAS part of Emilia from laion/Emolia (CC BY 4.0, the sadness round included) and CREMA-D (ODbL / DbCL); plus 159k windows pseudo-labelled by the rc1 base model, 112,996 Emolia clips no judge has rated and 46,021 conversation turns. Augmented with gain, noise, telephone band-limiting and codecs; never speed or pitch, since rate and pitch are the signal.
• Data for the encoders: the same recipe without the sadness round. Two of the four trainings used larger pseudo-labelled sets read by the rc2 base model, up to 277k windows.
• Labels: the two judges' verdicts on sarcasm, tone and upset; where they disagreed on tone, their vote share over the four tones; the sad head on the share of judges who heard sadness. No audio emotion model's output is a direct target in rc3. The pseudo-labelled windows carry the rc1 and rc2 base models' readings, and rc1 did use two such models (emotion2vec+ large and the LAION model) for its disagreed turns.
• Tiny: unchanged from rc2, distilled from the rc2 base model's predictions at half weight, with a LAION tiny encoder.
• Calibration on real conversational dev turns: temperature for tone, Platt for sarcasm and laughter; the sarcasm threshold keeps false alarms at 2 % of sincere dev turns.

Evaluation-only data, never trained on: MUStARD++ (CC BY-NC) and EmoNet-Voice Bench's clearly rated clips.