By @gopalanjSourceOpenVersions
Fork
Clone
git clone https://openversions.com/git/gopalanj/teaching-an-open-speech-to-speech-model-to-take-bookings.git
Cite
Version link
https://openversions.com/gopalanj/teaching-an-open-speech-to-speech-model-to-take-bookings/commit/d9d187dcb741dfd5fc4a062e0fea8ef85f0f084a
Version identifier (commit hash)
d9d187dcb741dfd5fc4a062e0fea8ef85f0f084a

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

First version

Published

Claims

Findings

Fine-tuning MiniCPM-o 4.5's language model with a LoRA on synthetic spoken calls transfers to unseen entities and an unseen business: on held-out cities and clinics, exact tool use rises from 25% to 67%, on an unseen car service centre from 25% to 79%, unconsented bookings or cancellations fall from 71 to 1 on the held-out test, and all 10 emergency calls are transferred at once.

After round 2 training, the primary remaining bottleneck is hearing rather than tool selection: the model picks the right tool on 94% of held-out turns but hears names, numbers, and codes right on 72%; when given the same calls as typed text, it gets 99% of those details and handles 92% of turns exactly, and training the audio projector does not measurably improve hearing.

Most of the measured loss in general tool use on VoiceAgentBench is due to the benchmark's prompt and call format: under VoiceAgentBench's prompt, single-tool schema-valid calls fall from 80% (base) to 23% (round 1) and 5% (round 2), whereas under the deployed prompt, round 1 writes a valid call for 99% of requests that need a tool.

The remaining loss in general tool use on VoiceAgentBench (round 2 calling on 82% of requests and asking back on 16% under the deployed prompt) traces to a confound in public tool data: When2Call's supervised split contains no calls, teaching the model to ask or decline on When2Call-styled requests; training without the generic tool data removes the asking-back effect entirely (100% calls, 0% questions).

On Full-Duplex-Bench v3 (100 recorded human requests with unseen tools, evaluated turn-based under the deployed prompt), fine-tuning costs nothing measurable: base, round 1, and round 2 all pass about half of requests (45-46% exact, 51-55% written rules) with tool-selection F1 of 91-95%.

Methods

Synthetic business call conversations are generated using Python templates with deterministic hash-seeded mock tools and filtered by a rule-based validator enforcing tool schemas, read-back confirmation before booking/cancelling/rescheduling, emergency transfer, and spoken-style constraints.

Source excerpts

“Conversations are produced by Python templates, not by a language model. Each samples a business, a caller, entities and a path through the call” · Show in paper

“Every training conversation must pass rules that encode the task: calls fit their schemas; every booking, cancellation or reschedule is preceded by a read-back question and a caller’s yes, and its arguments match what was read back” · Show in paper

Caller turns are synthesized into 16 kHz mono speech using Indic Parler-TTS and Kokoro-82M with added noise, room reverberation, or a telephone band on 45-60% of clips, while agent turns, tool calls, and tool results remain text.

Source excerpts

“Only the caller’s turns are spoken. Each conversation gets one voice, from Indic Parler-TTS [19] (21 Indian-English speakers, with style and room described in text) or Kokoro-82M [8] (US and UK English voices, speed” · Show in paper

“Clips are 16 kHz mono with added noise, room reverberation or a telephone band on 45–60% of them, and numbers are written out as words before synthesis. The agent’s turns, tool calls and tool results stay text” · Show in paper

A LoRA of rank 16 and alpha 32 (43.6M parameters) is trained on every linear layer of MiniCPM-o 4.5's Qwen3-8B language model using ms-swift for two epochs on one H100 80 GB GPU, keeping the Whisper encoder, projector, and CosyVoice2 speech decoder frozen, then merged and quantized to Q8_0 GGUF for llama.cpp serving.

Source excerpts

“We train a LoRA of rank 16 and α = 32 on every linear layer of the language model (43.6M parameters) with ms-swift [22], keeping the Whisper encoder, its projector and the speech decoder frozen.” · Show in paper

“For serving, the LoRA is merged, the language model is quantized to 8 bits (Q8_0 GGUF), and calls run in a llama.cpp [7] build for MiniCPM-o.” · Show in paper

Models are evaluated turn-by-turn on held-out synthetic business tests, typed-text replays (textcalls.py), VoiceAgentBench (under both the benchmark prompt and deployed prompt), and Full-Duplex-Bench v3.

Source excerpts

“Each test conversation is replayed against the model one caller turn at a time, with the caller’s recorded audio as input.” · Show in paper

“textcalls.py replays each test conversation through the same dialogue driver with the caller’s words typed rather than spoken, so a text score and a voice score differ only in how the caller is heard.” · Show in paper

“Each request is run with the benchmark’s own prompt, which asks for its own call format, and, for the base model, round 1, r2 and r2 without the generic data, also with the prompt the agent is deployed with” · Show in paper