Jay Gopalan
@gopalanj
Publications
By @gopalanj
0 starsUpdated
Claims 5
Teaching an Open Speech-to-Speech Model to Take Bookings
- Fine-tuning MiniCPM-o 4.5's language model with a LoRA on synthetic spoken calls transfers to unseen entities and an unseen business: on held-out cities and clinics, exact tool use rises from 25% to 67%, on an unseen car service centre from 25% to 79%, unconsented bookings or cancellations fall from 71 to 1 on the held-out test, and all 10 emergency calls are transferred at once.Verified
- After round 2 training, the primary remaining bottleneck is hearing rather than tool selection: the model picks the right tool on 94% of held-out turns but hears names, numbers, and codes right on 72%; when given the same calls as typed text, it gets 99% of those details and handles 92% of turns exactly, and training the audio projector does not measurably improve hearing.Verified
- Most of the measured loss in general tool use on VoiceAgentBench is due to the benchmark's prompt and call format: under VoiceAgentBench's prompt, single-tool schema-valid calls fall from 80% (base) to 23% (round 1) and 5% (round 2), whereas under the deployed prompt, round 1 writes a valid call for 99% of requests that need a tool.Verified
- The remaining loss in general tool use on VoiceAgentBench (round 2 calling on 82% of requests and asking back on 16% under the deployed prompt) traces to a confound in public tool data: When2Call's supervised split contains no calls, teaching the model to ask or decline on When2Call-styled requests; training without the generic tool data removes the asking-back effect entirely (100% calls, 0% questions).Verified
- On Full-Duplex-Bench v3 (100 recorded human requests with unseen tools, evaluated turn-based under the deployed prompt), fine-tuning costs nothing measurable: base, round 1, and round 2 all pass about half of requests (45-46% exact, 51-55% written rules) with tool-selection F1 of 91-95%.Verified