robot head voice server

#what it is

a small company builds a robot head on an ESP32 board: servos for the face, a speaker and a microphone. the firmware speaks an open-source websocket protocol, and out of the box it sends every conversation to the board vendor's cloud. this server replaces that cloud. the head connects directly and streams opus both ways. motion stays with the vendor firmware; the conversation is mine.

each head gets an engine from an admin page: Gemini Live, OpenAI Realtime, a free pipeline (Groq Whisper for speech, an open model on Groq for the reply, edge-tts for the voice), or a hand-back to the vendor's own cloud. it was multi-tenant from the first commit.

#the hard parts

most of this was finding out what the firmware really does, from its source and from recordings of real sessions.

audio sent at the wrong moment just disappears. the firmware only plays incoming audio while it is in its "speaking" state. in any other state the frame is read off the socket and dropped, with no error on either side. the vendor doc doesn't say this; the firmware source does. the server's only lever over that state is the tts start message, so every answer goes in a fixed order: start, frames, drain, stop.

the head was answering itself, and my first diagnosis was wrong. on the first real board, answers kept stopping. a capture showed a loud motor hum in the mic, so i called that the cause. the customer said the head was hearing its own voice, and they were right. 89 session bundles split cleanly by mic mode: in the mode where the mic stays open during playback, 75 self-interruptions in 882 seconds of speech. in the mode where the firmware closes it, 0 in 168 seconds. the hum came from one clip with a noise floor 17 times the median. the fix stops relaying the mic while the head speaks.

that fix swallowed people who talked over the head. the next day: the head stops when you interrupt, but never answers what you said. the gate was throwing the overlap away. now it holds the last 1.6 seconds and releases them once an interruption is confirmed, so the model hears the whole sentence. on the board that confirmation is its own abort message. on the phone test page, a detector measures speaker-to-mic echo per session from the audio we sent. it starts strict and only relaxes on evidence, so a mistake means a missed interruption, never the head cutting itself off.

the board hung up before the answer arrived. in one session the board closed its socket 2,283ms after the question reached the model. replaying the same bytes, the answer took 2,883 to 3,607ms. every local hang-up path in the firmware only runs from the listening state. so the server now puts the head into speaking as soon as the user stops talking, and holds it there until the answer exists.

newer firmware could do full duplex, and that flipped the problem. a later firmware cancels echo on the device (mic echo at the 99th percentile: 272, against 2,461 on the old board). on those heads the gate goes away, and now coughs, a TV or a voice across the room cut the answer. my first rule, 5 loud frames in a 7-frame window, was beaten by two noise bursts plus a quiet voice. now the head pauses on the first voiced frames, which is reversible, and only cuts after an unbroken run of 4 (240ms). end of speech needed more patience too: 12 of one user's short pauses fell between 350 and 700ms, so the model answered half sentences. the wait went from 350ms to 550, then to 700 or 750 depending on the engine.

#what else is in there

• voice choice: an AI listening panel ranking 30 voices couldn't reproduce its own order across 4 runs, so the shortlist is built from measured pitch range instead.
• the free engine has a server-side referee for the korean word-chain game. it repairs a misheard word to a valid one only within 2 letter edits; allowing 3 once rewrote a correctly heard word.
• over-the-air firmware updates, pushed per device through the server's provisioning endpoint, which is also how a head learns where to connect.
• a jitter buffer that waits for 5 frames (300ms) before playing and grows up to 10 when the model stalls, after speech got choppy on both engines.
• a phone test page that speaks the same protocol as a head, a knowledge base, memory shared across engines, and a remote for video shoots.
• one Docker container behind Caddy, SQLite through Prisma. deploys wait until nobody is mid-conversation.

#where it is now

the live database has 968 sessions, 21 hours of talk and 3,179 turns. 528 of those sessions came from the customer's 4 real heads. since late october it runs on the customer's own model keys. the last commit is from october 29, and the work is ongoing.