The brief
A voice-interactive companion device: you talk to it, it answers, with a personality driven by an LLM. Not a phone app pretending to be a product, but a physical device that has to boot, connect, listen, and reply on its own.
One requirement was non-negotiable from day one: every byte of user data, voice recordings, transcripts, and model calls included, had to stay inside the EU.
What we built
The whole stack shipped from one team. On the device, an ESP32 captures audio and streams it over a persistent WebSocket connection. A Laravel backend orchestrates the conversation loop: speech-to-text, the LLM turn, then synthesized speech streamed back to the device.
A Flutter companion app pairs with the device for setup and management, riding the same real-time backbone as the device itself.
Speech runs on Azure STT/TTS and the language model on Google Vertex AI. Both were selected region by region, so the pipeline stays inside the EU without resorting to exotic self-hosting.
The hard parts
Voice devices live or die on latency. A request/response API feels like a walkie-talkie, so everything is streamed: audio up, synthesized speech back down, over a single full-duplex WebSocket.
Microcontrollers are unforgiving. Reconnect logic, buffering, and failure states all had to be designed for a device sitting on a shelf with consumer Wi-Fi, not for a data center.
Key decisions
Managed AI, pinned by region
Self-hosting models would have guaranteed residency but ballooned scope. Azure Speech and Vertex AI both offer EU regions, so we got managed reliability and EU residency at the same time.
WebSockets over request/response
A conversation is a stream, not a form submission. One persistent connection carries audio both ways and keeps round trips low enough to feel live.
One team, whole stack
Firmware, app, backend, and AI pipeline were built together. When the interface between two layers had to change, it changed the same day, with no cross-vendor negotiation.
Outcome
- The full stack shipped in-house: ESP32 firmware, Flutter app, Laravel backend.
- End-to-end EU data residency, from microphone to model and back.
- A real-time voice loop over WebSockets built for consumer Wi-Fi.
Common questions
- What does it take to build a custom AI voice assistant device?
- Four layers that have to ship together: device firmware (here an ESP32 that captures and streams audio), a backend that orchestrates speech-to-text, the LLM turn, and text-to-speech (Laravel), a companion mobile app for setup and management (Flutter), and the AI pipeline itself. We built all four in-house, so when an interface between two layers had to change, it changed the same day.
- Can a voice AI product be fully EU data resident?
- Yes, and without self-hosting models. Azure Speech and Google Vertex AI both offer EU regions, so speech recognition, the language model, and speech synthesis can all be pinned to EU data centers. Voice recordings, transcripts, and model calls never leave the EU.
- How do you keep voice AI latency low enough to feel conversational?
- Stream everything. A request/response API feels like a walkie-talkie, so the device holds a single full-duplex WebSocket connection: audio streams up while synthesized speech streams back down, keeping round trips low enough to feel live.