Remember me
I've had several run-ins with voice assistants in the past. Google Assistant, Cortana, Alexa, Bixby, Siri, I've used them all, and I've had the same complaint with every single one of them: why do they need to make a round trip to a third-party server to turn a light on in my own house?
Apart from the glaring privacy concern, this behavior has the added side effect of making the assistants respond more slowly. So now that I'm supercharging my smart home with Home Assistant, I decided to spend the weekend creating my own voice assistant, and it responds faster than most voice assistants ever could.
Related
The hardware is cheaper than you'd think
An ESP32, a microphone, and a speaker are all it takes to build a local voice assistant
Image taken by Yadullah Abidi | No attribution required.
Image taken by Yadullah Abidi | No attribution required.
Image taken by Yadullah Abidi | No attribution required.
Image taken by Yadullah Abidi | No attribution required.
Image taken by Yadullah Abidi | No attribution required.Close
Image taken by Yadullah Abidi | No attribution required.
Image taken by Yadullah Abidi | No attribution required.
Image taken by Yadullah Abidi | No attribution required.
Image taken by Yadullah Abidi | No attribution required.
Image taken by Yadullah Abidi | No attribution required.
As is the case with nearly every custom smart home device I've made, the voice assistant too centers around an ESP32, the ESP32-S3 N8R8 to be specific. I had one lying around from a previous project I decided to decommission, but you can find a new one for as low as $5. The ESP32 acts as a satellite that captures the wake word and sends the audio to a local LLM running via Ollama for processing. It's important to choose the right ESP32 here, as not all of them will support this task.
Connected to the ESP32 is an INMP441 I2S mic and a MAX98357A I2S amplifier board. For the speaker, I salvaged a small 4Ω speaker from a broken Bluetooth speaker, and since I wanted some visual feedback for my voice assistant, I added an 8x8 WS2812B LED matrix. The exact cost of the parts will vary based on where you are, but it shouldn't be more than $15.
The wiring is also rather simple, or at least it seemed to be. In theory, you hook up everything to your ESP32, flash an ESPHome config, and call it a day. In practice, I realized that the LED matrix and amp board would require external power that the ESP32 couldn't provide. The LED matrix can end up drawing anywhere from 1.5 to 2 amps of current, which is just not feasible for the ESP32 and affected the amp as well, so both these components were wired to a separate 5V rail going to a 20W USB-C adapter. The mic module plugs straight into the ESP32 with no issues.
Everything happens on your own network Home Assistant, ESPHome, and local AI eliminate the need for cloud processing
Screenshot taken by Yadullah Abidi | No attribution required.
As mentioned before, the satellite runs ESPHome firmware and talks to my Home Assistant instance over an encrypted local API. From there, the built-in assist pipeline takes over with three components of its own, all self-hosted on my home server. My home server is a repurposed laptop running an 8-core Intel i5 with 16GB RAM and an Nvidia GTX 1650 Ti mobile with 4GB VRAM. It's not top-of-the-line hardware by any means, but it's enough to run a local LLM smart enough to process queries and control my smart home.
Speech-to-text runs on Wyoming faster-whisper, using the small-int8 model on CPU. Text-to-speech runs on Wyoming Piper with a natural-sounding American English voice. Both services sit in their own Python virtual environment on my home server, wrapped in systemd units so they start at boot and recover automatically if they crash.
Related
5 useful things I do with a local LLM on my phone
Privacy aside, a local LLM is just really convenient.
Picking the right LLM for the job that could run comfortably on my hardware was the real time hog, and if you're building something similar, I recommend you spend time here too. It started with Llama3.2:3b just to get everything working. It was fast, but couldn't handle tool calls reliably, often saying something and doing something else entirely.
I then tried Qwen 2.5 7B Instruct, which was too large to fit fully in VRAM. This model was smart, but the response time was far too long, extending up to several minutes on a cold cache. I'm currently juggling between Gemma4:e2b and Qwen 3.5 4B. Both models are smart enough to understand requests and call the right tools while also responding within seconds on a warm cache.
Getting the wake word right is surprisingly hard Sticking with the defaults is not as bad as it sounds
Image taken by Yadullah Abidi | No attribution required.
By default, microWakeWord, the on-device wake word engine, is tuned for three wake words: Hey Nabu, Hey Jarvis, and Hey Mycroft. However, out of the three, I could only get Hey Nabu to work reliably. I wasted a ton of time trying to get the Hey Jarvis and Hey Mycroft wake words to work, but the device kept going deaf after one successful detection per boot.
I do plan to train the model for a custom wake word, but that's a battle for another day. The wake phrase for now is Hey Nabu (or Okay Nabu). It triggers cyan rings expanding from the center of the LED matrix when listening, purple comet orbits when thinking, and green equalizer bars while responding. It's completely necessary, and I love it. If I could run a more powerful model, I would rather hand over control of the LED matrix to the model itself for it to emote how it sees fit.
It feels different in all the right ways Faster responses, greater privacy, and complete control over how your assistant behavesOn a warm cache, meaning the LLM is already loaded into VRAM, the response time from the end of speech to the start of the spoken answer is roughly two seconds. I say a command, and it does the job. No cloud, no wait, and no data leaving the house. And since I've got everything from smart lights to my 3D printer connected to Home Assistant, I can ask my voice assistant about almost everything in the house and get a response.
Related
On top of that, the whole system can run entirely offline. The mic audio never leaves the ESP32 until the wake word fires, and even then, it streams only to Home Assistant and my home server, both on my local network. Is it perfect? Not yet. The LLM could be smarter (much smarter if I decide to use cloud-based LLMs). There's also a quirk where streaming TTS causes the model's intermediate text to be spoken aloud before the final answer, which has produced some entertainingly confused responses.
But for $15 in hardware, running on a laptop from seven years ago acting as a home server, with no cloud subscription and response times snappier than any commercial voice assistant — I'd take it any day.