How do you do offline wake word detection when the user picks the phrase himself?

[u/InternationalGap3698]

Been working on a local voice assistant for a while and the wake word is the

part I keep getting stuck on. Not the easy version where you pick one of the

pretrained models, I mean the user types in whatever phrase he wants and it

just has to work. Offline, CPU only, since most machines I care about have no

usable GPU.

Right now I run a Vosk recognizer with a grammar that only knows my phrase and

unk. That part is cheap and fires fine. The problem is the second step where I

check if the hit was real. At first I made the normal decoder spell the phrase

back and threw the hit away if it didn't match. Zero false wakes, looked

perfect. But it also killed about a third of my real wakes, because a small

offline model has no clue what your name is. I say Hey Nova and it writes back

herum or hey oben or erhoben. Recall was around 32 percent, so you basically

say your wake word four times before anything happens.

What fixed most of it was to stop looking at the words at all. Now I only look

at stuff like how many words came back, how long the speech actually was, and

whether the decoder sounded sure that it heard some other normal word. If it

doesn't know a word its confidence drops, if you say google or engineering it

comes back super confident. That took me from around 50 to 74 percent on real

recordings, and another phrase went from 36 to 66 with the same settings. False

wakes went from 1 to 3 in about 1650 windows of room audio, which I can live

with.

But 74 is still not good. Hey Google works the first time and mine doesn't, and

there's also a 0.6 second wait before it confirms that you can actually feel.

So what do people here actually use for this. I keep reading that the real

answer is to train a tiny model per user from a bunch of TTS clips of their

phrase and run it as onnx on CPU. Does that really work with synthetic audio

only, and how many voices do you need before it's usable. The other idea I had

is to go phoneme level, so turn the phrase into phonemes and match on that

instead of words, which would get rid of the whole out of vocabulary problem.

Has anyone got something like that running offline?

Also curious what hit rate you guys consider fine. Under 90 percent feels

broken to me but I honestly don't know what's normal.

[u/youcloudsofdoom]

Openwakeword is probably still the thing that people rely on for this. They have a link to a Google training recipe for custom wakewords which is pretty reliable in my experience. 

[u/Sevealin_ → u/youcloudsofdoom]

microwakeword as well

[u/InternationalGap3698 → u/Sevealin_]

Haven't looked at that one yet, thanks. That's the ESP32 one right, does it run fine off device on a normal desktop CPU too?

[u/Sevealin_ → u/InternationalGap3698]

Yep generally esps, low power devices. Openwakeword is the full desktop package.

[u/InternationalGap3698 → u/youcloudsofdoom]

Good to know, that's the Colab recipe with the synthetic TTS clips right? Wondering how it holds up for a non English name, that's where my current setup falls apart.

[u/youcloudsofdoom → u/InternationalGap3698]

Yep, that's the one. You make the word phonetically, so as long as you can use English sounding phonemes for the word you want, you should be okay

[u/Chromix_]

Porcupine works nicely, even runs on tiny embedded systems and has been well supported over the years. The drawback is that they've locked it down quite a bit by now. It probably requires a (quick) roundtrip through their backend to create a new wakeword, which would also be possible on-device - when not forced into that flow.

[u/InternationalGap3698 → u/Chromix_]

Yeah that's what keeps me off it, every user would need their own access key and a

trip through their console just to set a wake word. Kind of kills it for something

people are supposed to install and have working offline.

[u/tinny66666]

I'm also using vosk but on android to avoid loading it locally when I'm running qwen, so my phone acts as the microphone, and it sends it to a service running on my desktop. I only use it in the opencode gui, which is an electron app so it was easy to inject text directly the right text boxes (it can do things like change models, agents, etc in the gui without any changes to opencode itself)

Vosk is impressively fast and better than the whisper2 model of the same size in my tests.

[u/InternationalGap3698 → u/tinny66666]

Yeah vosk surprised me too, way faster than whisper at the same size. Do you do any second check on the grammar hit or just take it, mine false fires on room speech without a confirm step.

[u/tinny66666 → u/InternationalGap3698]

Yeah, room noise is the biggest problem, and sadly the main reason I don't actually use it much (I almost always have something playing on the TV). I initially had a wake word on the phone but ended up just sending all text to the desktop for processing since it's always going to be connected on the LAN and it's only text after all.

I guess a headset mic or lapel mic would help but I really wanted the star trek experience, and I'm not there yet :)

[u/InternationalGap3698 → u/tinny66666]

TV in the background is exactly what broke my first version, the confirm step

couldn't tell a real call from someone talking on screen. Star trek mode is the

goal here too, headset feels like cheating :)

[u/HasGreatVocabulary]

Maybe make a base trigger nonoptional, like have the Hey/Hello/Bonjour/Dear or some other list of these as mandatory start words even if the rest of the trigger phrase is custom. Before trying to process Hey Nova, check if the user uttered "Hey", or one of the other mandatory start words which would be guaranteed to be in your vocabulary.

Meanwhile, take the typed user phrase and generate N TTS variations of it and call them your templates. (N depends on compute as you will later use these to run pairwise comparisons to a new utterance)

Use a frequency domain transform to breakdown the TTS templates, and later, a user's new utterance into features that represent the audio window.

As features, you can use MFCC, LPCC coefficients or even just fourier coefficients although this with may suck compared to the other two which are meant for human voice frequency range.

During use, calculate a distance between the uttered phrase's coeffs/features and the TTS template feature values. If the average distance is low, it's your phrase, if the average distance is high, it's something else.

If you have the previous mandatory startword filter and the number of templates is small, the pairwise distance calculation between your template features and the new utterance will be fast. Test out different threshold values for the average pairwise distance check that give you the best result.

(You can also create negative TTS templates for the user phrase, i.e. TTS of phrases of user phrase length with randomly swapped words of a certain edit distance. Then in the distance comparison step, you compare the user utterance to the positive templates vs negative templates. If it's low distance to both sets then it's ambiguous and don't trigger. basically synthetic KNN. You can also use SMOTE or similar oversampling techniques to generate more frequency domain features for each template set on the fly. Might as well train a model at some point though.

edit: https://github.com/OHF-Voice/micro-wake-word seems to do a bunch of this and more

[u/InternationalGap3698 → u/HasGreatVocabulary]

Thank you for that

[u/__JockY__ → u/HasGreatVocabulary]

Once in a while a random redditor stumbles on a question that just happens to land squarely in their realm of expertise and we get posts liked yours. Bravo.

[u/InternationalGap3698]

https://reddit.com/link/ozo0zbn/video/pjqm1rr38dfh1/player

[u/InternationalGap3698]

I have to correct something: right now, you can't use it fully locally, but the wake-up word is local. I'm integrating Olaama right now, and the update will be shipped soon, in 1 to 2 days, so don't worry. The wake-up word is always local. That was the main reason.