The Free Encyclopedia
Log in
Article History

The Fully-Local Voice Stack

Revision as of Jun 25, 2026 22:26 by migration. This is an old revision — view current.

This is the real loop behind theLAB's ADI voice interface: say "hey Addie," talk, and get a spoken reply — with nothing leaving the tailnet. Four stages chain together, and three of them run on local hardware (only optional cloud models ever reach out). Wake detection and ASR live on apexlabs-voicebox, the LLM runs on adi-cortex, and TTS comes back from the same voicebox machine.

Every host below is reachable over a private Tailscale network, so the entire round-trip — audio in, transcript, generation, audio out — stays on machines we own.

# the ADI voice round-trip, all over the tailnet
 browser mic (16 kHz mono Int16 PCM, 80 ms frames)
        │  ws ── 1280-sample binary frames
        ▼
┌──────────────────────────────────────────────┐
│  apexlabs-voicebox                            │
│   [1] openWakeWord  ws://<host>:9293/ws        │  hey_addie.onnx
│        └─ score ≥ 0.5 → WAKE                   │
│   [2] capture → silence → Parakeet ASR :9292   │  {text}
│   [4] OmniVoice TTS :8881  (sha256 cache)      │  wav bytes
└───────────────┬──────────────────────────────┘
                │  transcript          ▲ reply text
                ▼                      │
        [3] Ollama  adi-cortex 100.64.87.104:11434

01Wake word with openWakeWord

The browser captures mic audio as 16 kHz mono Int16 PCM and streams it as 1280-sample (80 ms) binary frames over a WebSocket to the wake server at ws://<host>:9293/ws on apexlabs-voicebox. Each socket gets its own stateful openWakeWord instance — state is per-connection, so two clients never bleed scores into each other. We run the hey_addie.onnx model; every frame is scored and compared against WAKE_THRESHOLD.

# one stateful oww per WebSocket connection
oww = Model(wakeword_models=["hey_addie.onnx"])
WAKE_THRESHOLD = 0.5

# for each 1280-sample (80 ms) Int16 frame off the socket:
score = oww.predict(frame)["hey_addie"]
if score >= WAKE_THRESHOLD:
    fire_wake()          # start capturing the utterance
Why 80 ms frames Small 1280-sample frames keep wake detection snappy — openWakeWord scores on every frame, so the gap between speaking the word and firing stays well under a second. See Training a Custom Wake Word with openWakeWord for how hey_addie.onnx was built.

02Capture and transcribe with Parakeet

Once the wake fires, the server records the utterance until it hears quiet: SILENCE_FRAMES_NEEDED=12 frames of silence (≈ 960 ms) ends the capture. A COOLDOWN_FRAMES=25 window (≈ 2 s) then blocks a re-fire so one "hey Addie" can't trigger twice. The captured WAV goes to Parakeet (NVIDIA NeMo), exposed on an OpenAI-compatible endpoint.

files = {"file": ("utterance.wav", wav_bytes, "audio/wav")}
data  = {"response_format": "json"}
r = requests.post(
    "http://<host>:9292/v1/audio/transcriptions",
    files=files, data=data, timeout=60,
)
transcript = r.json()["text"]
Parakeet over Whisper We picked Parakeet specifically for lower latency on English — it turns the WAV around faster than Whisper at comparable accuracy for our use, which matters when a human is waiting mid-conversation. Setup details in Parakeet ASR Setup.

03Generate with Ollama

The transcript becomes a user turn sent to Ollama on the primary host adi-cortex over the tailnet. This is a standard /api/chat call — any ADI model can answer, including the distilled local models, with optional cloud models the only thing that ever leaves the network.

payload = {
    "model": "adi-qwen3.5-4b-glm5.2-general",
    "messages": [
        {"role": "system", "content": ADI_SYSTEM},
        {"role": "user",   "content": transcript},
    ],
    "stream": False,
}
r = requests.post("http://100.64.87.104:11434/api/chat",
                  json=payload, timeout=300)
reply = r.json()["message"]["content"]

Because the reply is what gets spoken, this is also where tool calls and structured outputs slot in — see Tool Calling & Structured Outputs.

04Speak with OmniVoice TTS

The reply text goes to OmniVoice (the VibeVoice / Qwen3-TTS family) back on apexlabs-voicebox, also OpenAI-compatible. It returns raw WAV bytes the browser plays. Crucially, OmniVoice caches audio on disk keyed by a sha256 of the request, so any repeated phrase — "Sure, done." "I didn't catch that." — comes back instantly from cache instead of re-synthesizing.

payload = {
    "model":  "omnivoice",
    "voice":  "adi_ref",        # the ADI reference voice
    "input":  reply,
    "format": "wav",
}
r = requests.post("http://<host>:8881/v1/audio/speech",
                  json=payload, timeout=120)
audio_bytes = r.content         # sha256(request) cache → repeats are instant
Where the latency goes Three tricks keep the loop feeling live: 80 ms wake frames make detection feel immediate, Parakeet shaves ASR time over Whisper, and the sha256 TTS cache eliminates synthesis time on any phrase ADI has said before. Full TTS build in OmniVoice TTS Setup.

Putting it together

  • Wake and ASR (:9293, :9292) and TTS (:8881) all live on apexlabs-voicebox — one box owns audio in and audio out.
  • Only the LLM hop leaves the voicebox, going to adi-cortex at 100.64.87.104:11434 over Tailscale.
  • Nothing but optional cloud models ever leaves local hardware — the mic audio, the transcript, and the synthesized voice never touch a third-party API.