Whisper STT in VibeFuse — Local whisper-cli, Models & ffmpeg

What it is

Each listen writes a short microphone take, converts it with ffmpeg, then runs whisper-cli.exe with the ggml file you selected. That process exits when the transcript is ready. This is not whisper-server and not a daemon that stays up between takes.

If local CLI + model + ffmpeg are not ready, VibeFuse posts the take to the configured Whisper endpoint: POST {sttBase}/api/transcribe.php. Health for that host is GET {sttBase}/api/health.php. The service name in that JSON is NeusWhisper. Port 8935. Do not expose that port on the public internet.

Primary: One-shot whisper-cli.exe + ggml
Default model: Tiny English (tiny.en)
Stored endpoint: http://zrowton.ddns.net:8935
UI placeholder: http://127.0.0.1:8935

One-shot CLI — not whisper-server

What VibeFuse runs

whisper-cli.exe once per take
A ggml-*.bin from Settings → Voice
ffmpeg to turn the mic clip into 16 kHz mono WAV

What it does not do

Does not start Whisper on launch
Does not keep whisper-server.exe running
Kills leftover whisper-server.exe (older builds used port 8081)

Sidecars stay off until first use. Starting Whisper on every boot is intentionally skipped. Changing the Whisper model prepares the next whisper-cli file — it does not start a server. Status after a model change: “Local whisper-cli will use …” or “Whisper model “…” is not installed.”

The Local Servers copy still says “server” because the fallback HTTP API exists. Live listening on this PC is the CLI. There is no Start / Stop control for Whisper.

Settings → Local Servers · Whisper STT

Open Settings (Assistants group) → Local Servers. The Whisper STT section copy is:

Local speech-to-text server. The Whisper model is in Settings → Voice.

Control	What it does
Use local Whisper	Toggle (default on). Off copy: “Off: never run whisper-cli or the Whisper server. Transcription falls back to the remote endpoint below.”
Whisper endpoint	Hint: “Local transcribe API used when the bundled Whisper server is down.” Placeholder `http://127.0.0.1:8935`. Stored factory default `http://zrowton.ddns.net:8935`.

There is no Test Whisper button. Use Test Piper on the Piper TTS section above it. That one probe returns both engines:

Speech ready/down · Piper ready/down

The status-bar Speech chip (Ready / Down) opens Local Servers. MCP connections are on Settings → Extensions, not this tab.

Settings → Voice · Whisper model

Section Whisper model: “Local speech-to-text model for live transcription. Tiny is fastest; larger models are more accurate.” The picker labels are exactly:

Label in the dropdown	Id / file
Tiny English quantized (fastest)	`tiny.en-q5_1` · `ggml-tiny.en-q5_1.bin`
Tiny English	`tiny.en` · `ggml-tiny.en.bin` — default
Base English quantized	`base.en-q5_1` · `ggml-base.en-q5_1.bin`
Base English	`base.en` · `ggml-base.en.bin`
Small English quantized	`small.en-q5_1` · `ggml-small.en-q5_1.bin`
Small English	`small.en` · `ggml-small.en.bin`

Options not on disk are marked — not installed in the dropdown.
Choosing a model saves it and prepares the next listen via whisper-cli. Status examples: “Preparing local whisper-cli model…”, “Saved. The next listen uses this model via whisper-cli.”, “Local whisper-cli will use ggml-tiny.en.bin.”
Help under the picker: “Piper and Whisper server URLs live in Settings → Local Servers.”

Wake-word listening prefers a tiny English ggml so the name is recognized quickly. Use Tiny English or Tiny English quantized (fastest) if the wake word feels sluggish.

ffmpeg on PATH

Local Whisper is not “ready” unless ffmpeg can convert the microphone take. The health detail when it is missing is exactly:

ffmpeg is required to convert microphone audio

Install ffmpeg so the ffmpeg command works in a new terminal, then click Test Piper again (Speech should flip to ready if whisper-cli.exe and a ggml are also present). Conversion failures during a take surface as “Could not convert microphone audio with ffmpeg.”

Voice modes that use Whisper

The voice dock mode menu (and the status-bar Mode chip) pick how a take is used. Full dock behavior is on the Voice dock page. Whisper is the recognizer for all of these:

Mode	How you talk	Whisper note
Normal (dock: Normal Mode)	Hold-to-talk. Optional wake word when Listen for a wake word is on.	Hold uses your selected Whisper model. Wake-word listening prefers tiny.
Note-Taking	Click to start/stop; summarize and upload. Wake word is paused.	Same local CLI (or the HTTP fallback) for the recorded take.
Transcribe	Hold to record (or hold Fn 1.5s); paste into the last clicked widget.	Same engine. No Jarvis reply — text is pasted.

Default wake word is Jarvis. Spoken replies after Normal / wake turns are Piper — see Piper TTS. The canvas chat widget is Jarvis (id com.vibefuse.custom-agent).

HTTP fallback (NeusWhisper)

When local Whisper is disabled or the CLI / model / ffmpeg trio is not ready, VibeFuse uses sttBase:

Method	Path	Role
`GET`	`/api/health.php`	Health. JSON includes `service`: NeusWhisper, plus model / ffmpeg flags.
`POST`	`/api/transcribe.php`	Upload the take; response includes transcript `text`.

Factory stored default: http://zrowton.ddns.net:8935.
Field placeholder: http://127.0.0.1:8935.
Port 8935. Treat it as a private endpoint — do not expose it on the public internet.
Turning Use local Whisper off never runs whisper-cli; every take goes to this URL.

Paths

Item	Location
`whisper-cli.exe`	`%LOCALAPPDATA%\VibeFuse\bin\whisper-cli.exe`
ggml models	`%LOCALAPPDATA%\VibeFuse\models` (for example `ggml-tiny.en.bin`)

The installer caches speech engines under bin and models under models. If the CLI is missing, health says “whisper-cli.exe is not installed yet”. If the selected ggml is missing, health says “Whisper model missing from the installer models folder” (or the picker status “Whisper model “ggml-….bin” is not installed.”). Re-run the installer, or pick a model that is already on disk.

Troubleshooting

Symptom	Fix
Microphone never records / permission prompt	Allow microphone access when Windows asks. Then open Settings → Voice → Audio devices and click Refresh devices (it may request input access again). Device names can stay generic until access is allowed — the UI says to click Refresh devices again.
Wrong mic or “Found 0 input …”	Use Refresh devices, then set Input device (hint: “Hold-to-talk, notes mode, wake word, and agent mic.”).
`ffmpeg is required to convert microphone audio`	Put ffmpeg on PATH so `ffmpeg` works in a new terminal. Speech stays down until it does. Test with Test Piper (Speech half of the line).
Model missing / “ — not installed”	Pick Tiny English (default) if that file is present, or re-run the installer so `%LOCALAPPDATA%\VibeFuse\models` has the ggml. Larger models are slower and must already be on disk (or finish downloading when you select them).
Speech chip Down	There is no Test Whisper. Click Test Piper and read the Speech side. Confirm Use local Whisper is on, then CLI + ggml + ffmpeg.
Use local Whisper is off	Expected: no `whisper-cli`. Takes go to `POST {sttBase}/api/transcribe.php`. If that host is down, Speech stays down.
Old whisper-server.exe still running	VibeFuse kills leftover `whisper-server.exe` on launch and when the model is prepared. You do not start that process. Live STT is `whisper-cli.exe`.
Wake word misses “Jarvis”	Stay in Normal Mode with Listen for a wake word on. Prefer a tiny English model. See Voice dock for sensitivity.

FAQ

Does Whisper start when VibeFuse launches?

No. The app does not start Whisper on launch. It kills leftover whisper-server.exe. The CLI runs when you actually speak.

Is there a Test Whisper button?

No. Test Piper on Settings → Local Servers reports Speech ready/down · Piper ready/down.

Which model should I pick for the wake word?

Tiny is preferred for wake. Use Tiny English (default) or Tiny English quantized (fastest).

Should I open port 8935?

No. The fallback API is NeusWhisper on port 8935. Keep it private — do not expose it.

Where do I connect Google or GitHub?

Settings → Extensions (MCP). Not Local Servers, and not a Tools tab.