Better transcripts: why I made Sideprobe's speech recognition pluggableApple SpeechAnalyzerNVIDIA ParakeetOpenAI WhisperCohere Transcribe
Speech recognition sets the ceiling for every transcript. After benchmarking local engines across several real calls, I made Sideprobe's speech recognition pluggable.
Juarez Filho5 min read
Over the past few weeks, I have been benchmarking local speech recognition engines across several recorded calls in my archive. When building software meant to assist during live conversations, synthetic audio datasets are misleading: real conversations are messy, regional accents collide with technical vocabulary, and people talk over each other.
The latest recording in my benchmark set was a roughly 30-minute partnership call conducted entirely in Brazilian Portuguese. I was discussing technical integrations, product roadmaps, and pricing structures with a partner in Brazil. I was wearing AirPods Max, talking over Google Meet, and running Sideprobe locally on my Mac.
I already noticed the glitches unfolding in real time during the live session. As the text streamed across my screen, the default recognizer stumbled on key moments:
- It wrote mais ou -50 when my partner said mais ou menos 50 (more or less 50).
- It wrote MGP instead of MVP.
- It wrote Macend instead of backend.
- In one single sentence, the currency symbol flipped erratically between R$, $, and EUR.
Once the call wrapped, I went deeper to analyze the logs, inspect the timestamps, and evaluate potential tweaks. To keep my benchmarks grounded, I also run well-known cloud AI notetakers in parallel on these calls. The tech industry often talks as if transcription is a solved commodity in the cloud, but the reality is much messier: the moment you step outside English, even famous notetakers stumble repeatedly. They drop colloquial phrasing, mishear regional abbreviations, and mangle technical shorthand.
Still, for Sideprobe's local architecture, my first question was whether my live audio capture pipeline had dropped packets. I isolated the raw audio legs and ran Apple's SpeechAnalyzer offline over the exact same recording.
It reproduced every single mistake.
The live desktop pipeline was solid. The speech recognizer itself was the bottleneck.
A transcript can only be as good as the recognizer that wrote it. When speech recognition drops words or garbles acronyms, everything built on top of that transcript inherits the mistake.
The greetings around that box are the languages Apple's speech recognizer supports on my Mac today.
The English-centric blind spot
Speech recognition has an obvious blind spot: almost all research and benchmarks revolve around English.
When companies demo transcription, they speak clean Silicon Valley English into studio microphones in quiet rooms. The demos look polished on stage. But human conversation rarely fits into clean corporate molds.
In real live conversations, speech rarely sounds like a podcast. Whether people are navigating English with different regional accents or speaking Portuguese, Spanish, French, German, or Italian, they talk with colloquial rhythms, mix local idioms with technical jargon, interrupt each other, and take calls from noisy coffee shops or through AirPods with aggressive Bluetooth compression.
This one is personal. I am Brazilian and I have lived in Ireland for nine years. My son speaks English, Brazilian Portuguese, and French, he is learning Irish at school, and he wants to learn Spanish next. Every one of those languages comes with its own rhythm, its own idioms, and its own culture.
If Sideprobe only works well in English, it fails the vast majority of the world. I refuse to build software that leaves everyone else behind.
When building software meant to collaborate with humans in the loop, respecting how people actually talk is table stakes. If I support many languages, I must give people the flexibility to adapt the system to their reality.
Four engines on the exact same audio
Instead of guessing which model worked best, I built a local test harness to evaluate engines across my call archive. I ran four local engines over the raw audio from that roughly 30-minute partnership call:
Tested on the same 33-minute call
How four local models transcribed the exact same recorded client conversation.
Misheard colloquial estimate as a minus sign ("-50") and flattened English developer terms (MVP and backend turned into MGP and Macend).
What the benchmark revealed
Looking strictly at raw speed or word count is an illusion. When I inspected the transcripts line by line against the recorded audio, each engine showed completely different tradeoffs:
1. Apple SpeechAnalyzer was blindingly fast, processing the roughly 30-minute recording in just 16 seconds with true word-level streaming timings. But on Brazilian Portuguese phrasing and developer acronyms, it stumbled repeatedly, dropping accented terms and flattening conversational nuance.
2. NVIDIA Parakeet v3 handled Brazilian Portuguese nuances and technical terminology remarkably well. The tradeoff was delivery style: text arrived in roughly 8-second chunks rather than a continuous live stream, alongside a slight 1.75% drift into English words during quiet pauses.
3. Whisper large-v3 suffered from an overly aggressive voice activity detector. During quiet, rapid dialogue, it dropped 32% of the guest conversation, capturing only 1,262 words against 1,854 expected.
4. Cohere Transcribe (q8) revealed why word count can be deceptive. It returned 3,170 words on the microphone channel, which might look like richer detail at first glance. In reality, over 23% of its words on the guest leg were repetitive hallucination loops where it repeated short phrases during conversational pauses. On top of that, its 11.5GB memory footprint exhausted my 16GB Mac, triggering aggressive disk swap.
Why pluggable is the right product philosophy
Seeing four engines fail in four completely different ways led to a core realization: forcing every human into a single speech recognizer is an architectural failure.
Different humans have different requirements:
- One person needs instant, streaming live text for real-time nudges and live guidance during rapid English negotiations.
- Another person conducts bilingual Portuguese client interviews where domain phonetic accuracy matters far more than an 8-second delivery delay.
- Someone on an entry-level MacBook Air cannot afford a multi-gigabyte model that melts their battery, while someone on an M-series Max machine wants maximum acoustic resolution.
That is why I made speech recognition pluggable in Sideprobe.
Instead of hardcoding a single provider, Sideprobe turns the speech recognizer into a selectable slot in Settings → Languages → Voice Model.
Apple SpeechAnalyzer remains the zero-setup default because it provides instantaneous live streaming text without extra downloads. But people can now choose Parakeet v3 for multilingual depth across six initial languages: English, Spanish, Portuguese, French, German, Italian, with more engine options being added continually.
The engineering under the hood
Giving people the freedom to swap speech engines sounds simple in concept, but making it reliable in a native macOS desktop app required strict engineering:
This is only the beginning
The launch of pluggable speech recognition with six languages is just the first step.
The world is wonderfully varied. Human speech carries regional accents, colloquial pacing, unexpected idioms, or shifting acoustic environments that no single static model can capture alone.
As I continue developing Sideprobe, my plan is to expand support to more languages, test specialized open-source speech recognition models, and help people discover the ideal engine for their industry niche and acoustic environment.
Real-time guidance during human conversations only matters if the software understands what people are actually saying. Giving people the power to choose their speech recognizer is one step closer to that reality.
Has speech recognition stumbled on your native language, accent, or industry terms during important calls? Share your experience below.
Responses come straight to my personal inbox. I read and reply to each one myself. Or if you'd rather share your thoughts directly with me or chat about Sideprobe, reach out on X (@juarezpaf) or LinkedIn.