speech-compass

SpeechCompass: Enhancing Mobile Captioning with Diarization and Directional Guidance via Multi-Microphone Localization

Paper (PDF) | ACM Digital Library | Project Page | Google Research Blog

Artem Dementyev*, Dimitri Kanevsky, Samuel J. Yang, Mathieu Parvaix, Chiong Lai, Alex Olwal*

Official code release for SpeechCompass: Enhancing Mobile Captioning with Diarization and Directional Guidance via Multi-Microphone Localization, published at CHI 2025.

Video 4:24 | Presentation 9:30

*First and last author contributed equally to this work

Contents

• Repository structure
• Demo video
• Overview
• System
• Quick start
• Citing this work
• Previous work: LiveLocalizer
• Acknowledgments
• License and disclaimer

Repository Structure

Component	Description
android	Android app with Android Studio project (ASR + directional visualization)
dsp	Platform-agnostic C localization and beamforming algorithms, with Bazel unit tests
firmware	STM32 L5 firmware (GCC-PHAT localization → USB output)
hardware	PCB schematics for the custom 4-microphone phone case

Each component can be used independently — in particular, the DSP algorithms can be built and tested with Bazel without any hardware.

Demo video

▶️ Watch the full 4:24 demo on YouTube

The same video is also embedded below, split into 10 short chapters so it plays inline on GitHub (each chapter is also a quick way to jump to a specific topic).

Watch inline — 10 chapters (≈4:34 total)

1 · Introduction and problem — 0:00

speechcompass-01-introduction-and-problem.mp4

2 · Hardware overview — 0:24

speechcompass-02-hardware-overview.mp4

3 · Mobile UI: transcripts diarized by direction — 0:43

speechcompass-03-mobile-ui-transcripts-diarized-by-direction.mp4

4 · Mobile UI: customization — 1:17

speechcompass-04-mobile-ui-customization.mp4

5 · Live demo and group conversation — 1:29

speechcompass-05-live-demo-and-group-conversation.mp4

6 · Localization algorithms — 1:48

speechcompass-06-localization-algorithms.mp4

7 · Mobile UI and technical evaluation — 2:08

speechcompass-07-mobile-ui-and-technical-evaluation.mp4

8 · Foundational large-scale study (n=263) — 2:50

speechcompass-08-foundational-large-scale-study-n263.mp4

9 · Lab study (n=8) and large-scale survey (n=494) — 3:51

speechcompass-09-lab-study-n8-large-scale-survey-n494.mp4

10 · Conclusions and future work — 4:20

speechcompass-10-conclusions-and-future-work.mp4

Overview

Mobile speech-to-text apps have a fundamental limitation in group conversations: they transcribe everything into a single undifferentiated stream, making it hard to follow who said what. SpeechCompass addresses this by adding a spatial dimension — using multiple microphones to localize speakers in real time and overlay directional guidance on live captions.

The system is designed with accessibility in mind, particularly for people who are hard of hearing. Rather than relying on machine learning approaches that require video, speaker embeddings, or high compute, SpeechCompass uses classical DSP (GCC-PHAT + kernel density estimation) that runs on a low-power embedded microcontroller with low latency and no voice data retention.

Visualizations

The Android app offers multiple ways to display speaker direction alongside captions:

• Colored text — each speaker gets a distinct color
• Directional arrows and glyphs — indicate where speech is coming from
• Radar minimap — a persistent spatial overview of active speakers
• Edge indicators — subtle screen-edge cues for peripheral awareness
• Speech suppression — filter out speech from a specific direction

Performance

• Localization accuracy: 11°–22° average error at normal conversational volume (60–65 dB), comparable to human localization ability
• Diarization: 4-microphone configuration achieves 23–35% relative improvement in Diarization Error Rate (DER) over a 3-microphone setup across varying SNR conditions

User Research

A survey of 263 frequent captioning users identified speaker distinction as the most significant unmet need. In a follow-up prototype study with 8 frequent users, colored text and directional arrows were the preferred visualizations, and all participants agreed that directional guidance was valuable for group conversations.

System

SpeechCompass combines a custom hardware phone case with lightweight on-device processing:

• A 4-microphone phone case sends audio to an STM32 L5 microcontroller, which runs GCC-PHAT localization and streams azimuth angles to the phone over USB
• The Android app uses the phone's built-in microphone for speech recognition (ASR) and receives speaker direction from the case — keeping voice data local and processing costs low
• The DSP algorithms are written in portable C11 and can also run on phones with 2+ built-in microphones, providing 180° localization without additional hardware

Quick start

• Run the Android app — open android/ in Android Studio and run on a connected phone, or sideload a pre-built APK from Releases. Details and command-line build scripts: android.
• Build & test the DSP algorithms (no hardware required) — bazel test //.... Details: dsp.
• Build the hardware and firmware — see hardware and firmware.

Citing this work: SpeechCompass [ACM CHI 2025 Best Paper Award]

@inproceedings{10.1145/3706598.3713631,
author = {Dementyev, Artem and Kanevsky, Dimitri and Yang, Samuel and Parvaix, Mathieu and Lai, Chiong and Olwal, Alex},
title = {SpeechCompass: Enhancing Mobile Captioning with Diarization and Directional Guidance via Multi-Microphone Localization},
year = {2025},
isbn = {9798400713941},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3706598.3713631},
doi = {10.1145/3706598.3713631},
booktitle = {Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems},
articleno = {73},
numpages = {17},
keywords = {Assistive technology, hearing accessibility, localization, diarization, microphone array, captioning},
location = {
},
series = {CHI '25}
}

Previous Work: LiveLocalizer [ACM UIST 2023 Adjunct Proceedings]

SpeechCompass builds on LiveLocalizer (UIST 2023), which first demonstrated microphone-array localization augmenting mobile speech-to-text. The same hardware can run the SpeechCompass firmware.

@inproceedings{10.1145/3586182.3615789,
author = {Dementyev, Artem and Kanevsky, Dimitri and Yang, Samuel and Parvaix, Mathieu and Lai, Chiong and Olwal, Alex},
title = {LiveLocalizer: Augmenting Mobile Speech-to-Text with Microphone Arrays, Optimized Localization and Beamforming},
year = {2023},
isbn = {9798400700965},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3586182.3615789},
doi = {10.1145/3586182.3615789},
booktitle = {Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology},
articleno = {75},
numpages = {3},
keywords = {ASR, STT, Speech-to-text, accessibility, audio, beamforming, microphone array, speech},
location = {San Francisco, CA, USA},
series = {UIST '23 Adjunct}
}