T-Mobile Translate Phone Calls
T-Mobile Translate gives equal access to understanding by translating calls on the fly.
Team: Randolph Jenkins, Claire Ong, Lyubomyr Kryshtanovskyi, Rowan Osmon
Timeline: 24 hours
The Problem
Communication barriers between healthcare providers and multilingual patients increase the risk of medical errors by 25%, a challenge we know firsthand from having immigrant parents. To solve this, we developed real-time translation technology to improve communication and ensure accurate medical information exchange.
Backend Development
Using T-Mobile's Your Number Anywhere API, we created T-Mobile Translate, enabling browser-based calls with live translations in various languages through the OpenAI Whisper-1 model. Customers can make and receive calls without a SIM by linking their T-Mobile ID to the YNA API. During calls, users get a live translated transcript of the other party's speech in their chosen language.
Challenges
One of the main challenges we faced was live transcription. In the browser, we access the remote stream and record it in 100ms intervals, sending these chunks to the OpenAI Whisper-1 model for transcription and translation. However, OpenAI limits us to 3 requests per minute, which results in transcription updates every 20 seconds rather than in real time.
Next Steps
Moving forward, the next step is to explore more open solutions for transcription, as OpenAI's limitations hinder real-time performance. Another potential improvement is running a model alongside the client code, which would reduce latency but increase the app's size. We are also considering duplicating the app’s functionality on mobile devices and researching better methods to process audio in JavaScript for more efficient media handling.