ElevenLabs' Parallel Research and Product Strategy Drives Voice AI Innovation

Key Quotes

"we build foundational audio models so models in the space to help you create speech that sounds human understand speech in a much better way or orchestrate all those components to make it interactive and then built products on top of that foundational models and we have our creative product which is a platform for helping you with narrations for audiobooks with voiceovers for ads or movies or dubs of those movies to other languages and our agent's platform product which is effectively an offering to help you elevate customer experience built an agent for personal ai education new ways of immersive immersive media but all this kind of under under light of that mission of solving how we can interact with technology on our terms in a better way"

Mati Staniszewski explains that ElevenLabs develops core audio models to enable human-like speech creation and understanding, which then form the basis for their products. These products include a creative platform for audio content and an agents platform designed to enhance customer experiences through AI-powered interactions. Staniszewski emphasizes their mission is to improve human-technology interaction on user terms.

"the original belief came from poland it's a it's a very peculiar thing but if you if you watch a movie in polish language a foreign movie in polish language all the voices whether it's a male voice or female voice are narrated with one single character so you have like a flat delivery for everything in a movie that's a terrible experience it is a terrible experience and it's still you like if you grow up there the the as soon as you learn english you're like switch off and you don't want to watch content in this way and it is crazy that it's still happens until today in this way for majority of of content combining that and i worked with palantir my co founder worked at google we knew that that will change in the future and that all the information will be available globally and then as we started digging further we realized in every language in a high quality way exactly and the big thing was like instead of having it just translated um could you have the original voice original motions original intonation carried across so like imagine having this podcast but say people could switch it over to spanish and they still hear sarah they still hear madi and the same voice the same the same delivery which is kind of exactly what we did with lex back when he interviewed narendra modi and you could like kind of immerse yourself in that story a lot better"

Mati Staniszewski shares the origin of ElevenLabs' vision, stemming from a poor dubbing experience in Poland where foreign films used a single, flat voice. He highlights the desire to move beyond simple translation to preserving original intonation and emotion, using the example of a podcast that could be heard in Spanish with the original speakers' voices intact. Staniszewski believed this would be a significant future development for global information access.

"we realized pretty quickly that the models that existed just produced such a robotic and and not not good speech that people would not want to listen to it and that's where my co founder's genius came in where he was able to assemble the team and and do a lot of the research himself to actually create new version of of creating that work but like to your question i think the the way we are kind of organizing internally and how we think about sequencing a lot of that was looking at the first problem and then creating effectively a lab around that problem which is like a combination of mighty researchers engineers operators to go after that problem and the first problem was the problem of voice so how can we recreate the voice and like i say it needs to have that research expertise to be able to do that well so we started with effectively a voice lab which was that mission of can we narrate the work in in in better way"

Mati Staniszewski explains that ElevenLabs initially found existing speech models to be too robotic and unsatisfactory for their intended use cases. He credits his co-founder's expertise in assembling a team and conducting research to develop superior models. Staniszewski describes their approach of creating specialized "labs" to tackle specific problems, starting with a "voice lab" focused on improving narration quality through dedicated research.

"and then a lot of the technology wasn't possible for you to be able to um recreate a specific voice or be able to create that in that high quality way and then of course as we dived into further and and shifted away from the static piece the whole interactive piece is still crazy in the way it functions where most of us see this technological evolution over the last decades but you still will spend most of your time on a keyboard you will look at a screen and that interface feels broken it should be where you can communicate with the devices through through speech through the most natural interface there is one that can started when the humanity started and um and we realized we want to we want to solve that and i think now fast forward from 2022 i feel like many people will carry that belief too and that voice is the interface of the future as you think about the devices around us whether it's smartphones with computers whether it's robots speech will be one of the key ones but i think in 2022 it wasn't and um and as we think about the market for the creative side or whether for that interactive side it was like very clear it will be a huge a huge huge one"

Mati Staniszewski discusses the limitations of past technology in recreating specific voices with high quality and the persistent reliance on keyboard and screen interfaces. He argues that speech is the most natural interface for human-device communication and that this will become increasingly prevalent, even for devices like robots. Staniszewski asserts that by 2022, voice was poised to become a dominant interface, indicating a massive market potential for both creative and interactive applications.

"and then on the agent side you you you are like some of these things that of course customers that we work with or partners will will want to integrate which is we want integrations with xyz systems but then there are like other parts that might not be as easy to predict of as you interactive technology of course want to understand what's happening but you also want to understand how the things are being said and bring that into the fold which would be something we tried to prioritize on our side so then the people when they actually interact with the technology they realize oh expressive thing is actually so so much more enjoyable and beneficial and helpful"

Mati Staniszewski highlights that while customers often request specific system integrations for their agent platforms, ElevenLabs also prioritizes understanding the nuances of how things are said, not just what is said. He explains that by incorporating this emotional context and expressiveness into interactions, users find the technology more enjoyable, beneficial, and helpful. This focus on expressive communication is a key differentiator for their agent platform.

"i think this still the biggest one that hasn't yet kicked into the the system is like how education will be on i think this will be like i think learning with ai will with voice where it's like on your headphone or in a speaker it's just going to be such a big thing where you have like your own teacher on demand and who understands you very personified and kind of delivers the right content through your life i think this will