Google Project Relate: A Communication Tool | Accessibility NYC Meetup Recap

PAN-PAN JIANG: The app that I'll be talking today about is called Project Relate. And I'm Pan-Pan, the technical program manager for this project. Julie Cattiau is also listed. She's the product manager.

She often gives this presentation with me, but she was unavailable today. You'll often see her and sometimes me on a lot of the videos that I'm showing you, and just as an FYI, we last gave this presentation actually just a few days ago at ATIA. I don't know if any of you guys were there. It's one of the largest assistive technology conferences there are, and we were actually able to gather in person this year.

It was really cool to see all the technology in person. If none of you have ever been to one of these conferences, I highly recommend it. It's kind of all the assistive technology companies kind of gather, and present either their latest research or their latest products, and Google usually has a presence. So, thank you so much for having me today, Thomas and Meryl.

I'll just get started on what Project Relate is, and first I'll start with a little background on what artificial intelligence is. You guys are probably all actually really familiar with this term, but I just wanted to give a high level overview of what it is and how we use it for our project. Artificial intelligence or machine learning is a new way of understanding rules. This is an example of how a traditional rule-based algorithm could filter spam in your inbox.

So you could say something explicitly, like: If the email subject has this word in it, please put it into spam. But now we actually do it a little bit differently, we don't say explicitly look for this word, we say, you as a user can mark what emails you consider spam, and that gives us really good information to learn and infer what other emails may be spam. So, even if it doesn't follow an explicit rule, because of other hints we can get from that email subject, or the content, we could mark it as spam.

So, that's one example of artificial intelligence. The other example that you've probably seen a lot is artificial intelligence for images, and this is the most relevant, actually, to our presentation. Here you see that, in this example, we're really trying to understand how to identify a cat, let's say. So we give it a lot of images of cats.

And we also need to give it a lot of images of dogs, and other things. What the algorithm is doing here is, it's trying to generate rules for how to classify these images. They're capturing these rules and this data in different representations, and each representation is one of these layers that you see here. Sometimes there's lots of layers in the model, and so that's why you may hear the term ‘deep learning’ or ‘deep neural nets’.

It's just a lot of layers of learning. Then at the end, it makes a decision. Yes, this is a cat, or no, it's not a cat. But this, what I'm showing you here, is really just the training phase of the model building.

You're just giving it a lot of data, and giving it labels, so that you can start to learn. The next phase, once you have a model, is to actually do inference on an image, a test case. An image it's never seen before. In this case, we're looking at this image over here, and I don't know about you, but it's actually kind of hard for me to see this image on the left -- whether that's a cat?

Is it a dog? Is it just a pile of clothes? So the model is going to be looking at this brand-new image, this test sample, and then it uses the neural net to infer whether or not there's a dog in there or a cat in there or perhaps nothing. So that's the inference step of building a model and using it.

Now, this kind of machine learning is really helpful if we want to, let's say, just identify all cats. These are all cat images, it's pretty clear. It can even do things that are a little bit more complicated. It can distinguish cats from bunnies.

They look pretty similar, but it can do a good job. But how about blueberry muffins from dogs? So, this is a silly example, but we put it here, just to show that we really also need some controls in the data , to make sure that we are doing a good job on training the model to differentiate between things that might look very similar. Probably if we had never put in any images of blueberry muffins, it could think that some of these blueberry muffins are actually chihuahuas.

I'm going to move on to how artificial intelligence helps people with impaired speech be understood better. Our mission is to help people with atypical or impaired speech be better understood. This is just a diagram, kind of holistically, of the number of people who may be living with these impairments, worldwide. We estimate that it's more than a quarter billion people.

Some of these conditions are neurological, some are congenital, some degrade over time, some actually get better – you may get better after a traumatic brain injury with enough speech therapy, or speech may get better, and some are pretty stable. We don't have any particular conditions in mind for Project Relate, we really want to help anyone who has ever felt like strangers or people they've just met have a hard time understanding them, and also technology.

They can't speak to their phones or dictation software doesn't work for them. So, yeah, that is our problem statement. The problem is that speech recognition just truly doesn't work well today for people whose speech is impaired or atypical or is in some other way impacted by some condition, and so, just like the examples I showed of the cats and the dogs, what we want to do is turn speech data into a picture, and then connect that picture with a label.

In this case, the raw input is a raw audio file, it's a recording of someone saying a phrase. The picture is a visual representation of that audio file, or what we call a spectrogram. All of these are frequencies or hot spots of different signals that are very specific to your speech and the words that you're saying. Then, just like in the cat example, we run it through a neural net.

And it's very important that each of these audio files, which are turned into visual representations, are also correlated with a label. That label in this case is a transcript, what did the person actually say? Then we do inference, so that once we train a model, we can, in realtime, hopefully, be able to take something that you're saying, which maybe the model has never heard before, and then do a transcription of that speech.

Project Euphonia is something you may have heard. It actually launched four years ago now, in 2018. Project Euphonia is the umbrella research project under which Project Relate is a specific app or product that came out of that research project. Project Euphonia is a data collection effort, so that we could get all these samples of people speaking, and all the raw audio data, that started in 2018.

It was a collaboration between us and ALSTDI, which is a research institute focused on ALS. But then we launched more broadly in 2019, and in fact, in May 2019, in this image on the left, that's our big public kind of PR conference that we throw every year, in the spring -- that Google throws -- and that on the stage there is our CEO, talking about Project Euphonia and the launch of our data collection effort.

When the project launched, we were asking people to record themselves saying phrases in a program called Chitchat, which is what is shown right here, and we were asking people to say specific phrases that corresponded with key use cases of interest. These use cases were either interacting with technology, either with home automation, like turn off the lights, or through voice access, a way to control your phone by just using your voice, and it's for people who have mobility issues.

The other kinds of key phrases we were asking people to record were those related to either talking to a caregiver, or just having conversations with friends or family, even strangers. In this case, we're really trying to record data that might correspond to things that you want to do with your voice, communicating with other people, communicating with technology, and also had diversity of phrase lengths, words. The model that's trained from this data is really only as good as the data that goes in, so we're just trying to get a diversity of data collected.

To date, we've had more than a thousand people record more than a million phrases. This graph is a little bit out of date -- it's 3 months out of date -- but this derivative here, this rate of change here, is pretty consistent. Now, we're probably closer to 1.25 million phrases, and we have also had more than a thousand hours of audio recorded. So, we've made really good progress in the last three years.

This kind of hockey stick growth here was really facilitated by IO, the conference that I showed earlier, where our CEO talked about the project. Since then, it's leveled off a little bit, and we're hoping to pick it up again with some more efforts, we really are hoping to get more data. In terms of the types of speech we've gathered, we actually published a paper on this at Interspeech 2021. There are a lot of different conditions here listed that we've collected data for.

We've also collected from both males and females -- more males than females --primarily ALS, because that was the partnership we had at the beginning, but also many other conditions. The upshot of all of that data is that we showed that for people with speech impairments, you can either have mild or moderate or severe impairment. Here on the x axis, their word error rates or WER, which is a measurement of how well the automatic speech recognition technology is understanding you -- higher is bad in this case.

If it understood 0% of what you said, your word error rate would be 100%. Being lower on this graph means that the speech technology is understanding you better. For all of these different categories of impairment, we were able to make significant improvements on automatic speech recognition. When we personalized a model, meaning we take their speech, and we train a model to understand that specific type of speech, the errors that model was making were much less than if we use an unadapted model or a what we can call an out of the box model, a model that already comes on your phone.

That's probably not surprising to most of you, that if you take your own personalized speech and build a speech recognizer, it works better than a generalized model. Here I'm just going to show you a partnership that we had with the Canadian Down Syndrome Society, who really wanted to let their population know about the project. They started a marketing campaign called Project Understood, which was more marketing, but the goal was to let people know about Project Euphonia, and let them know that they could contribute to the research that we were doing, in the hopes of maybe making speech technology work for this population and others.

I'm just going to play this video. Because it's a nice summary of the goals of the data collection project.

VIDEO: [PROJECT UNDERSTOOD] [By 2023 there will be 8 billion voice assistants in the world] [DEVICE] Playing your jogging playlist. It will take you seven minutes to get there by public transportation. Here's a summary from the Baseball Hall of Fame. [MATT MCNEIL] My name is Matt McNeil.

I live in Peterborough Ontario. I went to Fleming College. [But this technology isn’t trained to understand people with Down syndrome.] [MATT MCNEIL] Hey, Google.

Set thermostat to 78 degrees. [DEVICE] My apologies. I don't understand. [ED CASAGRANDE] My daughter, she has a smaller mouth.

She has a longer tongue. Picture talking with a marshmallow in your mouth. [BOB MACDONALD] Technologies that are activated by voice commands are becoming a way of life. [WOMAN] Scheduling and reminders, practicing phrases, speaking to friends.

[BOB MACDONALD] For people where that doesn't work, that must feel very sort of disempowering, or like they're being sort of left behind. [Access to voice technology could mean life-changing independence] [WOMAN] I see him being able to not need to rely on others as much. [So the Canadian Down Syndrome Society went to Google to help teach their voice technology.] [MATT MCNEIL] Hey, Google.

What is the weather in California? [Google, Mountain View, California] [Introducing Project Understood] [JULIE CATTIAU] We're all very excited to partner with the Canadian Down Syndrome Society. The goal is to make speech recognition work better for people who have different types of speech. [GIRL] I'm teaching Google.

[MATT MCNEIL] I'm teaching Google. [ANOTHER GIRL] I'm teaching Google. [PAN PAN JIANG] Thank you for coming to Google. [MATT MCNEIL] Thank you!

[JIMMY TOBIN] Out of the box Google speech recognizer would not recognize every third word for a person with Down syndrome, and that makes the technology not very usable. [JULIE CATTIAU] What we really need is for people with Down syndrome to volunteer, go on the website, and record phrases and donate their voice. [Each voice is added to a database to improve Google’s speech recognition algorithm.] [MATT MCNEIL] I owe you a yo-yo today.

Strawberry jam is sweet. [ED CASAGRANDE] The more voices we have, the better the technology will get. [MATT MCNEIL] Hey, Google. Turn off the lights.

[JIMMY TOBIN] This is basically the fundamental frequency. And we kind of look at how these frequencies move. And that corresponds to the words that you're saying. [MATT MCNEIL] When I started doing the project, the first thing that came into my mind is that it's really helping a lot of people be more independent.

[WOMAN] I think voice technology is so important, to give a voice to the voiceless. I think that's fantastic, that we live in a world where that is even possible. [Google teaches us so much but they need our help now.] [Donate your voice at ProjectUnderstood.ca] [MATT MCNEIL] Hey, Google.

Thank you. [DEVICE] You're welcome.

PAN-PAN JIANG: Okay. That was just an example of a non-profit that we worked with, just to let this population know about the project. We're open to anybody with impaired speech, it's not just Down syndrome, but this campaign was very successful in getting us a lot of Down syndrome data, which is really helpful for us, for our research. I mentioned that we've now been doing the data collection and the research for about three years.

We've had a few publications come out in the last year, and I want to make sure that those of you who are interested in the details about the machine learning and the data collection -- it's all available now on our website at g.co/euphonia. In fact, the data collection program is still ongoing, and we're still asking people to contribute their voice to the research project. If you know of anybody who has impaired speech, please have them go on to this website.

We have lots more videos, and ways for people to contribute their voice. At this point, this is about a year or so after we started the data collection project, we also started to develop early prototypes of how the research could turn into a product. We worked with many trusted testers to gather feedback about these early prototypes. I'm going to show you some videos of what these early prototypes look like.

I also want to share the stories of some of these individuals. This first video you'll see is of Andrea, who is actually featured in this picture. She's had ALS for six years. As you will hear, she does have trouble being understood by people that she's just met.

I'll just play a video of her using one of our early prototypes. Hopefully you can see the writing on the phone. The phone has basically been trained on her speech, and is now trying to transcribe her in realtime.

VIDEO: [WOMAN] Andrea, what are you going to do for Christmas? [ANDREA] For Christmas, we're going to Philadelphia, where my in-laws live, and my parents will be coming with us. Yay! [WOMAN] Sounds good.

Ha-ha! Is it cold in Philadelphia in December? [ANDREA] There's snow on the ground, and I sit on the couch in front of the fire for five days. [WOMAN] Nice.

PAN-PAN JIANG: And then the second video is one of our trusted testers, who lives in the UK. She also has ALS and, as you'll hear, she's a bit harder to understand.

VIDEO: [WOMAN] I had a very tiring day. Today was a hospital day, which means I had to get up at 6:00 to be in hospital by 9:00. This is for a clinical trial I'm doing on something called MIROCALS. I have to inject myself five consecutive days every month with some medication.

Which so far is being used to treat patients with a particular kind…

PAN-PAN JIANG: All right. So you can see there that she was a little bit harder to understand in fact. If you close your eyes, it's likely that you won't understand more than half of what she's saying. I find it hard to understand even 20% of what she's saying.

But with the app, which is not perfect by any means, you do hopefully start to be able to convey some of the more important words, and with the context and the app, and her facial expressions, it's all part of the communication package that hopefully will help her be better understood. So now I've walked you through kind of the life of our project so far, up until -- now at this point, we're in early 2021.

And we have done a lot of data gathering. We've trained, we've done the research using machine learning on the data that we've gathered, we've tested early prototypes, and now we're at a phase where we're ready to maybe launch an app to the public. So, that's the app that I'll be talking to you about now, that Meryl will actually show you later, at the end of this presentation, because she was one of our early trusted testers, and that app is called Project Relate.

So Project Relate is an Android app, and it's available in the US and Canada, also Australia and New Zealand at this point, where users are asked to record 500 speech samples. Once those recordings are done, the app learns to recognize their specific speech patterns, and then it unlocks three different features of the app. The features are: Listen -- and again, we'll see Meryl demo all of these. Listen is really just transcribing your speech in realtime.

Imagine if you had kind of like a little closed captioning system on your phone that could help you either have video conferences, talk to someone in realtime, you could even -- actually, we had a user who had very limited mobility, and impaired speech write a whole book using our app, because he was able to use dictation in a Google document. That's the listen feature. Then there's the repeat feature, which you can maybe use if somebody can't see the writing on your app because you're too far away.

Use cases are transactional interactions, where you might be ordering a coffee, or ordering at a drive through. And then there's the assistant, which will allow you to control your home, or ask for help answering a question, or as someone said earlier in this presentation, set a timer. You can do any of those things with the assistant. A lot of people that we work with have trouble getting their assistant to understand them, because of their speech, but if you train the app to understand you, it uses your personalized model to talk to the assistant so that you can be better understood.

So now I'm going to play you my last video, which just summarizes Project Relate, and also shows a Googler, Aubrie, who not only uses our app, but also helped us name the app. It's really fun to work with her. She's very invested in the project, she helped us name it, and she actually uses it in video conference calls for work and, as you can see in the video, she uses it for a lot of other things too..

VIDEO: [AUBRIE] Communication is a fundamental human need, and I think that it's also a basic human right. I'm Aubrie Lee, a brand manager at Google. More specifically, I'm a namer, and it's a lot of fun to say that I'm a professional namer. I have a type of muscular dystrophy that affects the way I speak.

I have always loved making new friends, but when people can't understand me at first, they have trouble getting to know me. It's as if there's an invisible barrier between us. I believe technology can help bring down that barrier. [JULIE CATTIAU] My name is Julie Cattiau.

I'm a product manager at Google in the research team. Project Relate started in 2018, when we realized that our speech recognition technology could be improved to help people with speech impairments be better understood. Standard speech recognition doesn't always work as well for people with atypical speech, because the algorithms have not been trained on samples of their speech. So, we decided to create an app that would be custom trained on individuals' unique speech patterns.

[AUBRIE] We named the app Relate, because of what it enables you to do. [JULIE CATTIAU] To use the app, the first step is to record a set of phrases so it can get to know how you speak. The Relate app has three built-in features. Listen, Repeat, and Assistant.

Listen transcribes everything you're saying in realtime. [AUBRIE] So I have a custom voice model now. [JULIE CATTIAU] Repeat repeats what you've just said in a clear computerized voice. [AUBRIE] May I please have a hot chocolate?

[DEVICE] Hot chocolate. [BARISTA] Ah, gotcha. [JULIE CATTIAU] Thank you. [JULIE CATTIAU] Assistant connects directly with your Google assistant.

[AUBRIE] Take a selfie. [DEVICE] Okay. Get ready. [AUBRIE] I hope I live to see the day when disabled people can move through the world and expect the same level of connection that everyone else has.

I think Project Relate is bringing us closer to a better future. [Project Relate is seeking beta testers with speech impairments]

PAN-PAN JIANG: Okay. I hope you guys were able to hear that. Oh, sorry. Okay.

So... This is just our last thank you slide for all of our participants who submitted their samples to our data research project, and also to all our trusted testers -- and I realize, Meryl, we don't have a picture of you, so please send me a picture of you, and I'll be sure to include that picture in this slide. We have many testers, but we are also looking for more testers still. So, if you know of anyone -- you yourself, or anyone you know -- who has trouble being understood by others, is 18 or over, and lives in one of the countries I listed earlier, which is US, Canada, New Zealand, Australia, then please sign our sign-up form, so that we can send you an invitation to test the app -- and the last thing is that you do need an Android device.

Okay. That's the last slide, and at this point -- Yes. Have we connected to ASHA? Yes, we were actually at the last ASHA conference in November, we had a booth there, and we're planning to be there again next November.

So, I think at this point, what we should do is have Meryl demo to you the app in realtime, and then I'm happy to answer any questions.

MERYL EVANS: Hey, everybody. This is Meryl speaking. You are watching the Listen screen, which basically captions me in realtime, and look how correct it's been. It's been great.

I have done 1300 speech calls that have been recorded and it's recorded all of them, and I have customized some phrases, especially my name. many times. Meryl. Oh!

It missed that time. But it got it the first time! So that's something to celebrate. Ha-ha!

Anyway... Oh, not all of you may know me, so I'm Meryl Evans. I work with Thomas Logan at Equal Entry, and I was born profoundly deaf. So that's why I have an interesting and unique accent that automatic captions don't really like very much.

This one seems to be doing an amazing job. So this is the Listen feature, and what's so cool about it is: Project Relate will save it in the history, and I'll show you that in a minute, because that really blew me away, what I could do with it. Now I'm going to stop this, and I'm going to show you the Repeat feature. This is the Repeat feature.

If I'm ever at that big coffee place everybody knows about, I will be ordering... Oh, gosh. a Frappuccino. I don't know Let's see what it says.

DEVICE: This is a repeat feature. If I were at that big coffee place everybody knows about, I would be ordering oh, gosh a Frappuccino. I don't know. Let's see what it says.

MERYL EVANS: I'll do it again. Because I didn't have the volume all the way up. Okay. So this is Meryl.

I am calling the pizza place. And I'm ordering a thin pizza with all the vegetables. All the veggies possible.

PHONE: Okay. This is Meryl. I am calling the pizza place and I'm ordering a thin pizza with all the vegetables. Are all the veggies possible.

MERYL EVANS: Awesome. Okay. Now, I know the assistant -- it has the T cut off. We're not sure what's going on, there's a little bug, but it works fine on some other phones, so I don't know what the deal is.

This is also another feature that blew me away, because I have never used my voice to use voice assistants like Google Assistant, and you know the other one -- the other one -- I'm being nice to Pan-Pan and not mentioning the competitors. Ha-ha! It was really exciting to be able to issue commands with my voice. My biggest problem was coming up with commands, because I never used it, so I've never learned the commands.

Maggie had a great question. Actually, let me put the Listen on, in case you don't understand. So, Maggie asks, Would it be possible to use the repeat feature while talking on the phone? I would think so, yes, because what I would do is, I would talk into repeat, and then I would put the device up to the phone, like I just did with the microphone that I'm using here.

The next question comes from Kimberly Lin. Very cool. Has this been tested in spaces with background noise? Or was there any training data where people spoke with noisy backgrounds?

Pan-Pan, would you like to answer that one?

PAN-PAN JIANG: Yeah, sure. Yeah. So... Thank you so much, Meryl, for that demo.

Did you actually get to the Assistant?

MERYL EVANS: Not yet. We got sidetracked.

PAN-PAN JIANG: Why don't you do the Assistant? And then I'm going to go through all these questions one by one, because there were some on the top too. I'm happy to do that.

MERYL EVANS: Okay, perfect. Oops! Okay. Let's see.

What is the weather like on Thursday?

DEVICE: There will be snow with a high of 24 and a low of 15.

MERYL EVANS: This is Dallas, y'all! No, I'm in Plano. I'm in Plano, and they're talking snow? That's an improvement over the ice they were originally predicting.

Let's try again. What's the weather like in Purdue?

DEVICE: It's 48 with rain. Tonight the forecast is for a high of 47 and a low of 33 with rain and snow. There's currently a winter storm warning in effect.

MERYL EVANS: Yes. So, that's where my son is going to college. He's expecting a foot of snow sometime this week. Let's see what else we can ask.

What is today's date?

DEVICE: It's Tuesday, February 1st, 2022.

MERYL EVANS: Oh, better yet: What is tomorrow's date?

DEVICE: It's Wednesday, February 2nd, 2022.

MERYL EVANS: I'm a numbers nerd. I like that it's going to be 2/2/22 tomorrow.

PAN-PAN JIANG: [laughs] Thank you so much, Meryl. That was great.

MERYL EVANS: Okay. The history feature... It has everything I've recorded. What I love about it is: I came up with the way I can use it.

I've never been able to dictate notes. I always write everything down because nothing can capture my voice accurately. What I was doing was: I was creating emails with my voice, or making notes for article ideas. So I would take... that's not much of a note, but you can copy it and it will copy the whole thing, or you can share it.

It will send an email, so I can make an email out of it. I didn't have to copy and paste, you would just share it with your email. Gmail is there. It will copy the whole thing into Gmail.

So that was really cool. All right. Over to you, Pan-Pan.

PAN-PAN JIANG: Oh, thank you so much, Meryl! Well, you know, just recently we released a feature that I actually told you about in an email earlier today. But now you can directly -- the Relate model is integrated with the keyboard, and you can directly use your voice to type out an email, use your voice to send a message, in WhatsApp or a messaging app, or even take a note directly in the note app, as long as you're using your keyboard.

So there are a lot of different ways that people can use this. But let me go ahead and just scroll back to the questions. The first question that I see is: What version of Android is this compatible with? It's Android OS 8.0 or higher.

Most phones now will be on 8.0 or higher. Most phones now will be on 8.0 higher, all phones that you buy today. It'll work on all Pixels, but you might see a slightly slower reaction in an earlier Pixel like a Pixel 2. Anything from Pixel 3 or onwards, you shouldn't see any difference.

Are there any plans to expand this to Apple? Right now, we actually have a really small team of seven core people. We're trying to focus on a few things. So I see there's a question about languages.

We actually are starting pilots in different languages. We're going to start out with Japanese, French, hopefully German, these are just our initial ideas. We're just in the initial discussions phase, but we really do want to prioritize expanding beyond English. iOS is probably not going to be available any time soon, but we want to explore these other languages, and then we'll see about the other options.

Volunteer opportunities? Google doesn't have an explicit volunteer option, but we do have internships that become available that are more for people with like a research background. I guess the answer is, not really, there's not really a clear way to volunteer, but one thing that people have done out of the goodness of their hearts because they like this project is just tell other people about it, especially for Project Euphonia that's still looking for people to donate their speech samples.

Once you donate speech samples, and we listen to them, and they're a good fit for our program, you can actually get a gift card for that, so there is something you can get in return for your time, and the details are on our website. Are there any plans to create a unique synthesized voice for users? This is a great question. So that does feel like a really good opportunity for synergy between like a banked voice and Project Relate.

Right now, it's just the synthesized voice that we have, so you can't use it with something that you've created for yourself, but maybe hopefully in the future, if we have the ability, I think that would be really cool. Would it be possible to use the Repeat feature on the phone? As Meryl mentioned, if you're on a land line or another phone, I think you could. We haven't tried this yet.

No user has said that they've done this, but you could potentially just put your app next to the phone, but it's not an integrated feature that the phone that you're using to talk to is also the phone with the app on it. We haven't really tested that. How does this work with Talkback also active? I'm not familiar with -- Is that a different app?

I'm not super familiar with how it might work with other apps. Screen reading will probably just read everything on the screen. Part of our goal for this testing phase is for us to use the app with all sorts of different features, and we'd love for someone -- Meryl, if you want to let us know how that goes -- we're always open to this kind of feedback. There are a lot of different accessibility apps and technologies out there that we haven't tested every possible combination.

This is our goal for the testing, to test some of these combinations. Have we used it in noisy settings? Yes, actually, we have tested it in noisy settings. I would say it does work better when it's not a noisy setting but, inevitably, some of the training data that we have does have noise in the background.

We've listened to a significant portion of the recording, there's definitely noise in the training data. We think it performs relatively well in a noisy background, but I think you will get better performance when it's relatively quiet. Will it work with someone who has an accent? Yes.

It should work with someone with an accent. It really doesn't matter what speech impairment you have, whether or not it's with an accent, you just train the model to understand your specific voice better. People who stutter often experience silence blocks which last for up to a minute. For people who stutter, we actually work with a speech and language pathologist who stutters himself – Richard, he was in one of the videos.

I think you're right that the transcription feature may not be super helpful because of the block, nothing is being transcribed. I think what could be actually helpful is the Assistant feature, because the assistant feature -- you tell the assistant when to stop listening. So, instead of talking to your Google Nest, if you have a block, the assistant stops working -- in this particular app you actually control when the assistant stops listening.

So, you can just ask for it to wait for you and then continue speaking. We actually have not had many testers who have stutters test it, but we're hoping to get more data soon from just a variety of conditions, and to tell you how it's working in each of those examples. We do know that stuttering, I think, does pose a particular challenge. Along with live sessions, what other voice data is used by Google when testing Project Relate?

We only use voice data. The only data that we're collecting is voice data, and then the labels corresponding to them. Are there plans to support phones which aren't Pixels? Yes.

It's any Android phone, not just a Pixel phone. Are there plans to expose the API so it can be integrated with other apps? Not at this moment. Will this technology be Open Sourced at some point?

I think that's a really great question. I think those two are kind of along the same vein. Right now, when you record data on the phone, it really is your data, and Google promises not to share with external parties. But I think that a world where more people can use the technology could be really great for this project, and for more people around the world, but at this time, I don't have anything to share about our ability to open source that data.

The volunteering? Yes, so if you Google ‘Google research internship’, there's a whole internship page that tells you all about the internships available. Again, they're mostly geared towards people who have computer science degrees and backgrounds. We just had two interns who are both PhDs in CS.

One of them had cerebral palsy. It’s very focused on a technical internship, but if that's your background, yes, please visit the site for the internships. Do you have plans to integrate with Google Classroom? Potentially, for example, integrating with some of the workspace apps like Meet would be really life changing.

Imagine you want to give a presentation with Meet, and you can use your own voice model, so that it can close caption you in realtime and it's integrated, I think that would be really cool. We're trying to get some of these things going internally, but right now, the app is kind of a standalone. I would love to see both companies work together and make this tool available on both major OSs, as well as Windows.

We heard it a lot at ATIA. You either kind of choose iOS or you choose Android OS or you choose Windows and you're kind of stuck in that universe, and I think it would be great if we could work together and make everything more compatible. We are looking into this.

JAMES HERNDON: I think the next question is from Christina. She asked, Is this integrated with the emergency calls feature?

PAN-PAN JIANG: No, it's not integrated with the emergency calls feature. I think the integration with anything that has the label emergency is pretty extensive, just all the approvals and making sure it works 100% of the time. So, we’re not integrated, but I can definitely see the benefit. Are voices collected when people are going through multiple emotions?

That’s a really good point. When we first started, people were only recording themselves when they sounded their best, and it was actually kind of a learning process to say, you don't actually have to record when you're at your best, contrary to what you do with your speech therapy, where you try to sound your best all the time. We probably want to capture you later on in the day when you're more tired, when you're in different moods.

If you want the app to understand you when you're upset, you have to teach it to understand you when you're upset. You have to give it that data. So, if you find yourself not being understood when you're upset, I think the solution is to record when you are upset, but of course, when you are upset, maybe you don't want to record. But yeah.

the data in is what dictates how good it is on inference.

JAMES HERNDON: I want to thank Pan-Pan for that amazing presentation. This was excellent.

PAN-PAN JIANG: Yes, thank you so much for listening and thank you for having me.