Accessibility Virtual Reality: Augmented Reality Captioned Glasses

MERYL: Now I'm going to introduce you to tonight's speakers. Alex Westner has spent more than 20 years leading product strategy at Isotope, Gibson, and Fidelity. He earned an MS from MIT Media Lab for acoustics and audio processing, and is a former electronic guitarist. How cool is that?

He has immersed himself in the most profound human problems related to audio. He co-founded Xander, setting his vision on providing relief for the 1.5 billion people around the world who struggle with the effect of hearing loss. And now Paul Mealy has spent the past 20 years establishing innovations for various organizations, from small start-ups to Fortune 100 organizations. Paul is an internationally published author on virtual and augmented reality. And has specifically targeted developing technology solutions for children, education, and assistive technology solutions for all people. As the chief product officer at XRAI Glass, Paul is responsible for designing experiences at all levels of user interaction and communication.

Now we'll go over to Alex, so you can introduce yourselves, what is your company, and what inspired you to create the company. ALEX: Thanks, Meryl. Thank you for having us. Thanks to the organization. This is very exciting to be here. I'm glad to share the stage with Paul to talk about this exciting new area that we're participating in. My personal story is... So your introduction to me shared that... Yes, I've been spending 20 years in audio.

But what happened to me several years ago is I was diagnosed with a macular degeneration. So some early form of low vision. And as I was both freaking out about it and curious to figure out what can I do about that, I put my product hat on, and I started researching... Well, what solutions are gonna be available to me, if I start to continue to lose my sight? And I was thinking that as an audio person, I'm gonna have to rely on audio and sound. And I looked for products, and I found. .. There's a lot of interesting products and research and money and technologies that are being developed to suit that need. And so as a person, I thought. .. Well, that's great. As a product and businessperson, I thought. .. Well, there's nothing more I can add there. But then I

started thinking. .. Well, wait a minute. There's all these experts solving this problem. And I'm an audio expert. What if I can solve... Help solve the opposite problem? What if I could figure out how to provide visual information for people with hearing loss? In the same way that I was thinking about relying on sound for vision loss. And that's when I realized... Oh, this is a very large market of people that have this challenge. And there are very few solutions out there to address it. So that's how this company was born, was to look at: How can we use sensory substitution, how can we use vision to supplement when we can't hear? So that's the backstory of Xander.

Yeah. I'll just stop there. That was a lot. MERYL: Go ahead, Paul. PAUL: Oh, perfect.

Thank you. Yeah. Sharing what Alex mentioned -- hello, everyone. So happy to be here. Happy to be speaking with everyone. Happy to have the chance to... And I think Alex's story definitely parallels mine, in a couple different ways, as far as the end of the story.

But for me. .. So my father was a social worker for the state. And he worked for the Bureau of Vocational Rehabilitation for 30 years or so. So he worked with people that had experienced hearing loss, vision loss, a number of different things. And worked to find them -- helped to find them jobs. And this was 40 years ago. And I think it was a lot harder than it was today, when computers weren't as familiar to people and those sorts of things. And as I was growing up, I saw a lot of that happening. My life definitely went a lot more heavy into the tech route. He is unabashedly not very tech-friendly. Whereas I kind of swung way the other way. And went very hard into the tech world. And you know... Growing up and seeing some of the problems that he and his

clients were dealing with, I felt like... Oh, you know, there are a lot of solutions on the tech side that could actually help out. Over the 40 years when he was working... Not only just advances that we had made, but things that maybe he hadn't been able to tap into at the time.

And so I posted. .. I do a lot of. .. As you mentioned at the very beginning, Meryl, I do a lot of work in the innovation space. And so in that space, I'm very interested in assistive technologies and just technologies that can help users communicate, that can kind of basically uplevel all people in different ways. And so I posted a lot of things on my LinkedIn for people with vision loss, for people with hearing loss, lots of different experiments on that. And during that, the founder of XRAI, who is not here right now, but the founder of XRAI actually reached out and said: Hey, we're actually working in a very similar space to the types of things that you're looking at. So I spoke with him. And the rest is history, as far as what we started to look into.

Which is... You know, I think we do want to look at the broad spectrum of things. But we definitely focused on people experiencing hearing loss first and foremost. And just like Alex mentioned, what can we do on the visual side, on the technology side, to kind of help people out that are experiencing that? MERYL: Thank you both. All right. We're getting to the questions. The first question I have for both of you -- either one of you can go first. It's up to you. If you want to go back and forth. And we'll start with: What prompted you to go the augmented reality route for making those caption glasses?

PAUL: Yeah. Happy to go first, Alex. From my side of things, you know, one thing that I was really interested in -- and this was one of the first videos that I had posted to LinkedIn -- it's an idea I've been playing with for years -- is the ability -- having augmented reality glasses, VR glasses, is gonna take us a step beyond phone. You can do a lot of speech-to-text on your phone right now. But holding up your phone and walking around is not the best experience. Not the best user experience. So as glasses were coming out, I got more and more interested in... Hey, how can we take this and make this an experience that feels more natural?

And doesn't feel as clumsy as holding up a phone? My background is in virtual reality and augmented reality. I authored the Virtual Reality and Augmented Reality for Dummies series. So I kind of had a big background to go off of. And what I really wanted to come up with was: Is there a way to take these captions and lock them to people as they're speaking? So if I were talking to someone, and they were speaking back, as I look to them, what they were saying locks to them, either as actual captions on them -- and if I look somewhere else and someone else is talking, does it lock to them? And if you go to my LinkedIn, you can kind of see this -- even as simple as popping a word bubble over their head, almost, as they're speaking. Just having that capability,

that's something that I felt like augmented reality offered that kind of went beyond just writing captions on a phone. And even just writing captions directly on the glasses, as a HUD. Being able to actually put it in space, I think, offers another level of communication that just on a mobile device does not. So that's kind of why I took it that direction. Alex, I would be curious to hear yours. ALEX: Yeah. I mean, I didn't come from an expertise in AR/VR. But I was really interested in the sensory problem. Right?

So vision loss and hearing loss. And then it's funny -- we were chatting before this panel about this conference that I used to go to, called Project Bar-B-Q, an annual audio thinktank conference in Texas every year. It was at that conference that I learned about augmented reality and mixed reality and VR. And it was really exciting and interesting. But we couldn't figure out what to do with it. So I was exposed to the technologies, but I didn't realize what a killer app for AR could be, until I started looking at sensory loss. Whether it's vision loss, hearing loss, what have you. And then I learned... Well, AR is analyzing video. It's analyzing. .. It's analyzing vision and sound and everything, and it gives you the potential to add displays and sound

and haptics. I'm really interested in haptics. So that was sort of where I thought... Oh, AR for. .. As you say, Paul, leveling up. That is the right application for AR. I mean, forget about, you know, adding cute text messages to the side of your screen. This is something that is actually gonna change people's lives in such a meaningful way. So that was really exciting. And that's how I got into this. MERYL: Great stories, both of you. Thank you. Next question is. .. What have you learned in working with customers? ALEX: What have we learned from working with customers? Just. .. Um... I'll start with this one. I think we had so many interesting surprises. The first one was for people who have never experienced augmented reality. Is that gonna be weird? To just put on a pair of glasses, and see. . . Stuff

floating in space? And we learned... No. It's not weird at all. Which is a relief. For everybody.

We would put a pair of glasses on, and people would say. .. Oh, there's text floating. That's so cool. And after about 30 seconds, they would get over that. And then they would just have a normal conversation. That was just amazing. To see that people could adapt so quickly. That was the biggest lesson. I think the second lesson learned in working with customers was. .. Everybody had a different opinion. On exactly how the experience should work. That was. .. That is kind of humbling and challenging as a product designer. Yeah. PAUL: Yeah. I'll echo that. Echo your experience, Alex. You know, one of the interesting things, interesting challenges of working with augmented reality is: You don't necessarily have the same patterns established that you do when you're designing for a screen. Even when smartphones came out, which has probably been our biggest product

leap for most people, moving from a 2D computer screen to a very small 2D phone screen. .. People still had that reference of understanding how 2D screens worked. And generally how you should be interacting with them. And so there was a knowledge gap there. But people could generally put together an understanding. They had a frame of reference for understanding how phones work.

I think for augmented reality and virtual reality and any mixed reality, it's pretty wide open. Like... Those best practices haven't really been established. And so to Alex's point...

Everyone has a different opinion, and different expectations of like. .. Oh, when I did this in space, I thought this would happen. Oh, can I reach out and grab this thing? Do I use this wand to grab it? How is it supposed to work? And so I think. .. You know, customer feedback is incredibly valuable. Because you just have no way. .. How you're thinking that something should work is probably very different than how Alex is thinking something should work. It's probably different than how Meryl is thinking something should work. And so it's both. . . It can be incredibly frustrating at times. Because you'll plan something out and you'll design it, and you'll say. .. Oh, this is how you use it. And then you'll see people using it in real life and

you'll say. .. No, no, no, not like that. Not like that. But no. It's super valuable. It's all good feedback at this point. I think... You know... As augmented reality becomes more mainstream and as people start to use it more and more, I do think those patterns will start to emerge. But in the mean time, I do think it's kind of these cycles of customer feedback that are the most important and being able to kind of put those together and say. . . Okay. Looks like people expected it to look like this. So let's actually pivot this way, as opposed to... This way we planned out isn't working or isn't resonating with people. So taking that feedback and being able to react to it is super valuable. MERYL: I'm not surprised to hear from both

of you saying that everybody had a different opinion. That's the one challenging thing about working with people with disabilities. We're all different. I mean, when you've met one Deaf person, you've met one Deaf person. When you meet one blind person, you've met one blind person. And we all have different opinions. And I'll tell a story about vibrations -- it was haptics, the patterns of the vibration. So it would have one pattern for dog barking, another for baby crying, another pattern for the fire alarm. And what bugged me about this is it was expensive, for one.

Two, it required having another device. I'm like. . . I have a cell phone. I have a watch.

I don't need another bracelet on top of that. And the other thing that was very interesting. Another Deaf person commented they didn't like it because there was haptics. And he said he spoke for all Deaf people when he said we don't like haptics. And some of us are like. .. No, no, no. You don't speak for us. We like haptics. In fact, I use haptics. But I also use visual alerts.

Like badges. Because I don't want to be buzzed for every single thing. Then I'll never get work done. Right? So it's great to have options. So based on all the different feedback you got, I'm sure it was a challenge. What are you gonna do? So what have you applied from your customer learning?

ALEX: You want to start, Paul? PAUL: Yeah, happy to start. I think what we have found and what I've found, just from working in the AR space in general, is... You definitely need to kind of wade people into the waters of AR.

Because... So Alex's point early on... Because everyone kind of has different expectations.

It's not a technology that you can just say... Oh, okay. Here's our operating system. Here's how everything works. Go off. You're set. It's more like... You have to kind of teach along the way. And it's almost baby steps. This is for anyone using AR for the first time.

Even... I'll put on different AR devices. And I won't be sure -- I've used a hundred AR devices, and I won't know how to use it. Because what Microsoft is doing is different from what Meta is doing, and what Nreal is doing is different from what someone else is doing. So finding a way to ease people into your system, as opposed to kind of just dropping them in and expecting them to figure it out. .. I think that's probably been the biggest takeaway. And these fast cycles with customers, as you release features, just testing them with people, testing them with people, again, not going six months to a year and trying to drop some big...

All right. Here's our update, everybody. Because ultimately what's gonna happen is you're gonna end up having a bunch of rework, because nobody understands what you're trying to do. So those tight feedback cycles are really important. And just figuring out ways to kind of...

You know, wade people into the depths of your product. I feel like that's super important, at least in the AR space. ALEX: I can echo the wading people into AR. I was at an event last year. And somebody came up to me. Was very interested in the glasses.

Tried them on. Wait, didn't try them on, sorry. Was very interested. Was talking to me about the business. And then he shared with me -- he was a co-founder at one of the first pioneering display companies that made AR displays. And I won't share who or what the company was. But...

So he's been around AR since the beginning. And I thought... Oh, well, then, you know all about this technology. And so he said... Let me try the glasses. And he tried them. And he was surprised.

And he's like... Oh, wow. This is really cool. And I'm thinking... How could you not...

Wouldn't you expect this to be cool? I was surprised that he was still surprised. And I think... I guess what my experience from that... And you've been around this longer, Paul, which is... AR itself is also different for everybody. Even for someone who's been around it.

You just said you want to try on different experiences. Because you never know what it's really gonna be like. It's so new and different for everybody. And I think that's also what we learned is. .. This has to be simple. The technology is still complicated. The only way these solutions that we're talking about are going to work for people is if they are dead simple.

Just has to be simple. I mean, I've heard from people talking to audiologists, who say that the biggest problem they have with their customers, with the Bluetooth apps, the hearing aid apps, is that people don't even remember their App Store Passwords and they can't even download the app. They can't log into the App Store to even get the app, because they can't remember their password. Wow. Right away. .. Step one, there's already a big obstacle to overcome. And so. .. We really have to think about: How do we continue to make these experiences as simple as possible?

MERYL: So...

We talked about how you applied the customer learning. Now, what I'm hearing from both of you is that you involved the customer throughout the product life cycle. So many companies make the mistake of involving customers after they release the product. And they end up hating it. Because they never involved them. It's like... We don't need this! Everybody knows about the Sign Language Gloves. And nobody likes them. At least... Nobody that I know likes them. Because they put the onus on the person who speaks Sign Language. Not on the person that's listening to them.

So that was really appreciated by both of you, is that they were involved throughout the process. So I would like to know... What kinds of feedback have you gotten at the early stages, the middle stages, and the current stages? ALEX: Um... The earliest stage feedback -- I think it was similar. It was sort of what we shared. The first lesson we learned was: Oh, this experience is fine. You know. At its base, core, fundamental... Value proposition. ..

Putting on a pair of glasses, seeing captions, to supplement hearing -- that works. That was great. But then the middle stages was everything else. Well, I want to see the captions here. I want them there. I want them there. I want them there. I want them blue. I want four lines. I want two lines. All of that feedback was just... Oh my goodness. That was the next level.

I think the later stages... What's emerging now for us is... Location. Being able to... Initially you're sitting down with someone in one location. Testing in a simple environment. But when people are taking these glasses everywhere, and they're going, doing all kinds of things... And so now we're learning... What are the environments? Where the experience works and doesn't work? And that's sort of that next level. And I think one of the... For us, one of the biggest challenges that I've...

I experience more often, because even just trying it demo this to people is -- connection. Connectivity has been a problem. I don't always get good internet connection. I don't always get good Wi-Fi. I went into a trade show... When we went to the Augmented World Expo in June, I went into the trade show floor. As soon as I walked in, my glasses stopped working.

I was like... Well, what do I do? That's crazy. I can't believe they just stopped working. Because there was so much radio interference in that environment. And so... That's been. .. That was one of the more recent challenges. Is getting out to different environments -- has exposed some interesting new problems. PAUL: Yeah. I'll 100% agree with everything Alex said there. The one thing I would say... So to Alex's original point of...

You know, do the captions work? Are they accurate? Are they tracking as fast as possible? That's the biggest feedback, the biggest concern people always have. But once you get that base level working, it is all of the pieces, Alex, that you kind of called out. And everyone wants to tinker.

Everyone wants to have this level of settings where they can say... Oh, well, I want this to be a little bit more personalized to me. I want the captions here, I want them this color, I want the captions down here, I want 15 lines, I want 100 lines. So going back to what we discussed, you know, a few minutes ago, being able to provide that level of -- base level, for somebody that just wants to put on the glasses and kind of have things work -- we found that was really important. But then starting to progressively uncover all these different types of settings that they can start turning on and tweaking and tweaking, as they get more used to the glasses and more familiar with stuff, as they start to progressively disclose those things and let people

kind of tune it to their own preferences... That's probably been our biggest -- our next biggest step, as far as where we're taking things. And where people's feedback seems to be pushing more and more towards. As we've tightened up the foundation of the app, trying to start to uncover those progressive disclosure of... Oh, as you become a power user, now you can start doing all these different things and tweaking them to your liking. I think that's probably where we're at, as far as development in our app. MERYL: Before I forget, do either of you have a pair of glasses in front of you, so you can show the audience? Ah, Alex. PAUL: I do. But I end up having to wear mine on top of my glasses. So... ALEX: There we go.

MERYL: So we want to share how that works. Because I've tried on both of your glasses. And they work very differently. So would you like to share how yours work? How is the set-up? What is involved? ALEX: Sure. I mean. .. I guess... This is definitely one place where our approaches are very different.

So the glasses... Neither of us are making our own glasses. Which is a relief. Because hardware is hard. So the company we're working with right now is called Vuzix.

Here's the company. That's the case. Vuzix. They're publicly traded. They're in Rochester, New York. They do their own manufacturing there. But the glasses they run -- they're powered like a low end smartphone or a super charged smartwatch. So they have an ARM processor.

They run Android. They have RAM and they have a built-in display. So basically we're developing an app that will run on the glasses. And what we're now doing is we're putting the entire speech recognition engine on the glasses. So that the customer experience would be. .. We would sell a pair of glasses to you. You open the box. You push the button. You put the glasses on.

And that's it. So that's really the experience we're trying to deliver. Is... To address that simplicity problem. And the learning problem and the learning curve. And the reliability of not connecting to the internet. We're putting everything on the glasses. And of course, there's big trade-offs with doing that. But this is the path we've decided to go down. And so when you put them on, yeah. All you do is put them on. There's built-in microphones that are pointed away from you, to capture what other people are saying. And you just see the captions in a display about arm's length in front of you. We have video... A couple videos on our website that show what it's like. It's kind of hard to get a camera behind the glasses. It's really

challenging. But we managed to figure out how to do that. So you can see what that's like.

PAUL: Yeah. And from our side, so. .. The company that we're working with the most at the moment is called Nreal for our glasses. These are the Nreal Airs. They also have Nreal Lights with cameras on the outside. These don't have them, but the Lights do. So they can track more actively.

But for the Nreal Airs, they have an app you can download onto your smartphone. Either iOS or Android. So the glasses themselves are just the display. And the benefit of that is we do get the ability for users to not have to learn as much on the AR side. Because they can kind of fall back to a lot of stuff on their actual device to kind of set things up. And just kick off the conversation.

So that's been our approach. And the theory there is that: Hey, this can work for iOS, it can work for Android. It would be easy enough to roll into other apps as well. Right now, we're mainly working with Nreal. But basically anything that supports Android we could eventually go and install on. MERYL: So Alex, yours are standalone. I'm assuming that the plugins, the glasses -- how do you get software updates? ALEX: So we're expecting that...

Two paths. One is that if you are a power user and you have the ability to connect to something, either Bluetooth to your phone or USB to a laptop, we'll provide an app that lets you get updates. And some other things that we've been thinking about. If we can connect people, what else can we do to offer more? The other path is... As we partner with audiologists and optometrists, to resell the device, the professionals could also do the updates for their patients.

That's the other path. Which we're expecting could be just as popular as people doing it themselves.

That's a good question, though. It's one of the limitations. But we're gonna figure out some ways around that. MERYL: There's always advantages and disadvantages of everything. And Paul, I know I have a pair of glasses here. .. So I was able to play with it. And I know there's a cable from the side -- I don't have them in front of me. From the side that plugs into the Android. And I don't think you mentioned -- do you want to mention about what kind of Android phone is required? Because it doesn't work with just any Android. Correct? PAUL: In theory, it actually works with any Android. So the way it works right now is to get the AR placement of the captions -- there are specific Android devices that Nreal has on their website that you would have to purchase. One

of these 20 supported Android devices or whatever their number is. But you can essentially cast... I think iPhone calls it Aircast. You can screencast from any device. So any iPhone, any Android. Onto those glasses in the same way. So you would wire to your phone -- just like you would with the supported Android devices. Now, what you lose from that is you do lose the ability to kind of place captions in space. But you would still receive captions, just like you would anywhere else.

So all phones are supported. Only some phones are supported for the AR-specific captions.

MERYL: Awesome. Okay. For our audience, what are the privacy challenges associated with the caption glasses? PAUL: Yeah. I'm happy to start there. That's definitely one of the concerns that we kind of started... The foundational concern for the company -- we want people to own their data. And there's something interesting about. .. When I put these on and I'm live captioning, I am listening to the environment around me. We don't want to unwittingly bring other people into those conversations and have their conversations recorded when they don't want them recorded. How do we provide that signal to let people know that I'm recording and then what do we do with those recordings?

So it's definitely something we've been working with and trying to figure out. Hey, how can you let the user know that they're recording? How can we make sure that it's very clear to the user that we're not storing anything. This is all on your device.

And how can we make the other participants in that conversation kind of active participants in a way, to say... Do we enroll them beforehand, just to say... Oh, if you haven't captured any of these -- any voices of these people that have been enrolled, then you should just basically drop it out. So it's definitely... There's a lot of interesting things. There's a lot of interesting challenges both with this product specifically and with kind of AR in general.

And the story I always tell is... There was... I forget the artist that makes the giant shiny balloon dogs. There is an artist that makes the giant shiny balloon dogs. I forget who it is. And he did something with Snap, the company that makes Snap Spectacles, where they placed an AR version of the giant balloon dog in Central Park or something like that. And the very next day, someone had digitally tagged, basically, graffiti'd the giant balloon dog. Because their whole point was. .. Hey, AR space. Yes. I don't have to... I don't have to have my phone on. I can kind of ignore it if I want to. But this is still public space. And their broader point was: We need to start having this discussion early. Of what AR means. And can people put what was basically an

advertisement... Can people put advertisements all over? Because as these glasses get... As these glasses lean towards something that everyone is using... Is it okay if I put this on and there is now an advertisement in my house? On my wall? I didn't say that was okay. And companies will argue that... Oh, well, it's digital. It doesn't matter. You can just ignore it. But there's a lot of interesting problems that I think AR is starting to unveil.

That we haven't really tackled. And I think this is a really big, really interesting question that we probably can't cover inside of, like, I think this could be a conference in and of its own to talk through this stuff, is what I would say.

ALEX: I think it's interesting that privacy comes up with AR. In a way that it hasn't as much with a smartphone app. We've talked about how right now, there's probably a dozen great apps from my phone that I can do this now. So why have we not talked about privacy issues with recording, using a speech-to-text app, on my phone? It doesn't come up in the conversation. But it always comes up when we talk about glass. Even though it's the same function. Are Apple and Google warning users... Don't do that! It's privacy issues! Of course not. I think it's interesting that it comes up for glasses. And there's probably some good reasons of just it being a different form factor.

But... So our easy approach is: Because we're building these glasses like an appliance, it's not cloud connected, we don't have to record. By default, we're not actually recording any audio. We're not recording any conversations. Audio just passes through and goes away.

So there really is no privacy issue at all. When we allow users to record, then it gets more interesting. Then we have to think about... Do we educate people about two party and one party consent laws? We could. Again, does Apple do that? Does Samsung do that?

No. You can just put your phone down on a table and turn on voice memo and start recording. And there are privacy implications of that. So it is interesting that it seems like people want us in this AR industry to take on more responsibility for privacy than the smartphone industry.

And there are some good reasons for that. But it's definitely interesting that all it is... Is a different form factor. We're just taking the technology of a phone, putting it on your head, and now privacy is a huge concern. So it is. .. Yes, I'm not meaning to downplay it. Of course privacy is important. But it kind of always is. And it is now. It's just... This form factor is making us think of privacy in a different way. PAUL: Yeah. I think just to add on to that, Alex... So I previously worked at Meta, Facebook, whatever you want to call them. And I had a coworker who was wearing glasses, essentially, that allowed them to record. And they were in...

They were visiting their family for the holiday or something like that. And they were recording with these glasses. Just their family opening presents and things like that. And then later on, I think they posted these videos online. And their family was kind of annoyed. Like. .. Hey! Wait a minute. When did you take that? I didn't realize you were taking that. I think there is... You're totally right. I think it's a form factor thing. When had this person just been holding up their phone and kind of going like this, everyone would be like... Oh, you're recording. And they would have put on their "recording face", I guess. But they just didn't realize... Oh, wait a minute. You're recording? Because they didn't realize that, that was something that triggered something in them. I think it does speak to how much utility... To me, it's actually...

It's a good thing that people are thinking about this. And to me, it actually speaks to... Oh, I think this is where the future is going. And people are concerned about it, because I think they see. .. Oh, wait. This is gonna be something that we have to think about now and kind of head off at the pass, as opposed to closing the barn door after all the animals have gotten out. So I really do think this is like a deep, deep subject that I'm happy to hear that people are kind of digging into it early, as opposed to late. ALEX: Yeah, it is a big deal.

While I said the technology exists, we're actually creating an experience where we're encouraging people to capture conversations, personal conversations. So it is a big deal.

And there is a responsibility we'll have to make sure that that's all done safely and transparently. MERYL: So I'm a member of the W3 Immersive Captioning Group. And we talk about privacy a lot. And I was trying to figure out... Do we get the conversation -- I mean, you have to think about a person's hearing. If you're in an office, you can overhear someone's conversation in the next cubicle.

Can you hear two cubicles over? Those kinds of things. Because you want to be equitable. For those who don't hear. Like me. But you also want to draw the line. Because technology hearing and human hearing are different. So I would want, in a virtual reality environment... It was really cool, where we were sitting at tables, and we basically have privacy just at our table. We could not hear any of the other tables. I thought that was great.

It was a controlled visual that they had. So you could only hear who was sitting at their table. It made sense. But it's a little tricky when you've got people walking around.

Like in one of the environments I've also been in, where people are just having conversations, and the captions are all over the place. So the closer you are, the bigger the captions. The further away you are, the tinier the captions. You can't read them. So it kind of flags down when sound is further away and you can barely hear it. Interesting implications. And Alex, nobody is talking about privacy with the phone or speech it text.

Or notes. But yet the glasses and the captions... There's been a lot of conversations around that. So thank you both for sharing your perspectives. So I'm gonna ask one more question, before we open the floor. And that is: What do you see in the future of communications technology for the deaf, hard of hearing, and other accommodations related to accessibility?

ALEX: I think that to me that's what drew me to AR. Is... I mean, it's like human augmentation. Right? It's like...

The technology platform basically adds sensors to figure out what's happening in the world. Inputs. And then translates those inputs into outputs that help us understand. And I think fundamentally, that's really interesting for when, as humans, if we're deficient or missing one of those senses, we can supplement with this technology. To provide that information.

That's just fascinating. And I think we're just at the very beginning of that. And I think it's not a coincidence that companies like Google and Meta, in their very first big demos of what the future is like, of course they're talking about these very applications of basic human augmentation. I think they can see that too. As much as they want to sell glasses to consumers for entertainment and who knows what... Really the best use case is really this sort of human. .. Natural human augmentation. PAUL: Yeah. I totally agree, Alex. I think for all of the same reasons you mentioned around AR, again, it's in the name. It's augmented. And we're finding that. .. My hunch has always been exactly what you mentioned at the end of your talk, Alex. Which is just. .. Yeah. Games, things like that, will those be interesting in AR?

Yeah. Totally will be. But it's actually the utility pieces that actually are augmenting your experience, augmenting your communication, in ways that we can't necessarily think of right now, or can't accomplish right now -- that's actually gonna be the killer app. I don't know that I believe in a killer app, per se. But if there was a killer app, it's gonna be these utility pieces that can kind of uplevel your communication experience with other people, whether that's... If I'm in Spain, whether it's translating everyone's Spanish to English for me, because I couldn't understand it otherwise, facilitating communications in some different ways, I just think there's just a wide open space of things that we can do with communication in general that are hopefully gonna uplevel the Deaf experience, the hard of hearing experience,

but also just everyone's experience as people. Again, as we all start wearing these glasses. I can't even begin to brainstorm all the different ways that we'll be able to... If I'm meeting Alex for the first time, and I can see information about him -- oh, this is what... Here is what Alex likes. Essentially a "Hi, my name is..." short form that he could potentially expose to me or strangers or... There's just a million different ways that I think this could work.

For all people. But yeah. That would be my hope. Is that these technologies just start augmenting our experience with other people, in ways that we just don't have available at the moment.

MERYL: Awesome. Thank you both. So our first question is: What tools do you use to design and prototype AR experiences? Are those tools themselves accessible for disabled designers?

ALEX: I think Paul's answer is probably more interesting than mine. I think we're just developing an Android app. It's a boring 2D Android app that lives on glasses. So we're not using any specific tools, because all we're doing is putting text on a screen.

PAUL: Yeah. We use a couple different things. You know, what I would say is... Unity is probably what we use the most. And Unity can output to a lot of different devices. It's part of the reason we use it. It's also really well supported by Nreal, and they build their SDK for that. As for how well it's accessible for disabled designers, I can't personally speak to that. Because I've never used it in that way. But the way Unity works, it's essentially a game engine. So it does expose 3D a little bit more than just the standard development kit might.

But yeah. It's a really powerful tool. You can do a lot in there. It's really good for building just VR experiences, AR experiences, MR experiences. So definitely something to look into, if you're interested in the space. MERYL: Next question. What is the typical reaction when someone who is deaf or hard of hearing wears the glasses for the first time? Now, I'm sure you've had different reactions. So go ahead and share a variety of reactions you've seen. ALEX: We have a bunch of reaction videos on our YouTube channel. But really... We have one video where everybody is just saying... It's so cool!

Really, that's just our initial reaction. It's sort of. .. People just can't believe that it's just real and easy. And I've had one person say: This is every hard of hearing person's dream, is to see captions of what people are saying. It seems obvious to them. And they just can't believe it's actually in front of them and happening. PAUL: Yeah. I'll echo that. The one thing I'll say... And Alex, you mentioned it earlier in the conversation, is... Yes. Everyone... I shouldn't say everyone. But pretty much everyone loves it when they put it on. It's like... Oh yeah. Of course. This is obvious. And then 15 minutes later, that's when they start to say... Oh, you know what would be nice? If we could do this and this and this. And they're totally

right. Completely valid requests. It's just human nature to use something and start to immediately see. .. Start to want to tweak it in ways that you would use it within your life.

MERYL: So when I first tried on Alex's glasses, I was at the conference where we met. And it was noisy. It was dark. It was outside. And I put them on. I had no idea what to expect. And while I had them on, and I was talking to Darren Fox in front of me, what I was doing out of habit, of course, was trying to read his lips. And I'm trying to read the captions. And our eyes are just not made to multitask. So reading lips and reading captions at the same time is impossible.

So I had to tell myself... Okay. Let's try again. But this time, put your mask... Dylan happened to have his mask. So I told him to put his mask on. That way, I'm not tempted to read his lips. And it was a much better experience. And then trying your glasses, Paul, was different. Because I got them at home. So I wasn't in an environment with a bunch of people.

It was just quiet. It was just me and my family. And actually... No. I was actually on with the CEO of XRAI. Because he was showing me to set them up. Thank you. So we had... It was an adventure.

It was trying to give us a hard time. You know how technology is. It breaks when you need it most. So just like Alex's stopped working when he got to the conference. But after a while, we got it going. And it was getting better and it works every time. So it's been very interesting. Very different experiences. Okay. The next question is: Could you share your thoughts on the business landscape for AR and assistive technology? How do you navigate the funding for this kind of work?

ALEX: I could talk for an hour about this. The interesting thing about fundraising for assistive technology -- and it's changing -- but when I first started going out to fund raise, I mention hearing loss, and people just tune out. And they think. .. Okay. This is seniors in nursing homes with no money. I'm not interested. Pass. And I could put up all the stats and figures, how this is a very common problem, and people are spending tens of thousands of dollars to solve it.

I could go on and on. But they didn't care. It's just... Once you have an image in your mind as an investor, that's it. You can't get around that. And so not only is there a stigma for people to get hearing aids. There's a stigma for investors to want to invest in this problem as well. That's starting to change. As I guess either we've learned to talk to the right people... But there's a big movement in age tech and longevity. I think people are starting to realize...

We are living longer. And the generation that is living longer actually has more money than the younger generation. And they may not have enough money... There's an interesting space right now in this older generation of longevity. And that's where investors are starting to say... Wait a minute. This actually is a big deal. We should be paying attention.

That's what's changed probably in the last few months. I don't know why or how. But that's sort of something we've seen. I can't speak to AR funding. But I can speak to the... That's my experience with this assistive tech space. PAUL: Yeah. I generally. .. I think I generally agree with that. You know, one of the interesting things we've found in some of our research is actually a lot of the younger generation, Millennials or younger, actually use closed captioning almost more than any other generation, strangely enough. And I know I personally use it.

I'm not traditionally hard of hearing. But I just prefer to watch TV with it on. Because my kids are running around, and there is kind of temporary... Temporary times when I can't actually hear what's going on. And so it just helps me to have that on. So I'm not entirely sure why Millennials are kind of leaning into that. But that definitely seems to be something that they're doing. And it goes back to what we talked about a little bit earlier, which is just. .. I think it goes even beyond just captioning these conversations. But one of the things we've been doing too is. .. Hey. Inside our glasses, can we summarize these conversations? So I've been in a 20, 25 minute conversation. I might have forgotten something somebody said 20 minutes back. Can I pull that up? Or can I

just say... Hey, XRAI, can you summarize this for me? And it will actually spit out. .. Oh, this is what you've been talking about. Here are the salient points. If I was sitting in a classroom and a teacher was teaching, there are ways that this can be assistive technology that kind of goes beyond just like. .. All right. We're transcribing these things. And can actually assist you in different types of ways. And I think once you start tapping into that, that's where I feel like... The business side of things kind of perks up a little bit. On the straight AR side of things, what I would say is. .. You know, I think it was running really hot for a while. Kind of cooled off for a while. But I think it's slowly building back up. I think... If

you look at the S curve for innovation, there's that trough of disillusionment that I think some companies were in for a while. And now what you're finding... And I think what both Alex and I mentioned... Is as you start to find the utility of what these things can actually do and what these things can actually kind of build towards, that's where companies are starting to say... Oh, hey, wait. That's actually really interesting, and that's what people should be using this for.

It's not... Media and games -- it's not games or entertainment, necessarily, but it's these utility pieces that you can kind of build off of. I do think a lot of investors are kind of coming around on that. MERYL: We're almost out of time. So if you can give a quick answer to this one: About translation subtitles and the glasses. Have either of you made inroads in this area?

PAUL: Yeah, happy to go first on that. Yeah. So we actually offer translations on the glasses. So I don't have the language list in front of me. But it's kind of the standard EFIGS. English, French, Italian, German, Spanish, Korean, Chinese, Japanese. I think those are the eight big ones. Are all supported. So basically you can go into the device and say... Hey, I want whatever language is spoken to be transcribed into whichever one of those languages I prefer.

And then as people talk to you, it will get translated into that language on-device.

ALEX: The short answer for us is. .. Next. When we pivoted to go from cloud-based to local, to be really just building an appliance, we sort of had to think differently about which technology we were gonna use. And so we're starting to develop our own speech recognition technologies.

Which is great. And awesome. And exciting. And scary. But it's going really well.

And so we're starting to look next at. .. Okay. How do you then add translation to that? So we know we can. We're just not there yet. And so I don't have any details. But it was a great thread that you had maybe a couple. .. Maybe a month ago, Meryl, about Zoom and translations. And Zoom was holding back on offering caption translations. And they were charging for that. And your post was about. .. Well, what is that? ! Here we go again with Zoom charging for captions.

And it's interesting. Because I never thought of translation as an accessibility feature. And then when I really dug into it, and thought about it, and learned about it, thanks to you, I was like. .. Okay. What we're really talking about is enabling communication between people.

And if there is a gap, then that is exactly what we're talking about. You can call that... That is an inclusion gap. And so offering a solution to translate is sort of. .. Fits under the same umbrella. So yeah. We're definitely interested -- just sort of a day-two feature for us.

MERYL: Thank you both for coming and thank you, everyone, for coming to this. And I know there are a few more questions left. I would talk to Alex and Paul to see if they would be willing to answer them offline. And maybe I will post them on my LinkedIn, and Thomas's LinkedIn, so you can see the answers. So...

Have a wonderful rest of your day. And thank you again for joining us.

ALEX: Thank you. PAUL: Yeah, thank you.