Paradigm Shifts in Interacting With Virtual Environments: A Focus on Deaf People | Accessibility VR Meetup Recap

A11yVR with Wendy Dannels

Zoom has thrived since March 2020. Obviously, COVID-19 prompted an interesting fundamental change in how to socialize and network with people remotely. But what about the 3D virtual space?

Professor Wendy Dannels will point to some challenges and opportunities deaf individuals using American Sign Language as their primary language have faced when navigating and interacting with others through Mozilla Hubs.

She discusses ideas on making VR social platforms more inclusive including working with host and access service providers such as ASL interpreters.

Wendy A. Dannels is an Associate Research Professor and the Director of the XR Accessibility Solutions Laboratory within the NTID Research Center on Culture and Language at Rochester Institute of Technology. Her team develops real-time, immersive, and interactive technologies including mixed, augmented, and virtual reality with a focus on accessible and inclusive experiences especially for deaf individuals.

Here are some of the highlights from this presentation:

Transcript: Paradigm Shifts in Interacting With Virtual Environments: A Focus on Deaf People - Wendy A. Dannels

Wendy Dannels: Awesome. Thank you so much everyone, this is Wendy speaking through the interpreter Kat. So we will go ahead and get started. Hello everyone. Thrilled to be here. And first of all, I want to thank you so much for inviting me to give this presentation today. This is a very exciting time, lots of different things happening, and lots of different features that we're adding, testing them, seeing what works, seeing what doesn't work.

So, for example, just about 15 minutes, or half an hour ago, we were trying to test the capability of streaming the Zoom platform onto the Hubs screen, so that you could see both me and the interpreter at the same time. Unfortunately, that didn't work. Just as I predicted, it, unfortunately, was not able to be delivered in real time, it was a little bit slow and choppy. So, we decided to drop that ,and that's why you can only see me, and the interpreter is only visible in the Zoom room that we're in. But I think that's a good example of, you know, the research process, we try things out, we see what works, we take copious notes, and then we can inform Hubs on what could be improved in the future.

All right, you can see my title Paradigm Shifts in Interacting With Virtual Environments: A Focus on Deaf People . First, I want to give just a little bit of background about myself. My name, of course, is Wendy Dannells. I am a research professor, and also the director of the XR Access Solutions Lab.

And I'm located in the center and culture and language. At NTID, my focus is on real time interactive and immersive technologies, and specifically related to accessibility and inclusion, and even more specifically related to deaf people. I'm sorry, I'm not an octopus, I cannot, at the same time both type in chat, and sign at the same time, and fly at the same time, and move around hubs at the same time. So, I am taking a very simple and lightweight approach to this today. So, thank you for your understanding. And later, I would be happy to give my contact information to you if you would like. Again, the PowerPoint should also be very clear so that it's easy to see both the PowerPoint and myself presenting at the same time.

I imagine people gathering on Hubs, and people coming from many different walks of life. I think that it's also important to give another snapshot, which is again to narrow down to exactly where I work. I work at the National Technical Institute for the Deaf NTID which is one of nine colleges, including two credit bearing units, of Rochester Institute of Technology, and located in Rochester, New York. There are more than 19,000 students at RIT, and more than 1100 of them are deaf or hard of hearing. There are several different colleges that are part of RIT, we have quite a large campus, and then there are also two international campuses as well in other countries.

Why NTID is special is because it is very accessible. It is the largest Technical Institute for the Deaf in the world. Because of that, NTID provides a number of staff interpreters. Last year, there were over 82,000 hours of interpreting services provided, just in the classroom, and then another 35,000 hours of interpreting that were provided outside of the classroom for other activities, and that's only in one academic year. So now you can see why I say that this is the most accessible university.

Now I want to narrow down even a little bit further to the deaf population at NTID to give you a better idea of the population itself, who are the deaf students that we have on campus. About 40% of the deaf students use hearing aids. About half of them have cochlear implants. About 14% of them use both.

And then 14% of them don't use any hearing assistive devices. Personally, I don't use either hearing aids or cochlear implants. The reason I mentioned that is because I want you to understand that we have a very diverse deaf student population on campus. Next we'll talk about communication modes at NTID, 38% of the deaf students can speak and use spoken English, 37% of them rely on using ASL, American Sign Language. 19% of them use both spoken In language and sign language, and then 6% use other types of communication, maybe other languages, or something else outside of those two. But again, here I want to emphasize the diversity in our student population.

And my work is only focused on deaf people who use sign language. There is a lot of discussion, a lot of publications, and a lot of research on language, but my focus today is not on language more generally, but specifically on ASL, and using an interpreter with ASL. And, excuse me, my research is not on captions, but on ASL and on interpreting.

I also want to emphasize the importance of words, the words in my abstract that I'd like you to keep in mind. So, I focus on 3d. So, the X, Y, and Z axis. That's one thing that's unique about my work.

The second thing that's unique is the integration of American Sign Language. And then also related to discussions about the Hub, and service providers using a Hub, and how interpreters as well can use the Hub. Because of the COVID pandemic, we were forced to try lots of new things very quickly to experiment with a lot of technology on the fly. And I think that really was a blessing in disguise, because we learned a lot from it. I mean, look where we are today.

I also want to take just two seconds to focus on this picture that you should be able to see on the screen next to me, I'd like to ask what do you see here? You can see this picture looks a bit cartoonish. The eyes? A mouth? How it's kind of in a straight line.

Very standard.

Question: is that deaf friendly? Is that picture deaf friendly? That's something for you all to think about.

Now, I will be using the next slides to explain my story, beginning with last fall. To help you get a sense of what a person who is deaf or hard of hearing experiences commonly.

These experiences that I will be sharing with you all, as you're listening, I want you to think how, moving forward, you can help others, and you can improve things for others moving forward. The goal is for us all to work together for a better future for everyone. Last year, early fall, RIT had an annual symposium called Frameless. It's an annual symposium that they host and typically on campus traditionally, but because of COVID-19, they shifted to completely online. They sent the information to people who would be interested in being a moderator for various presentations. I decided I would volunteer. I could moderate one of the seven sessions for the symposium. So, I signed up to do so, and guess what happens? I'll show you. They sent me a copy of the

schedule, which I took a look at, and I was assigned to moderate three different presentations, different colleges across the country. But then, at the bottom, I saw something that said 12:30 AM was a meetup in Hubs with the presenter. I thought that should be interesting. I wasn't sure how that was going to work. From the information I received, I learned that all moderators would plan to have a Zoom room, and a presenter would come in, and do their presentation, and then the next presenter would present, and then the following presenter etc. And then everyone in the audience from zoom, as well as the presenters, would then move over to Hubs for that social time for questions and answers and discussion opportunities. And that was the concept, and the idea for the entire symposium. That was the format planned.

I'll show you the next slide. I decided immediately to call the interpreter for the hubs on campus to come -- for the person, access services, to come to meet with us, because this was new. I needed to figure out how I would be able to interact and socialize within hubs. This was new for everyone.

You can see this picture here. That was the first time we tested this, and had discussions about what we wanted to try, what might work, what might not work, and figure out what we might be able to do. But during that period of testing, we did come up with a few ideas. One example was that all committee members for the Frameless symposium would wear a specific color shirt, so that we would know that we were moderators. That was one potential solution.

And then a second potential solution. Now you see another avatar here. It's a really bright yellow.

You can't miss it really. And there's a big label that says interpreter on their front on their arm, and on their back. Interpreter, clearly seen from every view. The reason for this was because many avatars would be mingling in this space, and we wanted deaf individuals to be able to easily identify the interpreter, and tag onto them to be able to use them in that space.

So we had those first two ideas. And then thirdly, we thought that the people who joined the hubs might not be very experienced with this platform at all. So, we needed to think about how to address that, and how to help people navigate hubs from the git go. We needed to help them learn how to call the interpreter, and use the interpreter in this situation. So, we started online documentation, that you can see examples here, of the three potentials. The first is, instructions for those who use sign language like myself, who might be deaf. The next, guide for interpreters themselves, and how interpreters can navigate within Hubs. The third, was to provide a zoom link, and that would allow for communication if needed. We thought we would post that information, and when a person enters Hubs, you

will see the chat right away that says, Please click on this zoom link, please click here, and so on. So, right now, in this Hub, you see a big screen here, where I'm signing, you can see me presenting.

But, last year, instead, we had zoom projected into that space. So, someone could go to that wall, where the Zoom was posted, and grab onto it, basically, and connect it to themselves, their avatar. So, then I would walk around with the Zoom video along with me, and the person in the Zoom screen that I'm pulling along with me is the interpreter. So, basically, I'm walking along with the interpreter, alongside. So, you can see this picture, about here, showing when people enter the room, and you see the yellow right away. And then, on the screen, you see what to do.

The picture on this current slide is after several iterations of playing and testing with this to find the best picture of a good small group chat. You'll see people moving around in the space, and looking at each other. And then you'll also see the interpreters. Really, people just want a little bit of practice with it. So, on that screen, you see four people in the screen. Two of them are the interpreters, the one on the top. The second one is quiet not moving, really they're saying pretty still, but it's a new experience, again. So, the top left picture, of the interpreter, they were really motivated trying to figure out how to do this, and how to learn how to do this new technology. The interpreter on the bottom has many years of experience in the field, but not so tech savvy, and

they're very unsure of what to do and how to navigate this space. It becomes up to this tech savvy interpreter to take over sometimes, and run with it the whole way. Another thing that you see on this picture here of the Zoom screen, it's also to make sure that I don't block other views for other people. With my screen, here, it's down in front of me so that I can see other people, and see the space that I'm moving around in. So, something to think about here.

Suppose I'm walking somewhere, and I start to walk to a different space, and the interpreter loses track of me, and then they have to follow me, and track me, and catch up with me, they walk around with me, basically. It's kind of like an interpreter being at the center with their joystick kind of have to navigate in the space, while they're also trying to interpret. That can be a lot. So having to navigate in the space, as well as interpret,at the same time, or sign, communicate, can be pretty tough. It can be tough, it's a lot of work. But this experience I was just talking about was with a small group, I was the only deaf person in a small group of individuals.

Unknown: This would be good for maybe one on one interaction, or potentially large situations, but we might need more interpreters for that. But situations change, and strategies have to change based on the situation, you have to be flexible.

Wendy Dannels: So, that's the big picture from November 2020, starting at that point in time. But now I will shift my focus to talk a bit more about something a bit different now. I'll have you think about the deaf community. It's important to understand that deaf people want to be a member of a community, and contribute to the deaf community, and deaf culture. That's important to deaf people. Their primary language is sign language, here in America, American Sign Language. The National Association of the Deaf explains that ASL, American Sign Language -- ASL is the shape, the placement, the movement, as well as facial expressions, and body movements. All those five things I listed come together to comprise ASL, and that is how we convey our information. It has its own grammar, its own grammatical rules, and it is unique. It is special to the deaf community.

Now focusing more on deaf culture. Our primary sense is our eyes, that's what we use and rely on above everything. Eye contact, or the ability to see other people, is very important. If a person wants to get someone else's attention, they typically will bang on a table, or wave their hands, or tap the other person on the arm or shoulder, or maybe walk across the room to flash the lights, to get someone's attention. That is part of deaf culture. We become used to that, very comfortable with it, it becomes part of our lives.

Now, remember that the examples that I give do not apply to every deaf individual. It is important to remember how one person interacts is different from how another person interacts, and how you would interact typically, is that how we would interact in hubs? It is not, it's completely different. So, there are different communication approaches that we need to look at. We need to think about language grammar, the chat option as you see in this Hubs room right now. You'll notice the video is pretty large, you can see me pretty large on the screen. If you're looking at this on your phone, you'd probably have to squint to see it. You probably wouldn't be able to see that video very well. So that's something that we would have to take into consideration - the size of the video.

Another thing that needs to be taken into consideration is when you enter hubs and you start looking around, it's important to have some type of help center, or a help desk, or help person, that you can go to and figure out how to get help from there to be able to navigate the space and use it appropriately. Another example, real world example, is in a classroom where an interpreter is present. It's a large auditorium classroom and the deaf student is in the front of the room. The front row, the interpreter is interpreting upfront. Moving around the professor, the presenter. And a student sitting behind the deaf students has a question or a comment. It's very common for an interpreter to say that person with the pink shirt back in the far corner, and then continue interpreting, they will

give a little bit of information about where in the room the person speaking is located, and what they look like. So that the deaf person me, for example, could take a moment and look back to find the person who is speaking, or just imagine where they might be in the room, and make those connections with what's actually happening. The reason I mentioned this example is because of the space, we think about this hub space, if people are behind me or beside me or in front of me or above me. They're all in different places. Not only that, but our attention is not focused in one space, one location, it might move around,.

Unlike owls, we cannot look all the way behind us, we cannot turn our head all the way around to see what's behind and beside us. I can't do that. Unfortunately, none of us really can. So, instead, we have to think of different options, or different approaches, to addressing these various challenges.

And these challenges can lead to good opportunities. Alright, so next, I have a question for everyone here. When you came into this Hubs room today, when you first came in, did you receive any instructions? Did you get any information about how to use it? Did they ask, Do you need any accommodations? I think you already know the answer yourselves. That's also something that we had to think about. So, as the host or the coordinator of an event like this, an event taking place in Hubs, has to think ahead of time about who will be joining this room, and what might their needs be. And it's very important to ask people what they might need in the space because it takes time to prepare any necessary accommodations.

And although it does take time, again, it goes back to the quality of service and the accessibility which is so important. So, that's something to keep in mind, especially for anyone who's hosting or coordinating an event that's happening on Hubs or in virtual space. It also, again, requires planning ahead of time. Planning something with just a day ahead of time isn't enough time, it really needs to be several weeks. Even just, for example, the interpreter, the interpreting agency, may not understand what Hubs is, we may have to do some troubleshooting to make sure that video is working and audio is working. How do we take turns? How do the interpreters take turns? How will we manage q&a?

There are a lot of different details and logistics questions to think about ahead of time. So, then now, if we're thinking about interpreters specifically, and service providers, they will typically look for the person that they are assigned to interpret for. An interpreter will get their assignment number, they'll go to that place. But the interpreter may not be familiar with Hubs. So again, it may take some instructions, it may need a rehearsal, there may need to be a dry run, for the interpreter to become acclimated to this type of space. And that's very important to think about.

One big challenge that I have have noticed so far, is how interpreters can follow deaf individuals. Without, again, having to deal with that issue of both navigating with your hands using the keyboard, and signing or interpreting at the same time. That has been a challenge. And that's one of the key areas of concern that we need to solve. I think that once that's resolved, that will increase the communication capacity and the communication flow for all of us.

So now, if we shift over to VR, we need to think about the headsets, VR headsets. So, whether the headset is a fully immersive experience, whether you can see the environmental round view, or not at all, it is exciting to be in that type of environment, it's very cool. And really, the first time people do it, it usually blows their minds. It is very cool.

But there are a few things that we have to consider about that type of environment. So, for example, if I'm using a fully immersive headset, but someone is at the door ringing the doorbell, I wouldn't know.

And so, that's one of the types of details that we need to figure out, how to get VR systems to provide that type of information about what's going on environmentally in real space around you while you're wearing an immersive headset. And that's just one example. There are many more. I also want to tell you a quick story. This happened a few years ago. A group of people were excited to create something new. And during that era, this was around 2014 to 2019 or so, one of the ideas that came out of that time was signing gloves. People submitted competitions, they wrote grant proposals for it, they developed prototypes, we saw this pop up all over the place, these gloves that could translate sign language. And it seemed very exciting for a time.

But, however, in the deaf community, we realized that it really wasn't going to work. Because we know all of these other features of ourselves. The Atlantic magazine actually has a very good publication that explains why sign language gloves don't work. It is a beautiful explanation. And there's a quote in that article that says, obviously, they did not check with the deaf community.

So, I want to emphasize that again, as well. Anytime that you're developing content, creating something new, even on a VR platform, please keep that in consideration. And again, don't forget that deaf people use different communication modes, deaf people have different hearing abilities, and getting feedback from diverse deaf people will lead to a better product. So now, if we look towards the future of VR, the future work. Let me share a concept with you. Of course, Earth E-A-R-T-H. But let's switch that to E-Y-E-T-H Eyeth. So, we'll change the idea of hearing to the idea of vision. So, this is an alternate world we can imagine where no one uses audition, no one hears. And people who do hear are considered to be a little bit different, maybe a little bit odd. Where people who can't hear anything are able to develop content and live their lives

freely as they wish. So now, some of you may be wondering about that. And I just want to share a couple future predictions related to Hubs or other social interactive platforms. Here I'm showing on the slides one product called j5create, which is a pretty good 360 degree camera, that allows a full 360 degree view, and it can be plugged into anything. But it allows a person to see entirely all the way around them. And so, with this type of camera, it would be easy to identify who's talking, who's behind you, for example, especially for me as a deaf person. And there are also different settings that can change the type of video that it streams. So, I would encourage you to take a look at this product, and also to think about developments of other products like that can improve accessibility for deaf and

hard of hearing people. So, this video camera is currently available, then the next slide that I'll show you is not currently available on the market, this is completely new. This has not been released to the public yet. It will be in beta testing in the first quarter of 2022. And this is an actual 360 degree camera that can also provide real time video of its entire surroundings. And this new product is being developed by a number of different companies that have received funding for it. And it's very exciting. This company has received over 20 letters of intent from people who will participate and help develop this product. And it's really being geared towards the [inaudible] marketing. I'm very excited about this, I'm looking forward to a day where I can be able to see avatars who are real interpreters,

using my natural language of ASL. And that way everyone can have conversations with each other on an equal playing field. So I will close my presentation here. 7:50. I'm right on time. And at this point, I'm happy to open it up to any questions. And I will let Thomas take the lead now, or whoever else would like to facilitate q&a. Please let me know how we should proceed with q&a at this point.

Thomas Logan: Yes, I'll help facilitate it. So, I asked, is the yellow sign language avatar available to reuse? It was shown from the Frameless VR?

Wendy Dannels: Yes, yeah, you just it really is simply just a yellow avatar, so you can easily make your own? Absolutely, it should be very easy. There are instructions available that explain how to make custom avatars. So, it's really just a question of making the avatar yellow, and then adding that extra text to it.

Thomas Logan: Excellent. And now we have a question from David, has anyone considered how you can use the same sign language, gloves, technology, but rather than attempting to convert gestures into text speech, but provide a virtual representation of the signs in the VR environment?

Wendy Dannels: That's an excellent question. I had a feeling that someone would ask me about that. So avatars are very cool. That technology now is not quite there yet. It's not quite good enough to allow for real time signing with all of the details that are necessary, it can't do that smoothly yet. And so people now are thinking more about sign language recognition. There have been a few different articles and publications here and there. A 3d spatial model of ASL has not yet been created. I do know of one program that allows a person to use a webcam, and then it can identify fingerspelling and create specific hand shapes in a virtual space. And it's pretty good, but it is very slow. I think that is possibly something that could happen in the future. That's not something that I'm working on. I

prefer to have real time video personally.

Thomas Logan: This is a non- technology question, but I had a deaf advocate question because we some discussion is in part of your presentation. I had an experience of this is actually just a couple of days ago, riding a bus in Japan. It's usually very quiet, and there was sound coming loudly from a person's mobile phone. And when I looked over, I saw them signing on FaceTime sign language friend. So, my assumption is they they didn't know that the sound was playing out loud. What would be a polite way, or what should be an interaction be to let someone know that's happening?

Wendy Dannels: That is a really excellent question. Thank you for asking. You're right often deaf people unintentionally may have their microphones on. I've done that myself in zoom, it happens. And so, it's typically polite for someone in zoom, for example, for someone to send me a private chat just to say, Hey, did you know that your microphone is on? I'm not sure that you meant to do that. And so I think the same thing could be done. On a bus, for example, you could write a short note and just say, just for your information, I just wanted to let you know that your sound was on, in case you didn't realize it.

And, most of the time, deaf people will be happy that someone told them, and they'll be happy to turn off their sound.

Thomas Logan: Yes. Cool. Thank you, and I need the Japanese translation of that. That's what I will work on getting for the future.

Wendy Dannels: Thomas, that's your homework assignment.

Thomas Logan: But, yes, it really will be. I have a meeting with Japanese teachers. So, I'll learn that. I'll ask one more question. fFom Kristen Lloyd, as many people use VR spaces for personal use, online games, I feel there will be a need for cartoonish avatars with the capability to have full communication. How long do you think we have until we can have hardware that can take facial expressions, and present them in the virtual space?

Wendy Dannels: That is another good question. So, body movements, facial expressions, distance - both vertical distance and horizontal distance - all of those things come together to form language. A few years, at least, I would think. I can't promise anything. I don't know one way or the other. I would like it to be sooner. But I'm really not sure that's possible.

Thomas Logan: All right. And, thank you, Wendy, we have one last question that we will take today, from David. In a previous A11yVR project meeting, there was discussion whether live captioning for participants in a meeting should be positioned directly under the presentation, rather than directly under the avatar speaking,

Wendy Dannels: I think it depends on the goal. And it depends where the avatar is, as well. The best thing to always do is to ask the deaf person in the audience what would work best for them. If it were me, for example, I would maybe prefer the captions to be next to the person who's speaking. However, the captions also need to be in front of my visual field of view, as well. So, if a person is speaking behind me, I would want to see the captions in front of me, if I needed to look ahead. Also, it's always helpful in captions to have the speaker ID, so that I know who's speaking. And so, I think the most important thing is to ask the person from the deaf community who's using captions, what their

preference is, many people have different preferences, and they have good reasons for their choices.

Thomas Logan: All right, let's say thank you so much to our presenter tonight, Wendy Dannells. We really appreciated the presentation tonight. And we are taking notes, and even at the very beginning, thanks to Wendy for kind of helping clarify and something we want to work with the Hubs team on for thinking about, streaming, maybe, Zoom into this space. But, I've definitely learned a lot again, let's end with a big thank you for Wendy.

Thank you, Wendy.