Audio Descriptions for 360-Degree Video: Recap of Webinar

Van Gogh Starry Night with observer in virtual reality headset

On Tuesday, March 14th, I hosted a webinar with Chancey Fleet titled, “Live Audio Descriptions for 360-Degree Video: Best Practices.” The webinar gave Chancey and I an opportunity to share our research on audio descriptions (AD) for immersive media, and start a conversation with the wider accessibility community about where to go from here. 

Our Research

The primary question we posed was this: Under what circumstances could there be a benefit to having a live audio describer for content that was not originally designed with accessibility in mind? Could a live audio describer be useful for remediation? To begin answering these questions, we conducted research with the help of Blind volunteers at Data & Society and the Georgia Libraries for Accessible Statewide Services. Three different immersive experiences were presented to volunteers, along with a different type of AD for each one. Afterward, volunteers were given an opportunity to rate their experiences and share feedback with us. 

Video Highlights

Transcript: Audio Descriptions for 360 Degree Video: Best Practices

Thomas Logan: Well, hello, everyone. This is Thomas Logan from Equal Entry. And I want to welcome you to our webinar today, Live Audio Descriptions for 360 Degree Video: Best Practices. We have been very excited to give this presentation.

This presentation was originally planned for CSUN 2020 and we weren't able to deliver it there. So, we're very happy to be able to present the presentation with you all today. A few things before we get started. We have live captioning today provided by Amanda Lundberg from White Coat Captioning.

Is everyone able to hear me? I'm seeing a comment in the chat. You can chat if you can hear me? Okay.

Good. So, we will have the recording of this presentation go out to everyone that's registered. We will send the slides, the recording and the transcript that's produced. We're planning today for about a 40 minute presentation and we will have 10 to 15 minutes at the end for questions and answers.

I will be monitoring the chat window. So, type your questions at any time during the presentation, but I'll be moderating and giving those to our presenters at the end of the presentation. Again, thank you all so much for joining us today, and I'm now going to hand it over to James Herndon.

James Herndon: Thank you, Thomas. Hello, everyone. My name is James Herndon and I work for Equal Entry as a technical writer and accessibility consultant. My work focuses on creating media alternatives that includes text alternatives, captions, and, of course, audio descriptions.

And I also write and produce a number of training videos about accessibility for various companies. Right now, I'm working on audio descriptions for a virtual reality film installation called Queerskins, which was to be on site at the Tribeca Film Festival this year. Hoping for a new date after the quarantine ends. So, audio descriptions for 360 degree video and for VR have been on my mind quite a bit lately.

Excited to dive into the webinar to talk about the work Chancey and I have done together to talk about the different approaches one might take to provide audio description for the new medium. And I have a lot of demos to play for you all today. I'm doing this because I want to contribute to an ongoing conversation about all of this. This is obviously all very new.

And audio describers, including myself, are still trying to figure out the best way to approach this problem. And getting feedback and having a larger conversation with the community will, of course, be essential to making the right determinations. And now, I'll let Chancey introduce herself.

Chancey Fleet: Hi, my name's hi, my name's Chancey Fleet. I'm an affiliate at Data & Society research Institute and assistive technology coordinator at the New York Public Library. And one of my interests around advocacy is visual interpretation and description of visual elements in media. And so, I was really excited to have the opportunity to work with James to explore what it might mean to have more equity via live description or pre recorded description inside an immersive 360 degree content.

There was a symposium last year, and since that time, I have been paying attention to kind of the rise of immersive VR for applications in gaming and employment and education. And I know that as each year goes by, the prevalence of VR and XR will continue to rise. And this pandemic probably meant that it's going to rise even faster. So, I think that this subject is really timely, and it's been a pleasure to collaborate with James to figure out what works for folk when is it comes to VR description and what doesn't.

James Herndon: So, moving forward, I imagine a question some of you might have off the bat is: Why use live audio descriptions for 360 degree video? Under what circumstances to have a benefit for a live audio describer for content not originally designed with accessibility in mind. Could a live audio describer be useful for remediation and for non remediation? Just for the best experience.

So, some of the reasons that we wanted to try live is that live audio descriptions can be used during a test phase to identify publishable solutions. Right now, there aren't too many readily available tools to produce 360 degree audio descriptions. What Immersive Accessibility is doing with Anglatechnic is great. They're an organization I recommend to learn more about the tools and techniques they are building.

But again, it's all still very new and not been adopted into widespread use. People are still trying to figure it out. Another reason to try live audio description is that visual changes in 360 degree video can sometimes happen too quickly for programmatic description. For example, say you have a 360 degree video, a GoPro video, of a drone flight over Hong Kong.

Visual changes could be happening so quickly that programmatically stitching descriptions together into a unique sequence for each user in real time without interruptions or pauses, that might be technically challenging, depending on how many descriptions were available. And it could be difficult very difficult, potentially to provide standard audio description in that way. Which based on my experience is what most users prefer. They don't want pauses.

Another reason to try live is that it allows the describer to act as a kind of docent in virtual spaces. As we all know, accessible technologies get adopted more readily and are taken more seriously when they also benefit people who aren't disabled. It's sad, but it's true. And so, framing the technology and the amenity as a docent in virtual spaces is a way to get buy in from people who do not care about accessibility.

And this pandemic is the perfect time to make that push, as Chancey said. So, we can make that ask of culture workers to go ahead, explore virtual docent as an option. And you can bring accessibility into your organization that way. Chancey, did you want to add anything?

Chancey Fleet: Not right now. Go ahead.

James Herndon: Okay. So, the first video I want to show is called National Geographic: Great Hammerhead Shark Encounter. This is a 360 degree video with over 27 million views on YouTube. We partnered with Data & Society in New York to test audio descriptions for this video with seven different blind people and get their feedback.

Each person came into the testing area one at a time. They wore an Oculus Quest headset, allowing me to see what was in their headset's field of view in real time. And I read add you descriptions out loud to them from a script. For my script, I created a template that divided the video into scenes.

Each scene began with some introductory description. And from there, I where wrote a script for each possible direction that the participant might choose to face during that scene. Forward, backward, left, right, up, down. So, for the first clip I'm gonna play, the participant, we'll call Andrea, was facing forward during the first scene.

And in response to that, I read the description prepared for someone facing forward during that scene. In terms of standards for writing the audio description, I usually refer to the Described and Captioned Media Program. Otherwise known at DCMP. According to them, a description must be accurate, prioritized, consistent, appropriate and equal.

For those interested in learning more about writing audio description, 3Play Media has a number of high quality public facing documents about that. When it comes to writing audio descriptions for 360 degree video, the only significant linguistic difference was the use of second person address. And I did that because immersive video is intended to make the user feel as if the media were something that they're actually experiencing. So, I chose to use the word "You" to emphasize that this was all happening to you, the user.

So, let me play you a clip.

Describer: National Geographic logo. On tropical waters, you face a scuba diver at the edge of a boat, and together you jump into the ocean. As you descend to the bottom, on screen text says “Bimini, Bahamas.” Light blue water collapses into dark water in the far distance.

A small bait box hangs from the boat by a thin cable. A second diver points a camera at nurse sharks and other small fish swimming along the sandy ocean floor. Onscreen text says, “57 miles east of Miami, Florida.”

James Herndon: Here's an example of what happened when another participant, Latrelle, turned around to face backward after jumping off the boat. I did it accordingly and read it for her facing backward.

Describer: National Geographic logo. On tropical waters, you face a scuba diver at the edge of a boat, and together you jump into the ocean. As you descend to the bottom, onscreen text says, “Bimini, Bahamas.” Light blue water collapses into the dark blue water in the far distance.

Two gray green nurse sharks swim along the sandy ocean floor. Onscreen text says, “57 miles east of Miami, Florida.”

James Herndon: One question of this approach is questions about so called essential content. There's a scene near the end where a hammerhead shark bites you in the face. It's the dramatic finale of the video. If you're facing the opposite direction, onscreen text says, “turn around.”

Obviously, the describer needs to announce the onscreen text. But one issue we ran into is how to tell someone that if they don't turn around, that they're going to miss something important. And not everyone chose to turn around because they didn't have to. So, let me play that experience.

Describer: Onscreen text says, “turn around.” Near the sandy ocean floor, there's a nurse shark and stares back at you. Something white and fleshy slams into you from above, knocking you flat on to the sandy floor. The diver swims to you.

Jack fish and nurse sharks swim by from left to right.

James Herndon: And now I want to play a clip of a different participant, Nefertiti, who turned around and enjoyed the onscreen finale.

Describer: Onscreen text says, “Turn around.” A hammerhead shark swims slowly along the bottom, slowly coming towards you. It bites you in the face.

Nefertiti: Oh, my god!

Describer: An underwater cameraman swims over to check on you. A second diver picks you up.

Nefertiti: Oh, no. I like my face!

James Herndon: So, in terms of the outcomes for this particular video in our research, all seven participants said that they were able to follow and understand the descriptions and they found the descriptions sufficiently vivid. But the sense of lack came more from the nagging sense of possibility. And the idea of missing something just out of view. Because you couldn't interact at all.

And that's the lesson I applied to descriptions for the next two videos which I'll be playing shortly. Another challenge was that people said that they didn't want to be rude and interrupt you me while I was speaking the way you could, you know, easily feel you could be rude with voiceover, your screenreader. You don't have to treat it like a human. But with a lot of describers, people didn't want to be rude and move away or turn away from me while I was speaking.

Most people would be facing the same description for a fairly long period of time because I was speaking and they didn't want to, you know, be rude or turn away while I was talking. Another striking statement that we received was someone said that that description was a lot like video, and so, it didn't feel real. The lack of interactivity in 360 degree video is that to a blind person, it's in more immersive than Netflix.

In the larger project of accessibilities in the near term, we need to describe the content because people need to have access as Equitably as possible given what we can achieve. This was not an equally immersive experience. So, that brings us to our next video. The Starry Night in 360 degrees.

This is another popular video on YouTube. And I wanted to try this one because it's animated. And that would allow us to try something very different from the documentary style video that National Geographic created. I also wanted to try this video because the video keeps you in constant motion as a opposed to the relatively static position in the great hammerhead shark encounter.

And we wanted to get a sense what have that experience would be like. I tested this video at Data & Society too using the same scripting approach as the great hammerhead shark encounter. But I partnered with the Georgia Library for Statewide Services to test with three blind fellows there and get their feedback on what I call the "Optimal Path" approach to remediating a 360 degree video. And by "Optimal Path," when I say that, I mean audio description based on a screen recording that deliberately seeks out the most essential content in a 360 degree video.

Now, again, this was a video that was not designed with accessibility in mind. We're talking about an after the fact remediation. Which is not ideal. In the best case scenario, each person could decide what's essential to them and what's not.

But this falls under the something is better than nothing category for remediation. In terms of the script template, I found doing this was almost nearly identical to writing standard audio description. I provided as much relevant detail as I could within the time constraints of each moment in the video. There's no dialogue in this video, which simplified things somewhat.

But because the video keep use in constant motion, I needed the object to be for the most part described within the time is existed in the field of view. So, let's play a clip from the "Optimal Path" approach.

Describer: As you approach the yellow two story house with green shutters, its green door opens for you, revealing Van Gogh's bedroom.

James Herndon: So, the logical follow up would be, well, what's a non optimal path. I want to play a clip of someone at that same moment in the video entering Van Gogh's house. That's a useful reference point because if a participant was facing the opposite direction from Van Gogh's house, they would enter backwards. And that transition into the house could be awkward or confusing.

Let me give you an example. Here is someone going in backward.

Describer: A door creaks open behind you. You move backward through a doorway into a room with smoky blue walls. You turn around and find yourself in Van Gogh's bedroom.

James Herndon: So, in terms of the outcomes here, having a so called "Optimal Path" for 360 degree video had a higher satisfaction for the group at Georgia Libraries compared to the group at Data & Society who received the more open ended description in the same style as the hammerhead encounter. For the Georgia Library, the effect was a little bit closer to a conveyer belt moving past the crown jewels in the Tower of London.

They're in constant motion and you don't get agency. But they don't feel as if they were missing something. Which is very common in 360 degree videos for everyone. But especially for people who rely on description.

But for the folks at Data & Society, constant motion without choices or a defined optimal path was just not satisfying. The striking quote I received was, “I felt like I was on a runaway train.” Another striking comment I heard from the Data & Society group was, "There's no home base.” And by that, they meant this video, like all 360 degree videos is in its most basic form non interactive.

You play it like a 2D the video in the description. For non interactive video like that is in constant motion, that's a challenge for a blind person. They can't reset things. They can't turn their body back and get their bearings again.

Because the scene in the field of view is completely different every few seconds. You're always being pulled into something new whether you want it or not. Chancey can speak to some of the other laws inherent to this experience. Chancey, would you like to do that?

Chancey Fleet: All right. Hi. Took me a second to unmute. So, one of the one of the pieces of feedback that we received from participants and something that I experienced myself is that although providing described access is unquestionably the right thing to do and it's more equity than leaving a VR stream inaccessible, the illusion of immersion doesn't survive description.

It's more akin to watching a movie that's described where the action is moving and perhaps your perspective is moving. And then there's a describer who is doing the voicing. Someone commented that that it felt almost like television. And at first, I think James may have thought that that was feedback about his construction of description or style of description.

But as we reflected about it, we realized that it is just a natural to some extent a natural limitation of the format. The description is going to be voiced. It doesn't bring the immediacy that visually constructed VR worlds bring. I think the bigger challenge that we have to face as an industry and as advocates is that most research and development for virtual reality is going straight into visual experience.

And so, if there aren't haptic experiences, if there aren't sonic landscapes and other non visual ways of constructing environments, then even the best description that's crafted is going to become sort of analogous to description for film.

James Herndon: The final video we did was technically virtual reality given its interactivity. It was the Anne Frank House VR experience. And this is an app that's available in the Oculus store. It's also available for the Vive.

It allows you to explore the hiding space in Amsterdam that Anne Frank and her family lived in, which is now a museum. I tested this experience remotely with three people who are blind via Google Hangout. And for this experience, I was wearing the Oculus Quest headset and no one else was. They were only receiving my audio descriptions.

And because I was wearing a headset for this particular test case, I couldn't simultaneously read a script. And so, I had to memorize it. And the script template I created was organized by room. There are a total of eight rooms in the house.

And when a participant entered a room, I gave them a high level visual summary of that room and then a list of the selectable items available in clockwise order. And they could then ask for more detail about anything in the room if they wanted it. The first clip I want to play demonstrates the greater freedom of choice that this approach offered compared to the first two videos. This is for the clip I'm gonna show is for the first room in the tour.

And you will hear Peter, the first participant, make a selection. And after that, the clip shows the second and third participants, Nefertiti and Chancey making a selection from that same point in the tour. So, here's a clip.

Describer: A small narrow room above an old factory. Clean and well lit with a shaded electric light on the ceiling. A narrow, descending stairwell to your right, and tables and bookcases covered in books and binders. The selectable items in clockwise order are a quote icon against the floral wall paper to your left.

A bookshelf displaying a hand icon with text that says, "Move closer to interact." And a quote icon against the closed window to your right. Peter: The first quote.

Describer: Okay.

Anne Frank: Countless friends and acquaintances are on their way to a horrible destination…

Describer: What would you like?

Nefertiti: I’d like to move closer to the shelf, please.

Describer: Okay. I'm moving closer to the shelf. Chancey: Okay, let’s do the quote on the top of the window.

Describer: I’m selecting it now.

Anne Frank: Our hiding place has now become a true hiding place.

James Herndon: So, the second clip I want to show is the greater approach to movement. You aren't stuck in place and aren't on a runaway train. There are two floors in the house and they both circumstance a staircase. So, your exploration of the house can take a variety of paths.

And the path you take is entirely up to you. This clip I'm gonna show takes place just after we've passed through a secret door behind the bookcase from the previous clip. In this clip, I've already given the visual summary to Peter and now I'm listing I'm gonna list the selectables for him. You'll hear Peter decide in which he wants to go.

And then we'll cut to Nefertiti deciding where to go and then Chancey.

Describer: The selectable items are a quote icon above a coat rack, a door at the end of the left path, as described, the door at the top of the stairs, a quote icon located halfway up the stairs, or the door directly to the right, or the door behind you through which you entered. Peter: Let's first go up to the door on the left.

Describer: I'm opening the door.

Nefertiti: At the top of the stairs.

Describer: Okay, I'm selecting that.

Chancey Fleet: Let’s do the door at the top of the stairs.

Describer: I'm selecting it now. The selectable items...

James Herndon: So, the outcomes for this particular video, Chancey and I agreed that this experience was by far the most satisfying for all the participants. Unlike 360 degree videos, the more interactive format of VR provides agency and choice. And even though it's a mediated experience, the combination of getting feedback about your movements and the layout not moving without you or before you're ready to move forward, and the ability to get the audio components of the experience in response to things that you do.

All of this makes it seem a little bit more immersive. And hopefully there will eventually be a wide variety of simple modifications that people can make to a 360 degree video to create a similar sense of immersion and agency for blind folks that's closer to VR. One outcome that was really interesting to me was, in a lot of people's minds, what makes VR immersive is the headset. But based on the work Chancey and I have done so far, that has not been true.

Because for the first two videos in this project, the Great Hammerhead Shark Encounter and the Starry Night by Van Gogh, all of the participants were wearing a headset for those experiences. But they didn't feel very immersed because the audio descriptions were no different than one directional they receive when watching television. They didn't feel any more inside the experience than they would while sitting on the couch at home with the TV on.

But in the third experience, the Anne Frank House VR, I'm the one who is wearing the headset and yet it still felt way more immersive to every participant even though they don't have a headset on. That's when it became clear to me is what makes VR feel immersive is not the headset. What makes it feel immersive is the interactivity. So, when we think about conclusions and next steps, I want to let Chancey speak first here.

Chancey Fleet: Thank you. One thing that I wanted to add is that in that in that last video that was played about the Anne Frank House, one of the ways that you can deepen your understanding of a visual landscape through a description or interpretation is to dwell on an object. Part of having a gaze means not just moving through an environment at a set pace but being able to dwell on little details. So, for example, in the Van Gogh video, we passed through the green door.

Is it decorative in any way? Is there anything distinctive about the handle? And then there are countless objects in the Anne Frank House that an occupant would have had to dwell on because it was just a very small environment in which they lived their lives. And so, one thing that emerged for me is that I think, although VR description could perhaps be automated in the future to bring it to a level that's equivalent to what's available when you watch a film, in the same way that pre recorded, pre arranged description when you watch a film does deprive you of the ability to deepen your gaze and to question the description.

I think live description for immersive VR has a potential to to give you more autonomy about what you explore and the questions you ask. And so, as we think about the future, and in particular as we think about what VR accessibility might mean in this pandemic and what equitable access might look like, I think, as we explore our our virtual environments, it will be important to have a way to have live interactive description so that people who are receiving description can ask questions and examine things and zoom in, if you will, on things that are of interest.

And similarly, kind of zoom away or background their focus on things that they're not wanting to focus on.

James Herndon: Yeah. And I would I want to reiterate Chancey's earlier point about R&D. It's important to acknowledge that. Most research and development in VR is going towards visual rendering.

And less is going toward high quality audio, for example. So, in these particular videos, the audio is fine. But it's not nearly as exquisitely rendered in the way that the visuals are. And even more R&D is going towards haptics or kinesthetic touch and feedback.

It's necessary for folks in the industry to devote substantial R&D to the other modalities or it will always feel as if it's just a video. So, we're happy to take some questions.

Thomas Logan: And this is Thomas Logan here again. So, I don't have any questions currently in the chat. Please feel free, though, to send us messages into the chat for any, you know, discussion points or follow ups you would like to have. I'll kick us off with a question for Chancey.

Just as far as the feedback of the working group of James working with Data & Society and the participants. Any feedback on like lessons learned or things that could be improved or recommendations for doing like future studies or research?

Chancey Fleet: I think a lot of it we've already said. I think interactivity is going to prove to be more valuable than we thought. I think having the first person experience of wearing a VR rig like an Oculus and making movements that correspond to your desired gaze might be less important than we thought. I think providing for different if to the extent that there is automation, providing for different layers of detail will be important.

And I think the the conversational nature of the interactive description oh. Well. Not for me. Because I can't stare if I try.

For people that have the ability to focus their gaze, then maybe.

Thomas Logan: Chancey, could me pause just to read out the question. I think you were reading the question from I believe it's Joi. Do you imagine the automation of this sort of descriptive narration being triggered by the motion and gaze, turning and more descriptions, et cetera?

Chancey Fleet: staring is out for me and for a lot of us. Head movements could do it. That's a possibility. I would find that I don't know how I would find that.

I think that would work for a lot of people. Another thing that we talked about is maybe hand motion. Like maybe something like a game controller or an arrow cross. I think that that's a really exciting arena for experimentation.

But if you and if you're low vision, and I don't think we had too many low vision participants in this particular study. If you're low vision, you probably do want to have your own gaze and your own Oculus and have that enriched by the describer. That's a whole other area for exploration. But I think if you're completely non visual, you could just as readily use hand motions or head motions.

It will be interesting to see what people come up with.

James Herndon: I think one thing that will also need to be worked on if you're trying to increase the amount of interactive content available to people. One thing I've seen come up a couple times is there doesn't seem to be very good keyboard integration with the Oculus at this point. There may be some things that exist but based on the people that I've spoken with right now, that's definitely gonna be a barrier. And especially if interactivity, which right now in the Oculus is primarily pointer dependent.

And for bringing in other, you know, possibilities. Whether it's using the accelerometer in the Oculus to lean forward or backward to trigger things or use gestures as Chancey mentioned. That will be helpful. And within those steps, better keyword integration with the Oculus will be an important component.

Chancey Fleet: Someone says what about a virtual white cane. And guidance from O&M instructors? Okay. This harkens back to what we were talking about regarding soundscapes.

The soundscapes that were available in these particular videos, they were stereo. But they were not 360 degree audio. They were not equivalent to the level of detail and richness that the visuals provide. If you were going to do any virtual navigation in a virtual space, you would need haptic feedback from a controller that could simulate a cane.

And so, it would have to be fairly light. And it would have to be fairly precise and it would have to deliver back a range of different convincing haptic sensations. There has been some work on this. When we're done, do a web search for something called the Canetroller, it's a control we are a cane.

It's a proof of concept. Not totally there yet. And in terms of guidance from O& M instructors, if we're talking about people who are independent adults in their non virtual space, independent travelers, then I would just as soon experience my virtual environment without an O&M instructor. But if you're talking about developing skills in virtual space, then maybe.

It's been talked about. But we're not quite there yet when it comes to haptics and sound design.

Thomas Logan: Okay. I'm going it go up to the top of the questions queues here. So, from Mary Anne Funk. In the Anne Frank House experience, did the participant also wear headphones to experience the sounds that were associated with the movement through the space?

Chancey Fleet: Yes.

Thomas Logan: Okay. And how, from Alice Kruger, how would your description differ for blind people in 3D virtual worlds such as Second Life.

Chancey Fleet: That's a good question, for games.

James Herndon: That is a good question. I would probably need to consider a specific scenario in order to know how to answer that. That's a very good question. Obviously massive multiplayer online games are another huge area where things could be more accessible.

And there's been some interesting bits of accessibility in them that I've heard about. But I think, yeah, you're right. That is going to be something to consider. I would I would imagine that it's gonna need to be it's probably gonna need to be a couple different approaches you would take depending on the experience.

Because my understanding is those types of games have a wide variety of, you know, user experiences within a single game. It's not necessarily one path, one modality where you go through and do this. Some of these games have like a higher level meta game and then smaller mini games. And there's just there's a variety of things all packed into one thing.

So, yeah. I'm not sure I can answer that. I think it would depend on the specific experience. Chancey, do you have any thoughts on what might be helpful.

Chancey Fleet: I think that I might the closest analog we have experienced in real life is using a visual interpreter app like an AIRA, Be My Eyes. In a non virtual space at a conference or a concert. And as in Second Life, probably, sometimes it's easy to build rapport and do live interactive description and it's clear what needs to be described and sometimes not. I would imagine that in Second Life or any other 3D environment that's interactive, becoming concise would get more important as more things are happening faster.

James Herndon: Right.

Chancey Fleet: And a controlled vocabulary of asks and answers would start to be important. And it would be almost like working with an interpreter. It would have to be a live human connection I think at this point for it to fully succeed.

James Herndon: There would also probably need to be a way for somebody to request additional deeper information. Particularly around terms. You know, games often have their own world and you neat set of terms to the game that somebody playing it for the first time might need more information about. But somebody who plays it every day is not going to want to hear those descriptions over and over.

I imagine it would probably involve levels of description.

Thomas Logan: Great. We have a question from Charles Hall. Did any of the research describe sound effects like bubbles with the National Geographic Spark Experience or stairs creaking in the Anne Frank House.

James Herndon: Go ahead, Chancey.

Chancey Fleet: No, go ahead.

James Herndon: I was going say, in the National Geographic one, I mentioned the existence of bubbles going up and spreading. But I don't think I kept my focus on yeah. Just what seemed to be essential in the visuals onscreen. And the audio of the bubbles in that part did come through.

So, I'm not sure if I'm answering your question exactly. But if something if a sound coming through an earphones was occurring onscreen, I definitely announced it. And then but depending on the situation, there were scenes when I chose to announce the sound even though you couldn't see it. As in the Van Gogh one where I say, "A door creaks open behind you."

But there was others, the shark one, you get slammed into from behind what the shark bites you from behind. I didn't want to say there was a shark bite you from behind. That felt like too much liberty with the task that was describing what's going on on the screen. On the screen, it's you getting knocked over and something large and gray is over few you.

I chose to say something large and gray is over you. Circumstantially, I did it somewhat differently depending on the scene. Chancey, did you want to add anything?

Chancey Fleet: No, I think we're good on that question.

Thomas Logan: Yeah. And we've got a lot of questions, guys. We'll keep going through here. So, what about using 3D audio for the audio description?

So, instead of saying something is on your right, you hear the description for it on your right?

Chancey Fleet: That's a that's a good idea. Yeah. And I've heard it before. We should have mentioned it today.

So, that is feedback that we received that we could cut down on unnecessary words by just spatializing the description itself. So, that's an interesting possibility.

James Herndon: Yep. That's definitely an interesting possibility. I think doing this well is going to require the assistance and input of audio engineers and people who work with sound professionally to create vivid audio experiences. And as I mentioned earlier, I think there's going to be significant R&D into ambisonic video and make sounds feel like they're coming from any possible direction.

And, you know, while that technology does exist, I think the R&D to bring it up to the level of these extremely sophisticated visual interfaces, I don't think they compare. So, I think there needs to be a lot more done in that space. I would reiterate my plug for immersive accessibility. Some of the stuff that they have done involves setting beacons in a 360 degree video which you upload into their app in the cloud.

And then you can then set certain beacons in your video which when selected, they will pause the video. And people have different thoughts about standard versus extended description. Like I mentioned. But that's something being worked on.

But I agree, absolutely with that point. There needs to be 3D mixing I think in order for this to truly approach something close to equitable.

Thomas Logan: All right. Has there been discussions from drawing on experiences of gamers who are used to navigating around auditory worlds either in mainstream video games or complex video games?

Chancey Fleet: Yes. We did talk about that with our participants. Thank you for pointing that out. I was just thinking to myself when we were talking about designing soundscapes.

There's a lot that we can learn from sound designers in games in general. And there's also a lot that we can learn from blind gamers who have been experiencing virtual worlds non visually anyway.

James Herndon: I agree with that. Most AAA games are mixed for stereo to 3D sound now. Maybe not necessarily 3D. But mixed to where everything pans depending on the direction you're facing in the game.

That's become almost commonplace in new replaces. So, that is definitely an area where we could learn a lot.

Chancey Fleet: If I can just throw in a random thing. When we are done with our Zoom, if everyone goes to your search engine of choice and look up "Virtual haircut." The virtual haircut, in my opinion, is one of the most concise demonstrations between really stereo audio and a really 3D immersive experience. You're going to get exactly what it sounds like.

You're going to feel like you're sitting in a barber chair. We have a lot it learn from immersive sound design.

Thomas Logan: Another question from Peter. For the Anne Frank video, how varied was the time each participant spent in the house?

James Herndon: That's a good question. I we tried to keep it to maybe around 30 minutes or so. There's a there's another option in the app called "Story mode" that is a fixed path that takes you through room by room by room that reads everything. And that is about 25 minutes.

So, during these sessions, I allotted 30 minutes for someone to have in the house. And some people moved through it rather quickly. And some people lingered a little longer and wanted to hear the selections available or interactive items for everything in the room. So, in that respect, it wasn't unlike, you know, any group of visitors to a museum.

Some people wanted to buzz through quicker and kind of get a quick sense. And some people were much more interested in lingering. So, that's how we did it.

Thomas Logan: Okay. We've got two questions from Joel Snyder. First question: Some research in Europe and in my experience suggests that immediacy is the result of the use of vivid language and the quality of the voicing of the AD. I.E., not sounding read, but more like a friend chatting with you.

Was this considered?

James Herndon: It was considered. It was considered. That's a great point. I probably audio describers are true artists in and of themselves.

And I am a researcher and an accessibility specialist. But I am not a pro audio describer. I do the writing, but I do not normally do the performing. That's a great point.

I think and maybe Chancey could speak to whether or not a significant difference would be if she can envision a significant difference if someone was not reading from a script off a page but were essentially performing the description very well.

Chancey Fleet: I'd I don't think I have a fully set opinion on this. I think that interactive description, by its nature, is better when there's rapport. And it's not performed or read. It actually is conversational.

So, if we're talking about interactive live description, which is beyond the bounds of what we did in this initial research, I think that establishing that real rapport is important. And, yes, I think that when description is scripted in advance, a performed affect that's a little bit more immediate is different than if it seems like it came from a newscaster.

Thomas Logan: Yeah, I would comment when I went to your event in New York, meeting a voice actor in a breakout at least for me it was definitely interesting to hear more, you know more style in the delivery than what I had been used to. And definitely what we do in our work. I'm gonna add the second question from Joel, which I would be curious to know maybe some more feedback on this. Said folks are working on those include AED with video games, recorded and responding to movements made by a video controller.

Do you think there might be implications for video games?

James Herndon: I think so. That's a good point. I was recently reading about the AI integration that is planned with PlayStation 5 and how they want the system to be able to answer questions or jump in as sort of a helper in different games. Where if somebody has questions, the game can respond or answer to them more dynamically.

But also offering a lot more in terms of assistive technology. Whether it's audio descriptions within the game, which is another huge one, versus something like, you know, something closer to well done captioning. I know I'm seeing more video games that have audio descriptions in their settings menus or in the title screens. I haven't encountered as many that have them in the games.

Although I'm sure they are in development or out already. But I think that's absolutely an area where we could this could apply. Chancey, what do you think?

Chancey Fleet: Yeah. I think so.

Thomas Logan: All right. We've got a question from James Taylor. Have you considered experimenting with open CV to analyze images and an algorithm that then automates the audio description? Automation question.

Chancey Fleet: Not at all. And please send me some reading material about that.

James Herndon: Yeah. That's interesting.

Thomas Logan: Great.

Chancey Fleet: I think we're not quite there yet with computer vision. I think that if we think in the medium term, you know, 3 to 5 years, that gets more possible and more realistic every day. I don't think we're there in the next two years. But it's a fascinating field.

Thomas Logan: And then I see we had a question sorry, skipped it what are the barriers pre COVID 19, is it cost or awareness?

Chancey Fleet: If I can speak to that in a second, we're not even there as a blind community about still graphics. We pretty much understand that we have a right to alt text and textual accessibility on the web. But we don't all even push that hard for equal access to diagrams and video. I think many of us mostly are living in sort of a text only world.

And that's starting to change with the concerted effort that we've started to make towards getting video providers for entertainment at least to describe their content. We're not quite there yet when it comes to employment and educational settings. There's so much undescribed video content out there. So, I think part of it is that we, as a community, have not dreamed to demand it yet.

And often reflexively, when we hear about VR, we know it's not going it be accessible. We don't try to visit, we don't push back, and we don't demand. And I think that we have to confront the reality that we're doing that and that we have to invite ourselves into those spaces and we have to request and then perhaps even demand the access that we require. Because, you know, historically, media creators don't offer access if they're not compelled to do so or convinced to do so.

Thomas Logan: Great. Well, I think this is a great way to end the presentation today on a on a note of us, you know, all collectively advocating, you know, for more work and more of the future here. I did want to mention, there was several nice resources linked to into the chat today. I will annotate, or we will annotate the information that we share out after this call to also include some hyperlinks to some of the great resources that you all have mentioned in the chat.

If anyone else wants to include something in the chat, please do and I will use that to help annotate sort of interesting areas to continue further exploring this topic. I want to sincerely thank Chancey Fleet and James Herndon today for their awesome work. As I said, you know, we're really happy to finally get to share this. This was planned for CSUN 2020 this year.

And now that we have this information out, we hope, and we thank you for your participation today and we want to keep working on this and moving it forward. And we appreciate any feedback you share with us. So, thank you very much for attending today and be in touch.

James Herndon: Thanks, everyone.

Q&A

Because feedback has been so essential to this project, we were very interested to hear what sort of questions the webinar attendees might have for us during the Q&A. They did not disappoint: We fielded questions for over 20 minutes!

One attendee asked us if developers have considered giving users a virtual white cane, or guidance from O&M instructors. Chancey said that if we — Blind people — are going to do any virtual navigation in a virtual space, then we’ll need haptic feedback from a controller that could simulate a cane. And so, it would have to be fairly light, precise, and deliver a range of different convincing haptic sensations. She mentioned that some work has been done on this, and told people to check out Canetroller, a proof-of-concept that’s “not totally there yet.” In terms of guidance from O&M instructors, Chancey said that she would prefer to experience a virtual environment without one, but if the circumstances involved skill development then she might be open to it. 

Joel Snyder is the founder of ACB’s Audio Description Project, and a true pioneer in the audio description field. During the Q&A, he asked if we had considered the quality of the voicing for our AD. I replied that while I am an accessibility specialist, I am not a professional voice actor, and the AD might have felt more immediate and conversational to people if it had been performed by a skilled thespian. Chancey added that interactive description, by its nature, is better when there’s rapport. The describer doesn’t need to perform a conversational tone if what they’re doing is literally conversational.

Additional Resources

Attendees also shared some helpful resources in the chat:

When you think about audio descriptions for immersive media, what are your questions? We might have addressed them in the webinar, but maybe not. Check out the webinar for yourself, and send us your thoughts.