Accessible Virtual Reality: Advice from a Deaf Person on How to Make It Better

Typically, I never shy away from new software and technology. However, the thought of diving into virtual reality (VR) feels daunting. Still, that doesn’t damper my excitement in unboxing the Oculus Quest 2 VR headset and controllers.

I haven’t been this excited about a box unwrapping in a long time.

[Angelic music] ♪ Ahhh … ♪

Of course, I couldn’t try it right away. The first step is to charge it! But I can get going by downloading the Oculus app for my iPhone. That’s how you manage the device as well as browse apps to install. Unfortunately, you can’t remove apps through your smartphone. You must do it through the goggles, which is tedious.

Here’s the list of the first apps I install:

• AltSpace VR
• VRChat
• Netflix
• YouTube VR
• Anne Frank House

Equal Entry Founder Thomas Logan and I schedule a VR walkthrough session. With goggles in hand, we enter our meeting on Zoom. Before putting them on, Thomas gives me instructions on what to do. We pick an app and meet there. He shows me around the virtual world and how to move. But I struggle with transporting up to the second floor of a room. Talk about feeling like a toddler just learning how to walk.

My bionic ear detects he’s talking to me, so I pull up my goggles to see him on Zoom. It was hard to communicate with him in VR. Since I can’t listen without lipreading, I constantly pulled down the goggles to read his lips on video. I’d like for the VR platforms to take a page out of video meeting platforms like Zoom, Google Meet, and Microsoft Teams and add automatic captions.

On top of all this, a lot of times when I lifted my goggles, I’d accidentally hit different buttons including the one that turns off the screen. These buttons need to be in better places so you’re not constantly and accidentally pressing them while slipping the goggles off and on.

He doesn’t have a way to send me text messages. His Bluetooth keyboard doesn’t work in VRChat or Oculus Messenger. Using the VR on-screen keyboard is slow. What’s interesting is that Oculus has support for Bluetooth keyboards in their settings.

Oculus has Voice Dictation, but it currently only works in the U.S. and Thomas is in Japan. After an hour, of trying to communicate and navigate, I don’t achieve much. The most exciting accomplishment of our session is my finally transporting to the second floor.

Later, we found a better way to communicate through Google Chat. This still requires pulling up the goggles, but this time I’d listen for the ding and know there’s a message waiting. He didn’t have to repeat himself as the full text is on the screen.

Vertigo: No, Not the Hitchcock Movie

Before describing my first forays into VR, let me give you a little background. I’ve had vertigo my whole life. When I experience it or sense a dizzy spell coming on, it makes me feel like the room is moving even though I’m perfectly still. Other times, I feel lightheaded and tired, have bad eyestrain, or all of the above. I never knew it was likely tied to my deafness until having the surgery for the cochlear implant.

That’s because a few days after the surgery, it felt like a ton of bricks hit me. The surgery caused the worst vertigo of my life and lasted weeks. I couldn’t do anything but lie very still. I couldn’t read. I couldn’t watch TV. I could barely sit up to drink.

It’s important for people not to walk and talk on video calls because it’s dizzying, especially for lipreaders. Anyway, after the cochlear implant vertigo attack, I received a referral to a vestibular specialist. It was here I finally learned that my lifelong problem with vertigo is due to being born deaf.

This aroused my curiosity. So, I looked up statistics about vestibular disorders and deafness. The Association Between Hearing Loss and Vestibular Disorders study has found 70 percent of deaf and hard of hearing children with sensorineural hearing loss have a vestibular disorder.

The specialist gave me exercises. While I’m good about doing my physical therapy exercises, I failed to do my vestibular exercises because they wore me out and left me feeling drained.

I read somewhere that it’s possible to get the same kind of benefit with VR goggles. I couldn’t recall where I read this, but I found “Effectiveness of conventional versus virtual reality-based balance exercises in vestibular rehabilitation for unilateral peripheral vestibular loss: results of a randomized controlled trial.”

The authors concluded exactly what I was thinking. They wrote virtual reality balanced exercises weren’t superior to conventional balance exercises, but VR may provide a more enjoyable experience.

Exploring the VR world is more fun than vestibular exercises that have me staring at a wall. Thankfully, the Oculus app store lists the Comfort Rating (Trigger warning: The page has repeated animation. If you scroll down upon landing, it’ll move off-screen) for the apps.

An app’s Comfort Rating will have one of these: comfortable, moderate, intense, and unrated. When I look for apps and games, I search for those with “Comfortable.” Unfortunately, there are no search filters. All I can do is search for a name or keyword and then look at the app’s description.

Steam is another store that has VR apps compatible with the Quest 2. And guess what? They have filters. You can filter for apps that have captions! Unfortunately, they do not have Comfort Ratings. The industry needs a neutral third party that doesn’t have a financial stake in virtual reality to create the comfort rating system.

Meta needs to add filters, especially to help people find apps by Comfort Rating as well as captions. A search for “captions” or “subtitles” won’t pull up games that indicate this in their description. Besides, some games have on-screen and audio instructions. These aren’t called captions or subtitles. Something needs to be done to let people know the on-screen text is available for audio.

At the moment, I’m more likely to buy apps from Steam because I can easily find the ones that are captioned. When I find a captioned app that interests me, I’ll study the description to see if the app will be comfortable or too intense.

Vestibular disorders affect more than those who are deaf or hard of hearing. The Vestibular Disorders Association says more than 35 percent of US adults aged 40 and older experience vestibular dysfunction at some point in their lives. It’s always a better user experience to let viewers have control over any motion.

Hence, it’s important for the VR industry to think about vestibular challenges when creating products. Any steps to minimize motion can make a big difference. Granted VR is filled with motion, but it is possible to minimize it to create a more comfortable experience.

Getting Started with Virtual Reality

The first thing I need to learn is how to adjust the strap on the goggles. My bionic ear fell out every time I put on the goggles. After a little practice, I find a method that works without knocking it off most of the time.

This experience shows the need for headset makers to do user experience testing with people who wear hearing aids, cochlear implants, and other devices on the face. (They have an attachment for those who wear glasses.) Maybe they can find a way to design them to be more effortless to put on for people with such devices and contraptions around their heads.

I start with the Anne Frank House app as it’s the first captioned app I’ve found. Due to vertigo, it takes me two separate sessions to get through the tour. The app’s captions don’t help my vestibular issues because the captions scrolled. And they were out of sync at times. Trigger warning: The video looks like a shaky camera even though I held my head still. It wasn’t shaky like this inside.

It’d be a better experience without scrolling. Pop-in captions work better than scrolling captions because they let the viewers read at their own pace. Plus, it has less motion. Anything to minimize motion in VR helps. Despite feeling tired after touring the house, I was blown away by the experience. It evokes all kinds of emotions.

Time to try something else. I find First Steps for Quest 2. I think the Getting Started Guide needs to recommend apps like this as a first step into VR. It helps you learn how to use the controllers and move around. The app supports Standing and Roomscale player modes.

Here are the player modes:

• Sitting: Play while sitting and have three feet of movement around you from where you sit.
• Standing: Play while standing and have three feet of movement around you from where you stand.
• Roomscale: Requires a minimum play space of 6.5 feet by 6.5 feet or larger.

I had a blast dancing with the robot. Don’t ask me why, but I’ve always been partial to robots since first meeting R2D2 and C3PO of “Star Wars” and V.I.N.CENT on “The Black Hole.” This app gives you a great taste of what’s possible in VR.

Searching for Captioned VR

In hopes of finding captioned apps, I searched online only to find others asking the same question. No one had answers. This was long before I discovered Steam had a filter for captioned apps.

I knew that I could find captioned videos on YouTube VR, so I opened that. YouTube captions were all over the place as they had any one of these problems:

• Very short lines of captions.
• More than two or three lines of captions.
• All upper case.
• No music notes to indicate song.
• Scrolling captions instead of pop-in.

Just because a video says it’s captioned doesn’t mean they’re quality captions. Yes, quality captions matter.

One thing I like about YouTube is that it has a filter for captioned videos. Enter a keyword or a title, select CC and it’ll only search those that have captions. Unless the creator applies a workaround, the [cc] only appears when the video uses closed captions. Thus, a video with open captions most likely won’t appear in the list.

Next, I loaded Netflix, which works like regular Netflix except it immerses you into a home theater. The captions work great there. But I want to do more than just watch videos in virtual reality.

I finally found a couple of games that have captions.

Exploring Captioned VR Apps

Vacation Simulator, Moss, and Cook-Out: A Sandwich Tale couldn’t be more different. Vacation Simulator and Moss have captions that must be turned on. Cook-Out technically does not have captions, but it provides on-screen instructions along with the audio. Many games have this feature, but you don’t know which ones until you try them out.

Vacationing with Vacation Simulator

In Vacation Simulator, you’ll pull a tray and turn the dial to the language you want. This took me a good five minutes to do.

You’ll meet different robots. To identify the speaker, the captions show the icon of the robot speaking. The captions in Vacation Simulator are transparent and move with your view. This is called fixed captions. If you turn your head right, the captions shift right. When captions don’t move with your head, it’s called headlocked. I prefer headlocked as fixed captions upset the vestibular system.

I advise not using transparent captions because you can see what’s behind them. Sometimes it’s hard to read the captions or there’s distracting movement behind them. You can always move the captions by moving your head to make them more readable.

The best gesture is waving! It’s so simple and gets the robot’s attention. It’d be great to have a menu of hand-tracking gestures and a way to take photos without depending on the controller. This game supports Roomscale and Standing player mode.

Enjoying Dizzy-Free VR with Moss

In Moss, the game started without taking me to settings. After flipping through a few pages of the book, I finally find the settings to turn on captions. It worked, but I had to start over. It’d be a better experience to provide the options as soon as you start the game.

I’d like to see a feature on Oculus that lets users customize captions preferences in the operating system. And then each application would apply it. When I set my preferences on Netflix, Hulu, and other services, the captions show up every time using those preferences.

The style of captions is traditional with a black background and white font. I prefer this style over Vacation Simulator’s caption style. In Moss, the captions are headlocked as they don’t move based on your head movements. And this works well because that’s where the action is happening. Too much movement with the captions can upset the vestibular system.

I played Moss for more than 30 minutes. That’s the longest I’ve ever played in VR without taking a break. There’s something about how Moss operates that minimizes vestibular system disturbances.

What is that something? My best guess is that it’s similar to PC games in which the character is doing the moving instead of you and your eyes. Whatever it is, it’s working because it’s the most comfortable VR app I’ve seen to date. Moss supports sitting and standing player modes.

Get Cookin’ with Text-based Instructions

Thomas asks me to install Cook-Out: A Sandwich Tale. The game’s description says nothing about captions or subtitles. The developer’s website also doesn’t show any videos or images of the instructions. Fortunately, the instructions appeared on-screen along with the audio.

I believe a lot of games take this approach in giving gameplay instructions. That’s how it was with many of the games I played as a teen. My kids have shown me games with on-screen text and captions. With audio playing a greater role in apps, you just don’t know what to expect especially in VR.

App developers need to indicate whether the app gives instructions using audio, on-screen text, or both. The advent of dictation and voice assistants tells me there will be apps driven by voice commands. This, too, needs an alternative way to give commands as not everyone is verbal or has an accent the apps understand. It turns out the Oculus has a Voice Command feature. More on that shortly.

Diving into Virtual Reality Networking Platforms

The first one up for social virtual reality platforms is AltspaceVR. Think of it as an immersive social network. Attending an event in AltspaceVR requires downloading the software. You can either go to the 2D world through a web browser or the 3D world through a VR headset. I took the 2D route as I haven’t quite mastered navigating AltspaceVR.

But I did learn the most important thing: right-click or tap the Esc key for menu options. That’s where you can change the settings and turn on the captions. Here’s a detailed article on what it’s like for a deaf person at a captioned VR presentation. The amazing thing about AltspaceVR is that it has captions in multiple languages. If someone speaks in Japanese, I will see the English translation in captions.

For me, watching captioned conversations in AltspaceVR is like watching a ping pong match. My head goes back and forth to read the captions. This is a sure-fire way to get vertigo. I tried moving back so my head doesn’t bounce back and forth as much, but the captions were hard to see.

Considering it’s automated captions and the feature is in beta, they’re far from perfect. With so many people talking in the VR world, the captions are impressive. The team is working to improve the experience.

Next is Facebook’s Workroom. The app wants to give teams a place to meet virtually and collaborate. This requires installing software on your computer and a headset. It was a nonstarter. When I enter the room and connect to my computer in the virtual world, the monitor is shaky and bad for vertigo.

For meeting and collaborating with coworkers, I’ll stick with Zoom meetings and my regular computer. I don’t need to view my computer through a headset.

One of the virtual reality social networks charged for captions on a per-user basis. Accessibility should not cost extra. Not even if the technology to make the product accessible is expensive. Accessibility makes you a profit. I’ve chosen companies over competitors because of their accessibility.

What Do Virtual Reality Platforms Need to Do?

Oculus and other VR platforms must add a filter like Steam does to help users find apps with captions, subtitles, language options, and on-screen text instructions. If Steam can add a language and subtitles field in their app descriptions, Meta and others can do it too. Don’t just filter for language and captions. It needs search filters for Comfort Ratings, language, player mode, ESRB game rating, and accessibility options.

Since there’s no option to search for accessible apps in the Oculus store, companies developing accessible apps need to promote their accessibility options. A lot of people are searching for them. Spell out what those accessibility features include. An accessible games database is coming, but it does not include VR games.

Oculus has published Designing Accessible VR guidelines that include captions and subtitles. The guidelines are a good start. There are no captioning standards for virtual reality. It’s a different ballgame than captioning videos because of the 360-degree factor.

What’s missing is a guide for hand tracking and controller movements. Or at least, make them easy to find. I could not remember the shortcut for Voice Command. It took a bit of work to find this information. Speaking of Voice Command, I found this by accident. It was in “Oculus Essentials” of the main menu in the Oculus browser. “Try Voice Command” is one of the steps in Oculus Essentials.

The first time you run Voice Command, the app lets you know the shortcut. But that’s the only time. I couldn’t remember how to use it when I wanted to try it again. I opened the Voice Command app and no instructions for the shortcut. It lists voice commands, but not the shortcut you need before issuing voice commands.

Voice Command did an amazing job with my funky accent that hails from nowhere in the world. So far, I’ve used it to take photos, record videos, and stop recording. And it has worked every time. I need to expand my command vocabulary and see how that goes.

VR Accessibility Recommendations

Since this article contains a lot of recommendations for VR companies and app developers, here’s a summary.

• Add automatic captions to VR operating system.
• Let users customize caption preferences in VR settings to apply to all apps.
• Support Bluetooth keyboards and improve performance.
• Add filter to find apps by Comfort Rating, languages, player mode, and ESRB rating.
• Have a neutral third party create definitions and standards for comfort ratings.
• Add setting to provide instructions by audio, onscreen text, or both.
• Review headset design to work better with hearing devices and other body hardware.
• Avoid using moving or scrolling captions.
• Use headlocked captions or provide choices.
• Show settings before starting with the app. (I had to start over with a game after turning on the captions. It’d be better to be taken to the settings first.)
• Avoid making verbal commands the only option for input.
• Provide easy-to-find instructions such as how to use Voice Command, hand tracking, etc.

Know of VR apps with captions, subtitles, or on-screen text for audio? Please let me know. The hunt for captioned and on-screen text VR apps continues!

Have an Oculus account? We invite you to become friends with Thomas Logan at tlog333 and me at TexasMeryl.

Virtual Reality (VR) Accessibility Consulting Services

Our years of experience working with virtual reality and being speakers on the topic have given us a unique perspective when it comes to consulting on VR projects. If you’d like to innovate in the accessibility of VR please, please contact us to discuss how we can help you.

Meryl Evans

Marketing Director and Accessibility Consultant | Plano, TX

Visit Profile

Meryl is an author, speaker, trainer, marketing director and accessibility consultant, is the author of Brilliant Outlook Pocketbook and the co-author of Adapting to Web Standards: CSS and Ajax for Big Sites. The native Texan resides in Plano, Texas. She's a Certified Professional in Accessibility Core Competencies (CPACC).