
Virtual reality (VR) offers new and compelling ways for users to interact with digital content. These include immersive experiences that are beneficial in various domains, such as gaming, training simulations, education, communication, and design.
As VR technologies mature and commercial VR systems grow, an opportunity exists to understand how to incorporate accessibility as a fundamental component in the design of VR systems and applications.
At the Accessibility VR Meetup entitled “Accessible Virtual Reality for People with Limited Mobility,” presenter Martez Mott, a senior researcher in the Ability Group at Microsoft Research, shared ongoing research to understand and eliminate accessibility barriers that prevent people with limited mobility from engaging with VR.
Here are some of the highlights from his presentation:
- Seven VR Accessibility Barriers
- Challenges with putting on Head Mounted Display Headstrap
- Chairable Computing
- Defining Abstract Gestures
Transcript: Accessible Virtual Reality for People with Limited Mobility - Martez Mott
Martez Mott: So I wanted to kind of preface this and maybe apologize up front just by saying that my background is in academia so I'm typically, you know, I think of myself as an academic so I hope my presentation doesn't draw on too long but I want to be able to provide not only the research that I've been doing over the last year or so but give a little bit more insight into my research approach and how I think of research in this space. So another note before I kind of get started here.
I tend to like to have kind of more interactive presentations and events. So by all means. Whatever mechanism you think, feel free to stop me or interrupt me during the presentation if there's something that piques your interest or if I didn't explain something in enough detail or went past something so feel free to stop me because I love having these kinds of conversations. So just to get started here. I just want to provide a little background here.
I did my bachelors and masters in computer science from Bowling Green State State. a smaller university in the State of Ohio. and after Bowling Green, I started my PhD in the University of Washington and I graduated with my PhD there in 2018. This is a really great picture of the University of Washington campus. and currently I'm a senior researcher at the Ability Group at Microsoft Research. and the picture on the screen is Building Ninth nine in Redmond, Washington. and currently our group research various types of accessibility concerns and tries to come up with novel solutions to help us kind of overcome them and we do that through partnering with various people and really try to strive towards coming up with useful inventions that can improve a lot of the accessibility concerns that people have.
Really quick I just wanted to give an overview of what I'll be talking about today. So I give this presentation typically to kind of like more general types of departments or, like, at an university or something. So I know people here already have an interest in accessibility. So I think this first part around accessibility, I wanted to talk about in my research how I think about accessible computing and how does my conception of accessible computing then lead to my research in the accessible computing space. How it impacts my research approach. and then the final parts I'm going to talk about are going to be what work are we doing to move towards more accessible VR, particularly for people with limited mobility and then I'm going to end by talking about future
possibilities so things that we've been thinking about currently that we haven't maybe yet have the opportunity to explore further. So, yeah, first I wanted to start off by providing a definition. So this is kind of the way that I think about accessible computing in some of the work that I do. So I kind of think about it in two kinds of ways. First is devices and services to be able to used by all kinds of people. So I really thinking about accessible computing is performing these kinds of two different roles. and in doing so. We've come up with a lot of different design paradigms so help us accomplish that. So probably the most common, to most well known is Universal Design. So I have a definition that I want to say. It says Universal
Design is the design of products and environments NASCAR used and to the extent possible without adaptation. So Universal Design is really popular. It basically comes out of architecture and built environments with the idea being you want to create built environments that can be accessible to all people but it was kind of co-opted by the accessibility computing community as a way to also think about how can you conceptualize Universal Design for other different types of productions.
So this is really useful, right, because it can give you really great things like the curb cut example is kind of a canonical example of this perhaps people who might use a wheelchair but then you get kind of these second-order benefits for people that might be pushing a grocery cart or riding a bicycle or something like that. Same thing with closed captions. It could be great for providing audio. Providing textual descriptions of audio so I wanted to return back to this Universal Design definition really quick because I think there's something really interesting here. So just to recap. the definition says the design of products and environments that can be used and experienced by people of all ages and abilities without adaptation. So this notion of abilities here is interesting because it kind of begs the question, what is ability. What do you mean by
ability if you want to be able to accommodate the greatest extent possible of what these things are. So we can have a couple of different once again these shorthand definitions that can help us out with that. So one way to think about ability is. You can think about it as just being as just having the possession of means or skills to do something. So if that's kind of your notion of ability, I think that gives you a really broad understanding because I think we can all relate to this. I think we all possess the means and skills to do something. but we don't possess the means and skills to do other things. So it gives us the. So then it also asks the questions, what is disability? So if you go through the World Health Organization's updated definition of what is
disability, you know, we moved towards more the of social model of thinking about disability which is that disability isn't something that's inherent within a person. It's something that's contextual and it's something that's influenced by the environment. So it's not just what a person can do. It's what does the environment allow a person to do given the constraints, et cetera.
So this is all useful. Kind of design philosophy and it's something all day ability-based design.
but in this approach what we try to do is we try to think about the human abilities required to operate or interact with a system and we try to question them and we try to create systems that can adapt or can be adapted to alternative abilities and the useful framing here is that it provides us something with this positive affirmation of ability. So it says that all have abilities some people more than others. and this is try to for anything and we have to design for abilities of all types. So this is a really quick background of how I think about accessible computing and how these different types of different frameworks can be useful for orienting yourself when trying to accomplish a research project or something else in this space.
and so I started off with how I think about accessible computing. I wanted to talk about my specific research approach. So typically what I do in my research is I try to understand what I call ability assumptions so I'm going to make a statement here which is that all human-operated technologies have ability assumptions implicitly or explicitly. it could be a hammer if a person designs a hammer, and we do the same thing when we design computing systems. So what we try to do. or at least what I try to do is I try to interrogate those and say, what are we assuming? What are we taking granted? What are we trying to assume that people can do and how are we limiting ourselves in assuming what people can do and things like that blood pressure so just really quick. This isn't VR
related., for example, for my dissertation work, I kind of looked at the accessibility of touch screen devices with the idea of how can we make touchscreens more accessible for people who have limited mobility? So these might be people with limited dexterity in their hands or fingers. This might be people with poor coordination or limited movement or poor strength. All of these dieselings can impact how you impact with a touchscreen device, for example. and it's then we really need to reconsider how we're designing these types of systems. So my dissertation work was investigating these different types of questions and I'm trying to come up with solutions that would help that in some different ways. So in general my research approach kind of follows this three-step process of really trying to question these ability
sumps and in my work what I do is I create interaction techniques that take advantage of what people can do. So these interaction techniques dictate how people interact with their computing systems and if you have interaction methods or input methods that are inaccessible to people, you want to devise alternatives that are accessible to people. and, finally, we do an evaluation approach where we build these things and have target users. We actually do kind of a more integrated design process where we actually work with people throughout the desire designing and building multiple rounds of iteration, et cetera, but at the end, we also want to have a prototype that we can actually test and say, how well did we achieve our goals for these different types of scenarios.
So that's a little bit about my different research approaches and how I think about that. I just wanted to be able to get that out of the way because I think it might give useful framing for the research that I'm going to be talking about right now. So in this next section I'm going to be talking about the research that I've done since I joined Microsoft research and this has been ongoing about the last year in half. So me myself, I haven't been working in this area. Why I do my dissertation work. So it's been. for me to kind of get embedded in this community and try to understand the accessibility challenges that exist and to try to hopeful propose some useful solutions and ideas that can be useful for the community.
So that's kind of what I wanted to talk about today. So this is very preliminary work. It's very current stage work. the coronavirus pandemic has impacted our lives in very significant ways and it's very hard to do this type of work of actually being able to bring people into a lab. Have people try different VR experiences and things like that when you can't be near people. So some of these things I'm going to talk about today are going to be studies that are kind of in process or things that we have thought about broadly but haven't been fully been able to explore yet just because we haven't been able to work with participants as closely as we'd like because of the pandemic. So just keep that in mind, that's kind of the state we're at right now.
So this is an accessibility VR meetup. I don't have to explain to people here that virtual reality has an accessibility problem. So what I have here is a clip from the Scientific American that says VR's been touted as an empathy machine and lets users see what it's like to have a disability. but people with disabilities can't often use it. So in general it's been documented for a while now all of these different types of accessibility concerns that exist with virtual and augmented reality types of systems. When I joined Microsoft Research my colleagues and I we ourselves wrote a paper that we submitted to a conference. It was kind of like this short position paper titled accessible by design and opportunity for virtual reality.
and the idea here was we wanted to identify some areas that we thought the academic community may be missing out on in terms of thinking about how virtual reality should be designed because, you know, virtual reality research has been conducted for the past 40 years or whatever the case may be. So people have been working on these problems for a really long time. but accessibility, unfortunately, is something that often gets considered primarily when commercial or once systems kind of reached that commercial viability point. So we kind of missed out an opportunities in the '80s and the '90s to do groundbreaking work in accessibility and VR so we're kind of calves camping up and trying to do that now unfortunately. So we highlighted five key areas of focus. So these were things that we thought that the academy could focus on and
really make a big push. We tried to. So the first is content accessibility. Second is interaction accessibility. Device accessibility, inclusive representations, and application diversity.
So I'm not going to talk about all five of these key areas right now. I'm going to talk about a couple and then kind of hit the rest at the end kind of like as a closing kind of thought. So first I want to talk about device accessibility. So there's this thought, you know, that I mentioned earlier about these ability assumptions. So when commercial VR devices come out, they, too, have these embedded assumptions about how people can use them. So, for example, if you have a VR system that kind of comes with the standard HMD and then comes with two motion controllers and things like that and if that's the primary way, way that you expect someone to interface with a technology what happens if someone can't use a emotion control technology.
So one of the ways that we think about this is this can facilitate the need for more customizable hardware. More customizable virtual reality, augmented reality, spatial computing reality hardware that's not just focused on 2D scenarios focused on desktop and mobile computing. and the other thing is how can we integrate accessible technology with mobile technology.
and just integrate those with VR. It shouldn't have to be an either/or. People should have the option to choose. So some of the prior research that's been done in Microsoft in this space.
Not just for people with limited mobility has been this emotion of the put that would give people the opportunity to use their kind of standard white cane practices in a VR environment and kind of give them haptic and tactile feedback about what's happening in the virtual world. So the idea would be that you encounter a trashcan in the virtual world and your canetroller stops and makes a sound as if it hit the cash trashcan in real life. So once again this is work that has been done before I started. So when I came in, I was really trying to think about how can we really think about device accessibility for people with limited mobility. So I really started with these questions with what are the accessibility barriers that people encounter now
using VR hardware and then for people who have used VR in the past what have their strategies been high pressure how did they overcome their accessibility barriers? What can we leverage from their knowledge and practices and expand that to make it more general for everyone else. So to do this we conducted a study. We recruited 16 participants with limited mobility. So people with various types of injuries or medical conditions. at this time, we were still able to do in-person studies. So this was kind of last summer, well, not last summer, but summer of 20193 so we did two groups, one in person and one remote and what we did is we have them try out different VR applications. Just play around with these things and kind of let us know what your experience
was like. Oculus first-contact application. and also had people exploring around on Google Earth and from our study what we find or what we were able to steal down from those interviews were seven accessibility barriers primarily related to the physical devices themselves. So this may not speak to any one particular type of interaction that one particular game or application might provide. It's really just thinking about okay. Yeah, so looking at the physical kind of accessibility devices and trying to and figure out kind of hey, at this level if we can't have the devices themselves be accessible, but then the applications they run on be accessible. So we have to kind of think about the device accessibility first. So couple of things. First adjusting the HMD headstrap. but they're slightly different
and I'll talk about that in a second. Cord management. So if you have any kind of tethered HMDs that could be a problem. So manipulating kind of dual motion controllers. I'm not going to go through every single one of these but I did want to just kind of talk about a few of them. So the HMD head strap, or putting on the HMD in general was a big problem for people primarily for people who may have limited range of motion. So, for example, some people may not be able to lift their arms above their chest or above their head. So the idea that hey, I have to put this wearable device on and off myself and then there's also this problem that these are not built in a way
to facilitate someone helping you do this. So it's not really a two-person it wasn't really designed that this could be a two-person activity. and I think there's going to be a big barrier to entry to VR if we can't come up with more accessible ways for people to put on HMDs or other types of wearable devices. It interacts with lots of different things. So this one participant wrote.
One thing I noticed was the backstrapping is kind of thick. So the adjusting so they said that the backstrap thing is thick. So it does touch my headrest. So that's one thing that makes put more difficult to move my head. So what they're kind of speaking to is if you use a power wheelchair and if you have a headrest and you have this kind of clunky head set on as well and you have this big adjustment knob or other types of things that are located on your HMD it's interacting with other assistive technologies that you might be using, right?
So we have to think about how would these people use these devices in conjunction with other types of hardware they might use. So the other thing is manipulating dual motion controllers which is a common problem that people hear or may know about or think about. but I think quotes gives texture. So another participant, participant 1614 said, I use my right hand to control one of the sticks and I think having the one hand barrier is quite common for people with strokes. So if you have done a lot of these interactive VR experiences the idea can be, we really want you to move in a way that simulates how you would perform an action in the real world. So, for example, it might be something like we want you to climb a ladder on this VR game and put and in order to do
so, you have to climb using. but the idea would be. What do people do when they encounter this situation in the environment and perhaps it might be that they can use one controller. but how would this work if these VR experiences are expecting people to use two hands all the time, or maybe not all the time, but some of the time to perform certain actions or engage in certain experiences.
and then this probably is the most common. I think inaccessible buttons probably the most common from a more traditional kind of console and PC gaming so the idea being on these VR controllers you have a lot of these buttons and triggers and it could be really difficult to be able to manipulate these independently or simultaneously. So we had one participant, participant six wrote or said, yeah, I think walking was pushing it down. So this was an experience where they were trying to walk in.
So they said yeah, I have to use two hands. So I have to lead the other controller to do it. So as soon as I heard, you had to push it down, I was like, nope, that's not gonna happen.
So one of the other things that we found in terms of talking to people was what are some things that that could be potentially really useful to them. but maybe we don't have really great methods or design strategies to think about that. the first thing was. It would be great if I could use whatever input controls to be able to control whatever VR interface I have. I have a level of proficiency with the devices that I already use. Why can't I use these things in a Jr.
environment. and other people said, why can't I use Gaze or Voice. I think if we wanted to expand choice, I think you would also want to err on that side. but there are a lot of these tricky things, right? For example, some of them have diorectic speech, which makes it really difficult for speech recognition systems to recognize their Gaze input can also be really problematic depending on a lot of different external factors, just like changing the lighting that you're in can make Gaze input that which more accurate or inaccurate. There are a lot of challenges here trying to understand how these things fit together. but I think a useful thing is how can we leverage these things to make this more useful. So one thing we thought about after doing this is how can we leverage
what devices and things do people already use. So a colleague of mine, Dr. Patrick Carrington in Pittsburgh, Pennsylvania. He conceptualized this term, charitable computing.
So it takes advantage of the space. As well as worn technology. So what did was say, hey, if we had something where someone can leverage devices that they already use and have some familiarity with and place these things in a location that's more accessible to people. and requires less energy to reach, that could be a really powerful combination, a really useful way to think about accessibility in that space. So one of the things we were really thinking about was this kind of general question of how do we leverage more general 2D input devices.
if we could understand how joysticks and buttons and sliders and rotary dials these kinds of traditional 2D input devices, how can we leverage those to do more complex spatial tasks? That could be really good for this and one way one that we identified was virtual hand manipulation. So this might be a slide that you might actually want to zoom in if you're in the room to be able to see more clearly but essentially what I have here is a list of different types of hand manipulations that you can do currently kind of like with just like an Oculus touch controller setup, right? So you can do things like perform a thumbs up and you can also interact with objects.
So you can grasp things, and throw them. and release them. Rotate them in various fashion, right? So this is something that with current kind of VR technology if you're using Oculus or similar types of touch or motion controllers, you can perform these types of interactions but what if you were just using a joystick or a touchpad or if you were using some other combinations of devices but how would you perform these types of interactions given a more limited or restricted input device. So if we were to use these things, how would we use them? So we did something called a user elicitation study. So what you do in a user elicitation study is you provide something and they give you something back called symbols. So all these different hand manipulations would be referenced and what we would do is reference back and say,
how would you do this kind of roll-right rotation given some type of input device. So there was a total of 16 different hand manipulations and we used the touch bad pad as kind of our preferred input device for this investigation. There's a couple of reasons for this that. So one is that in some of Patrick's prior work is that you can use the armrest on a power wheelchair it could actually be a really good vehicle a lot a touch capacitate surface. So you can instrument that to create a 2D surface. and also people may mount touch pads on or near wheelchairs for slayers situations. So that we thought the touch pad could be a useful implement to understand.
Hey, if I gave you this device how would you perform these different types of interactions and for our analysis, I'm not going to stay on this long. for our analysis what we used was the taxonomy surface gesture meaning the way to analyze them and to get a better understanding of how people are performing different types of gestures and it gives you a way to be able to categorize and compare different strategies and things like that. So it's useful kind of taxonomy for performing these types of analyses. So some things I want to share, for example, is that some of our initial findings. So this is like I mentioned ongoing research. So when we noticed is, for example, that when we had people do what we called discrete hand gestures. So a discrete hand gesture would be something like performing a thumbs
up, or extending an index finger or extending a Clostridium difficile fist, things like that. People can difficulty trying to generate an action that they would perform, that would perform these actions. So what we would get as responses most of the time or what's called abstract gestures. So abstract gestures are things where the gesture that you suggested doesn't really have any relevance or maybe not relevance is the incorrect work.
Isn't really similar in kind to the general being performed. So, for example, if I asked someone to make a thumbs up. Make a thumbs up, I would tap the touch pad three times.
That's an abstract gesture. There isn't anything inherently related to touches a touch pad three times to making a thumbs up or extending a index finger or something like that. but if we compare that to more action-style gestures we had, we found 57% of those were metaphorical gestures. So a dichotomous reference. You're doing the same thing but not the same thing. You're doing kind of so what we noticed was that people use metaphors quite well to reference which gesture they would use to perform this gesture. So when we asked participant 16 how they would grasp this cube using a touch pad, he said I would squeeze my fingers to you're actually squeezing your fingers together. It's kind of mimicking what you would do if you were grasping an object.
and similarly to release an object. They said they would extend their fingers to release. So we've been finding that people have actually been able to come up with strategies to actually do some of these complex interactions with input devices that they weren't originally designed for. and what I hope with this research is we could distill these recommendations down and give designers a sense or at least understand how they might from the start support these different types of interactions considering that future interactions with VR may occur on different types of devices that aren't going to be these kind of motion track, touch devices that exist currently.
So really brief on the takeaways from this kind of one project or at least this device accessibility part that I talked about really quick here. So one of the things that we talk about in one of the research papers that I talked about is design for interdependence. So most people know independence. Being able to think about how people can be self-sufficient and do things on their own but the similar Cindy Bennett, introduced this notion or not introduced independence but she didn't come up with it but introduced it into this HCI context saying, you know, people with disabilities have these very complex relationships with family members, caregivers, friends in different ways and we can actually leverage that to make hardware more accessible.
So if you're thinking about how to make VR hardware more accessible, don't just consider an.
You might want to consider that people are working interdependently to be able to, you know, for example, put on a VR head set or be able to set up their VR environment. So if you use an oculus system and you want to use a guardian system, how can you use that system so that two people can use that in a more collaborative way that's more accessible. So designing for customization. We need to be able to provide more options. the more customizable our input controls, this notion of designing for accessibility is really the notion of really thinking about ability, right? So, you know, you want to be able to think about different abilities when designing these different types of systems and diversity what we really mean here is in our interviews people said they want to be able to interact with VR in different ways
and in some of the conversations that I have had with people that create VR systems they come off of it with an expectation of how people use it, right? and if you tell them that hey, I'm going to alter that experience, evening it's breaking trauma that original vision and what we have from our participants is I don't care that I'm not using or engaging in this experience the way that the developer intended or so I think those are kind of high-level takeaways from this device accessibility part that I mentioned. and so the last thing not the last thing the next thing that I wanted to talk about is this interaction accessibility. So I kind of mentioned the devices, that's kind of like the hardware level how do you interact with things
the interaction part is you have an interaction device or some type of device, how you interact with the system is this language that you have to construct, right. So you have to think about how can we extend the use or how we can think about why VR extends the use of these inputs in these different ways. and how can we overcome that through these different types of interactions that we can create. So one of the things that has been done previously once again looking at people looking at solutions for people who are blind or have low vision is this notion of seeing VR.
and I'm sure many people might be familiar with this because I think it's made a lot of rounds and but it's essentially a set of tools that people can add in and it essentially gives you post hoc accessibility improvements. So you can do things like adding a magnifying lens or edge enhancements in order to make a VR environment more accessible. So when I talk about, like, interaction accessibility this is the kinds of things on him. Kind of more at the software level rather than what are we doing to make the physical devices themselves more accessible.
So I wanted to go through another quote that came from this interview study that I mentioned and this really kind of made us think about accessible bimanual input. Tim wrote, I'm not sure that it would be possible for me to do them simultaneously. Using two motion controllers and doing something simultaneously. They said if I had to do it, I could do it one handed in if I had to do it both hands at the same time, that's not gonna work. So that got us really thinking about, hey, there's going to be a big population of people in which two handed or this bimanual interaction is just not going to be a great solution so we need to think about what else can work there.
So we kind of Stewart off with this question of well, how do we allow people to control kind of bimanual interfaces with unimanual input. So I want to give big credit to Momona Yamagami, my intern last year, they say a PhD candidate at the University of Washington, and she joined us this summer to help us think about this question. So there's actually been a lot of prior work looking although people with asymmetry bimanual movements. So we were actually able to say okay, if we want to understand bimanual actions in VR, what in general are the types of bimanual interactions that exist in everyday life. So we can actually use this framework in VR here to actually give us a way to think about that. So I'm going to go through this really
quickly but I think it's a really powerful way to think about this. So there's really four different categories of bimanual interaction. So bimanual meaning interaction with two hands. So there's symmetric in phase interactions. So those are interactions in which both hands are performing the same interaction at the same time. So if you think about if you're jumping rope, both hands are using in a circular emotion in tan let me with each other. So the next motion is symmetric autophase. Performing the same action but just at different times. So, for example, if you're climbing a ladder, you might move one hand up, move hand up another rung and then your next hand would move up the next rung. So your both hands. There's asymmetric coordinated. So this is when both hands are working together on a single kind of object. You can think about it. So,
for example, if you're cutting an apnea Apple, one hand holds the Apple while the other hand does the cutting and you can think about that if you're riding, one hand holds the paper steady while the other hand starts the writing. and the next one is asymmetric uncoordinated, which is the hardest one to design for but is probably the most interesting. Because in asymmetric uncoordinated, both hands perform two independent interactions. So in a video game context, you might think of this as with one hand you might be able to thrust with your sword while in the other hand you can bash with your shield. So both hands are doing two different kinds of actions independent of one another. So we have this framework. So we can think about these different types of interactions.
So next part is how do we want to think about designing the interaction techniques. So for us, we really thought about it as a spectrum. and we thought about it as more or less autonomy to users depending, again, on what's their preference. So, for example, you can imagine a situation in which users have little autonomy. So that could be a person encounters an action or task that requires bimanual input, you can imagine with no autonomy, you can have just like the system automatically perform the task for them. So, for example, they approach the ladder instead of requiring the person to climb the ladder you just say, they approached the ladder.
We'll just send in ladder for you kind of deal. So there's kind of something that's more between and that's something that you can think of more as what if we can infer the output of one hand.
Can we just say, I'm just going to put this secondhand on autopilot and it's just going to go off what our primary hand is doing. and then the last one is we can give people a lot more autonomy, right? We can just say give people an alternative input method in general. Instead of saying do something automatically or doing something where something that's being inferred from a little bit of input. We can say, hey, in order to control this secondhand or second control whatever the case may be. What if we can use HMD position or eye gaze or voice or something that's kind of a proxy for what that secondhand may be doing.
So what you're able to do then is you can graph these things. You can kind of put these things on a spectrum or you can put these things in a constructed design space and say, okay for each of my different types of bimanual interactions, how would I design an interaction technique that kind of fits these different types of that fits on the spectrum on here about autonomy. So, for example, let's think about spectrum out of phase and I think it's a really useful example. So if we think about spectrum out of phase.
Once again automatic could be something like I mentioned before where you approach a ladder and you automatically climb the ladder, right? but you can also think about something in terms of, like, how do I want to infer the virtual hand. So there's a couple of different approaches on this. and what we have is a simple na ve approach. but we can actually think about this more complex.
You can have a simple example where you have the off hand mimic the speed and distance of whatever the real hand is doing. and once again, this is for these symmetric out of phase interactions.
Climbing a ladder or rope or whatever type of activity this might require. So you can imagine something like I'll have a script with my virtual hand so it's going to mirror or mimic what my real hand is doing and that could be really useful for if the person is doing what's symmetric or out of phase. You don't have to do double the work and you can perform these movements simultaneously but once again the drawback is you have less control so if you didn't want that virtual hand to do something but once again it's being inferred on what it's trying to do. You can think about this as kind of like a data training example. So if you have data about how people perform actions with one hand, can you train a model how the secondhand would react in a given type of situation.
and then the other one was like I mentioned was this alternative input approach. So this could be something, for example, in which where your real hand or whatever hand the person is manipulating that's going fine but whatever the off hand is doing, you just map that to some other type of input method so if you want to say, what I'm going to do is actually map, you know, a rate I'm going to shoot a ray from my HMD position and use that as a pokes to determine where to place my next hand next and we can do that in order to understand what might be going on, or might be a useful way to something I have people more power over what's going on. So they can actually have a a little bit more control. So like I said the benefits here are more autonomy
and control. Once again people can perform these movements simultaneously. the problems come up depending on what you need are on this off hand. So if you want an example where you want two hands holding two objects and the off hand is performing some rotation. or rotating a key to try to get it to fit you probably need something that has similar degrees of freedom as the hand that you want to manipulate, right? and the idea being if you map too many inputs given this one input device you may have difficulty understanding what exactly you're doing when you need to perform some action. Once again that's why we want to be able to provide choice in this way. So this is kind of the this is kind of what I wanted to hit on in terms of some of the work that we're doing
currently around interaction accessibility. So I wanted to provide that as one example as what came out of our input study. Once again, we hope to do we're doing more work with this study, with this project. but once again it's really hard to be able to actually validate and designs when you can actually have people into the office and actually try these things and give feedback in that way. So my last thing here is I wanted to talk about these future opportunities and mostly future opportunities from the document that I shared, the accessibility by design document and what we can do next in that space. So I wanted to just kind of hit on these different things. So content accessibility is another big thing here. So in general here, I think
this is something that's taught in this community a lot is that we need more agreed upon methods to understand what kind of metadata is necessary for different types of content, what standards need to be applied. When might you want to provide alternative representations of something and how can we allow people to with the idea being people might be able to bring in accessibility options that work particularly well for them and maybe some designer didn't think about that a priori. So how can they bring in those designs in different types of spaces. and another project here that's ongoing is this notion of inclusive representation. So the idea being that if if your application allow people to embody these different types of avatars.
There's going to be all these different diverse applications for education, for work, for social, for different types of events, job search, whatever the case may be. and it's unclear at least from the research point of view right now how people with limited mobility would like to be represented in different types of VR environments put so the idea here being that you want your VR applications if it has kind of default avatars or an avatar creation system that there's inclusive representation so if a person chooses to, they can craft a avatar that's representative of them in whatever context it may be.
and then the kind of last thing here is this notion of application diversity which is just to say, and this is not new to anybody in this room probably which is that VR allows us to do dieselings that traditional 2D interfaces don't allow us to. So the opportunity here is how can we leverage. So things, for example, as virtual orientation and mobility training. There's already been a lot of work on thinking about rehabilitation and physical therapy which is not a space I'm in necessary but it's an interesting space just more broadly and there's kind of this way to virtualize existing tasks. Ideas that have been floated around for a while.
Can we just replace physical monitors with virtual once and what are going to be the accessibility ramifications whenever we get to that point. So just in kind of in conclusion. I wanted to wrap up by saying, we've identified and like I mentioned before people in this community are probably familiar with these things already but people encounter a lot of these accessibility barriers when interacting with VR systems and I think it's important to really think about ability when designing solutions to these problems. So really putting the focus on what people can do.
Really focusing on how we can leverage existing systems, existing systems, existing platforms to make things more accessible and then also kind of this notion of ensuring that VR systems are designed to promote equity for people with disabilities. So we don't want it to actually be a way in which we create VR systems and all we're doing is importing the inaccessibility of the physical world into the virtual world because that's not what we want, right? Yeah, so that's it for me. I'm happy to answer any questions or, you know, provide a little more context or information around any of the projects that I mentioned. So thank you all for your time. I really appreciate it!
Thomas Logan: Thank you so much, Martez. and we will now open it up to questions for those of you on the YouTube screen. Also please type your questions and we'll voice them on your behalf. but for anyone in the room you can go off of mute and ask your question tammars directly or if you would like to type your question into the chat. and I see the emojis for the clapping.
Thank you, Martez. Excellent presentation. So anyone have a question you'd like to ask?
I'll start off. but I was reacting to your comment about when you did the user study and your participants who were giving feedback. I don't care if it wasn't designed this way. It doesn't have to be the way that it was designed for it to work for me. I think you were saying that people don't consider that. but have you heard how people have responded to the users' feedback?
Martez Mott: Okay. We want to have this experience. and the experience feel like this. That you're in the jungle and you're doing something. and they create the experiences and the interactions on top of of that that facilitate the feeling that they want to be able to invoke.
and the idea that we have heard from our participants is I don't really care that this participant thought that their primary or idea of the way people interact with this was to use controllers in this kind of way and have the timing be this. They're like, hey, I want to be able to enjoy the experience or experience the experience, right? I don't necessarily care that it doesn't meet this kind of content creator's vision. So I think this has been a useful way of thinking about that and like I said being in the community for a short period of time I've been able to talk to people who have not had interactions with people with disabilities who were saying, people just want to be able to enjoy it.
I understand that you want to come up with something that's useful and beneficial and mind-blowing but in a way we want to come up with things that people enjoy. I think that's awesome and I think a lot of times I think people need to be able to see or experience someone that does need to interact differently and hopefully that does change this creator's mind to think about it differently.
Thomas Logan: Okay. We have a question here from Mike: Are you aware of any standards of progress with VR similar to WCAG, web content accessibility guidelines. Does Microsoft have any guidelines for infrastructure in place for making VR around this?
Martez Mott: Yeah, so there are some emerging standards. So there has been a lot of these different types of, I don't know exactly what to call them. Associations and working groups. You know, there's like the XR kind of association. the XR access organization and different places. So there's been a lot of work that's been done developing these standards and I know there's been one in particular that I'm blanking on that I can't come up with the name for that is like the web more like a web type of like XR standard. and I think so to your question internally at Microsoft standards compliance. So in general there's kind of like compliance in place, you know, for, I think, Microsoft products more broadly just as common business practices kind of compliance type of things. and I think,
you know, I'm glad this question was asked, right? Because I think, you know, it gets to this point and I tried to talk about this or I try to evoke this in my research sometimes which is I think that a lot of times when people think about accessibility they compound it or they conflate with with compliance. I think when people think about those things in certain ways, it's like, hey, when is is something accessible? When it meets some type of compliance or when it meets some type of legal standard or something like that. and I think one of the things that I've found really fascinating in my research is really thinking about, hey, shouldn't accessibility in this context be people yes, of course, you want to be compliant but I think that's the floor. I think a lot of people
think of compliance as the ceiling. We've kind of covered our bases. but I really think that, yes, compliance should be kind of like the floor and we should really be striving to create kind of like the most amazing experiences in the best way possible for people with a range of different abilities, right? and I think that some of the work that we've been trying to find or at least in the area that I've been identifying is that it's really hard right now to understand how we would create accessible experiences for the range of different types of abilities that people have on the input side that I think makes it really tricky. So I think, for example, on the output side. So I think if you're thinking about well, how can you make
3D videos more accessible. So you have this content consumption problem. How can you make it more accessible to me. Whereas I primarily look at on the input side. How do I interact? Can you make these interactions more accessible to me. and I think we're, unfortunately, really far on the input side of how we can come up to these notions of standards and compliance to make it so that people with a range of different abilities can use these systems.
Like so I think a simple example for that would be, right, if you imagine someone who might have, who who has like a neurodegenerative disorder like ALS or something. and uses eye gaze for input.
There's ways that you can create accessible interactions with just eye gaze. There are things that you can design bespoke designs with eye gaze. I'm not aware of any eye-gaze solution for VR systems. I'm not aware of anything that would necessarily be, hey, if you have an eye tracker, it doesn't matter what you load up, you would be able to use it with just eye gaze. How can we make it so that VR whatever the case may be is wholly accessible to somebody who has purely a gaze-based interaction method.
Thomas Logan: Cool. All right. We're going to take a couple more questions and Mike just said thanks to you for that. Anyone have any other questions or wanna go off mute and ask a question?
Okay. Well I'm good with questions. I'm going to ask one more. So you did a study with three different well, multiple studies but you had the slide where you had done a comparison or maybe an evaluation of three titles. Was there any VR title or sort of experience that stood out as a better implementation or that had more availability, you know, from the ones that got evaluated.
Martez Mott:Yeah, that's a great question.
I would say not really. I would say we try to we were trying to look so we chose those applications specifically because we were looking for kind of a broader range of different types of of interaction styles that might exist in those applications. So we had them with the Oculus First contact applications. So for that one, we would have them play around with different objects that we have in the lab. but it was really good because we would have people perform these different kinds of hand gestures. Like you would have to grab a cassette and place a cassette into something and it was great because we could evaluate how well. and then it was kind of like. Okay, when people did Google VR, or, yeah, like the Google VR or experience, it was like oh, yeah, hey, people have to kind of use
these two controllers in this way to do something. How would that work? All space VR is kind of VR application-y. How do people teleport when they were trying to navigate the space? How do people walk or interact with a different object? So I would say that particularly there wasn't anything that stood out outside of those kind of higher level things because we were really looking at for that project, really the hardware. We were trying to look at. Hey, at what part does the hardware provide people difficulties. and I think in some of the work that we're going to do hopefully once we're allowed to be around human beings again is looking at more specifically looking at these different interaction styles while after we can come up with some hardware I think would be really beneficial.