Leveling the Playing Field: Toward Equivalently Accessible Video Game Worlds

Good evening, everyone. My name is Vishnu Nair. I'm a PhD candidate in computer science at Columbia University. And much of my PhD has focused on video games. Video games are a pastime, and for many of us, an escape. Often allowing us to explore and experience complex, fantastical worlds on our own terms. Just to illustrate, here's a clip of a player in a video game running through a small town with little houses and other small buildings. Just here, they're constantly scanning the environment around them, making split second decisions, depending on what's around them. And fully exploring the world on their own terms. This clip here is from the Legend of Zelda: Tears of the Kingdom, which is an action-adventure game that was released last year. But this and many other video games, and by extension,

these experiences, are largely inaccessible to blind and low vision or BLV players. Now, this doesn't preclude blind players from gaming. There exist what are called audio games, which are the audio-centric equivalent to video games. They often feature no graphics, and instead center on sound effects and other announcements to provide a gaming experience.

But the issue with these audio games is that they just aren't the same, in terms of how fun they are for players. And to illustrate this, let me play a short clip from Terraformers, a classic audio game from 2003. Now, Terraformers has rudimentary 3D graphics, but for the purposes of this clip, I've removed them and replaced the visual portion with captions. So here it is. And that audio...

So I don't know about you. But the game play here seemed a bit simple. It almost reminded me of one of those old school text-based adventure games. And despite Terraformers being one of the best audio games out there, the game play felt linear, it was less dynamic, and it was just less interesting overall. And we've seen this sentiment in our conversations with blind gamers as well. Here's a quote from a blind gamer during our studies, and they said: Quote, we don't really get a lot of choice in audio games. They tend to be really simple and not a whole lot of fun, so I just play whatever is out there that I can actually stand. End quote.

So if audio games typically aren't a suitable substitute, then we should bring our attention back over to mainstream games. And it's a fact that many blind players desire the ability to play mainstream games. We've seen this in our own interviews and even beyond. For example, there was a great article in the Montreal Gazette last year that focused on gaming for blind gamers.

And there was a great quote from one of the blind gamers they interviewed. And they said, quote: The main thing for so many of us is to see more mainstream games accessible, versus solely having audio games with zero graphics. It makes it difficult to talk to sighted friends about games. Some of my friends don't comprehend that while I enjoy gaming, I don't play anything remotely similar to most, short of Hearthstone. End quote. For those who are unfamiliar, Hearthstone is an online card game for PC and mobile, and the PC version has an accessibility mod that makes the game screen reader accessible. But how do we go beyond simple card games? Let's think through it. As we all know, video games contain a lot of video and video contains orders of magnitude more information than audio. So the first question that would come to mind is,

well, how do we communicate all this info, all of these rich worlds, through sound? But I propose that maybe that isn't the correct question we should be asking ourselves. At least, first. And to motivate this, I want to ask another question. That question being: What makes a game fun? And this is a question that many game designers over the years have attempted to define, and they've attempted to formalize this notion of fun. And one such formalization is called the 8 kinds of fun. Which posits that games and experiences within those games embody some combination of these 8 categories, in order to facilitate a sense of fun within players. And just to kind of go through them, read them out quickly, they cover sensation, which sees games as sense pleasure, fantasy, narrative, challenge, fellowship, discovery,

expression, and submission. Submission, seeing games as mindless pastime. Now, what's important to note is that only one of these eight kinds of fun deals with the sensory, the visual experience of playing a game. That is sensation, which sees game as sense pleasure. Everything else focuses on what a player can do, what they experience, and what they feel within a game. So I propose that the approach to more equivalent accessibility in gaming is to focus on the experiences we can grant players. The information that we need to communicate will then follow. For example, when a player -- when a sighted person plays a mainstream 3D game, like the Legend of Zelda, what kinds of experiences are they having within the game world? And how can we translate those experiences,

those abilities, to a blind player, so that they can not only play those games, but also they can feel like they're having fun and feel fulfilled in the process? So my PhD has looked at how we can take steps toward this. And so during this talk, I want to talk about the stuff we've done. Focusing specifically on the experience of traversing and exploring 3D game worlds. And as part of this, I'm gonna talk about three projects that I've done during my PhD. And I want to motivate the first project by going back to that clip from the Legend of Zelda. Again, this is a player controlling the main character, Link, through this small town. Now, one of the basic operations that this player is doing is they're scanning their surroundings. What's in their line of sight

right now? And where specifically are these things that are within their line of sight? At one point, there's a statue to the player's right, to Link's right side. In the distance, there's a woman sweeping the ground. Further into the distance, there's a workshop in that large building. And if they go near the workshop, they're gonna see that rail car that's way out in the distance.

This thought process is facilitated by our ability to quote-unquote "look around". Which is the ability to scan the environment in any direction, to find items, detect enemies, and otherwise interact with the world. And it is this ability that we focus on, within our first project. Which is called NavStick. Now, as I mentioned, a fundamental component of the exploration experience is the ability to look around. And it is through this fundamental act that players develop a plan of action, and attempt to execute that plan of action. Allowing them to exercise their agency and control within the world. And it involves surveying in any direction at any time on demand. Like how a sighted player would use the in-game camera to do the same thing. Without needing to go through a list or needing someone to describe, for example, what's

in the world around them. So to facilitate this experience, we created a tool called NavStick, which is an audio-based tool that allows blind players to look around themselves. Quote-unquote "look around themselves", within the game world. And the concept behind NavStick is actually pretty simple. Basically we took the right stick on a game controller and we repurposed it such that in whatever direction they point the stick in, the game is gonna announce what lies in that direction via line of sight. So in this particular screenshot, which is from a prototype game we created, the player is pointing the right stick in the 1:00 direction, and we perform a raycast in the 1:00 direction from the player within the game. So in this case, that raycast is going to hit this little enemy sitting right there, which we call the Chomper. And the game is gonna

announce: Hey, you hit a Chomper just now. And as I mentioned, it works via line of sight. And all these announcements are made via 3D or spatial sound. So NavStick, at its core, is pretty simple. But we really wanted to test how well it worked in different situations. And so we did two studies with NavStick. For the first one -- so for the first one, we looked to compare NavStick with the status quo means of surveying an environment. And that status quo means is basically reducing the world down to a menu of items. We call this tool NavMenu. We compared NavStick and this menu-based system with each other across a variety of tasks within a virtual grocery store, the aisle that we created. And the goal was to basically explore each tool's advantages and disadvantages

across six tasks. The first five had players use NavStick and NavMenu to survey the items within this aisle, to answer some questions we asked. And there are different categories of tasks. But the gist is that we wanted to see how well the tools performed in the context of wayfinding. How well, for example, did NavStick allow participants to gain a cognitive map of the aisle versus the menu system that we created? And for what tasks the participants prefer each tool for.

And after that, for the 6th task, we also added a very simple video game level. This was a recreation of a room found within Terraformers, which is that audio game I played a clip from earlier. To look at how participants perceived NavStick within that sort of environment. So as for a few findings, we saw that participants felt they were able to build better mental maps with NavStick over NavMenu, they felt they were able to have more fun with NavStick in that very basic game level from Terraformers, but we also found that they preferred the menu for what we called quote-unquote "non-directional tasks", which is a task that allows you to find whether an item is in the room. As you can imagine, just scrolling through the menu is much easier

than using NavStick to look around you. The paper itself has actual numbers to support all of these and more. I didn't want to bore all of you with those numbers. But what I will say is that while it's nice to more objectively look at how well people understand the world using NavStick, it's more interesting to put NavStick in an actual 3D video game, where it belongs, and see how it performs there. And so that's what we did for the second study.

Where we custom built a 3D video game that we called the Explorer, derived from Unity's 3D Game Kit, and integrated NavStick into that game. And using this game as our study's test bed, we had 7 participants use NavStick to help them traverse that 3D game world that we created, avoiding different obstructions and defeating enemies as they wanted, using a laser gun. This game world consisted of 8 different segments or shorter levels. And we designed them to possess different environmental characteristics found in 3D games that could pose a challenge to NavStick.

This clip right here shows one of these segments. And the character assisted movement included different things. Here you have a player pointing a laser gun at something. That was one of the characteristics. There were large trees scattered throughout the levels, that counted as occlusions, that could potentially make NavStick a little more difficult to use, and in other segments we also had a timer going in the background, that added an element of time pressure as well. You can actually watch this video on my website. I'll provide a link at the very, very end. Now, as for a couple of findings from this study, we saw that every participant actually completed every level within this game. All these blind participants completed the game fully. And we were also able to see how NavStick offered players with an enhanced sense of agency and control. Looking

at the actual paths participants took, we can see how participants employed different strategies to get through the level. For example, avoiding these black circles that in this figure -- which is an overhead view of one of those segments -- represent trees, in addition to avoiding and defeating a lot of those moving enemies that are just roaming throughout the levels. Some of them even roamed near the end of the level as they were waiting for the door to open. But we also found some of the characteristics, those environmental characteristics, that we integrated, caused participants some trouble as well. In particular, levels featuring occlusion had some of the highest frustration levels. Which makes sense. Because NavStick in its current form only works via line of sight. So this perhaps points to needing ways to allow players to almost...

Explore behind or see behind those occlusions. And we kind of get to that a little later. But even despite that, participants constantly remarked that NavStick made the experience of playing the game just fun and enjoyable. As an example, here's a quote from a participant in the study, who like all the participants was blind, who said: Quote, "if you stop to think about it, in most games, the whole point is that you can move the camera. And to me, NavStick was the equivalent of a camera. You're able to see objects around you. You're able to know where they are." End quote. And so that is NavStick. And before I move on, I'd like to mention that our lab is actually working to convert NavStick into a plugin for the Unreal Engine. So that developers using Unreal can integrate it into their games to make them more

accessible. I think it's almost done. It should be out relatively soon. So if you or anyone you know is looking to make your game more accessible, you might be interested in this. But moving on, that is NavStick. Where we looked at that ability, that experience, of looking around the game world.

Now, our work with NavStick led to a broader question, which was the subject of the next project. And to motivate that project, let me go back once again to that clip from Legend of Zelda. An important part of traversing a game world involves understanding your own state within that world. In addition to what's around you, there are additional questions, like: Where are you?

What direction are you facing? What is the size of the area? What is the shape of the area, even? These questions fall under the realm of spatial awareness. What we call spatial awareness.

Which is a person's awareness of their surrounding environment, and how they are situated within that environment. For us, as sighted players, we obviously have the use of vision, as well as small tools like mini-maps, for example, to answer those questions. But what could an equivalent tool or tools look like for blind and low vision players, that would grant that experience of being spatially aware of the environment? Or more fundamentally, how well can different tools communicate a sense of spatial awareness to blind players, and what do players desire in terms of the information they want to get? And so that is the... That was the subject of our second project, which looked at the concept of spatial awareness tools, or as I'm going to abbreviate them, SATs. This is a term we coined to refer to this class of tools. So to start off,

let's look at different existing ways of communicating spatial awareness to blind players. One of these simplest techniques for communicating spatial awareness is via the use of environmental cues. For example, if a player hears a waterfall in some direction, they know there's a waterfall around them, of course. But using just environmental clues can become overwhelming in much more complex environments that have a lot more audio cues and a lot more going on. And so there exist other more explicit approaches to communicating spatial awareness. These include the use of tactile maps and displays, for example, that show the player's current area. This figure is from a paper that introduced a blind accessible game called Shadow Rhyme, which was able to make use of a dot-based touch display to visualize the walls and area around the player. Other

approaches include the use of techniques inspired by echolocation, which, for example, The Last of Us's enhanced listen mode, which basically sends out a shock wave around the player, so they can discover the locations of items around them. There's also what NavStick introduced, which is the concept that we referred to going forward, as directional scanning, where you basically just select a direction that you want to survey in, and you basically learn what's in that direction. And finally, there's the use of menus, simple menus, like NavMenu, for example, which just lists out what's in the world, and you just point and click and move on from there.

Now, these techniques that more explicitly communicate spatial awareness are what we, again, call spatial awareness tools. However, the issue is that we don't know how well each of these SATs facilitate spatial awareness, and we also don't know what aspects of spatial awareness are important to blind players. Now, given that game designers and developers can't implement everything, which of these tools should they pick for their game? And so to answer these questions, we performed another study, where we -- you guessed it -- we created another video game.

Where we implemented the four SAT approaches into this 3D game world that we created. We represented tactile maps and displays using a smartphone-based mini-map system. For this system, basically as the player goes through the game world, the map would pan and rotate along with the player, as they move through the game world. So that represented more tactile ways of exploring an environment.

We also represented echolocation using a whole room shock wave technique that was inspired by the enhanced listen mode's technique. Directional scanning is of course represented by NavStick, which is basically the same thing I just showed you a few minutes ago. And menus are represented by a simple menu, representing the contents of the room, similar to what NavMenu did in the NavStick study. The game consisted of four levels within a dungeon-like environment that we created. And we had nine blind players go through these game levels using the four tools that we created.

For each level, like the one whose overhead view is on the screen right now, we gave players one of the four tools to use. And they used that tool to traverse the level from a starting point, going through multiple rooms, to collect a key that was hidden somewhere, using that key to open a lock that was hidden somewhere else, and then opening that lock to reach a finished checkpoint that was just past that lock. Now, every level basically followed this same sequence, but they all had unique layouts. And so some of our findings from this particular study -- we first wanted to figure out how important players felt that different... The importance of different spatial awareness types to blind players. And so we looked at 6 specific types of spatial awareness that were mentioned as important to people with vision impairments, across a wide

body of prior academic work. These included the scale of an area, the shape of an area, a user's own position and orientation within that area, the presence of items within that area, the arrangement of items within that area, and even an awareness of areas adjacent to the player's current area. And then we interviewed players to learn which of these types they found were important. And we found that participants believed that knowledge of their position and orientation was the most important, and that was always helpful in knowing their current situation within a room, in order to plan out future actions. Tied for second, right below that, were item presence, item arrangement, and adjacent areas arrangement. Because participants thought that that information was helpful, but that communicating it too much could make the

game less fun. It would remove the element of surprise, often. And finally, at the very bottom were area scale and area shape awareness. Which participants thought were wholly unnecessary.

Most of the time. Now, these results are kind of important. Because you would think that a tool for communicating spatial awareness should communicate position and orientation well. But we found that none of the four tools that we implemented excelled at communicating position and orientation. Participants didn't feel that any of those tools allowed them to easily ascertain where they were within the level. So we think there's an opportunity for building tools that better communicate position and orientation within game worlds. And on top of that, we also explored the concept of combining tools. Many participants wanted to combine tools. They wanted multiple ways of surveying their environment. And one combination participants desired was the directional scanner, AKA NavStick, alongside the simple audio menu. And we found that combining these actually gave players the greatest spatial awareness out of all the tools. So that was a

finding that we had, and participants desired to mix and match tools also. That kind of taught us that customizability is very important. Whether through choosing the tool that you want to use, or even turning on or off specific info, or tweaking the range and sounds of different tools. And all this is helpful, because we saw that players have specific preferences on what kinds of information they want to hear. And that there are opportunities to better communicate certain types of spatial awareness. But it's clear that there are paths. For example, by combining these existing tools. To give blind players more of that sense of being spatially aware. Within a complex 3D environment. And so that is the second project, where we looked at the spatial awareness tools and the ability and the experience of gaining spatial awareness.

So we have our work with NavStick, and we have our work with spatial awareness tools. But this led to an even broader question, which was the subject of the next project. And so to motivate this, let me go back to... You guessed it... The clip from the Legend of Zelda. I swear this is the last time you're gonna see it. So what is actually going on in this video? So what the player is doing is they're running through. They're looking around the environment. Eventually they see something in that environment. Something near that building over there. And begin running over to it.

That is a typical example of a sighted player's experience within a game. They see something that piques their interest in the distance and begin running toward it, or find an opening that might lead to new things. And this all connects to the concept of discovery. Which sees games as uncharted territory to be explored and uncovered. The joy and excitement that players feel when going through a game world is an important part of many games. So how can we grant that same experience, that same feeling of discovery, to blind players? And so that was the subject of the third and final project that I'll be talking about. Which is called surveyor. Where we looked at that concept of discovery. Now, the main issue is that existing approaches for communicating game worlds to blind players don't really grant this experience of discovery. And these approaches

generally fall into two main categories. The first involve simplification techniques, where the game world is reduced to, say, a list of items in the environment. This obviously reduces the game to a simple point and click task, and isn't that all equivalent to what we saw in that clip in the Legend of Zelda? And the other major approach involves what we call spatialization.

Giving players some sense of the objects around them, through 3D sounds. The Last of Us games are especially noted for this. Sounding pings that provide players with a rough sense of what's around them. But they can't actually do much with those pings as-is. And they're forced to follow a predefined path to the next objective. In that case as well, that sense of discovery isn't there.

And the experience that blind players are getting isn't at all equivalent to what sighted players get within that same game. So in this project, we wanted to create a different way of approaching exploration, I should say, to give blind players that sense of discovery. In order to do that, though, we needed to figure out what kind of scaffolding would be needed to achieve this. And so we spoke to a pair of veteran blind gamers as part of a pair of interviews to learn about the abilities that blind gamers desire, in order to effectively explore game environments and experience that sense of discovery. And our interviews yielded three of these abilities.

For the first one... Both participants mentioned that they wanted to move through and look around environments, rather than rely solely on menus and maps to discover items. For example, in this image from our test game, a sighted player would see the opening between the crates in the distance, and run toward it. Possibly discovering something right beyond that opening in the process. Rather than just being told where everything is, these players... These participants... Wanted to go through the actual process of exploring.

Second is that they desired the ability to keep an exploration log. Desiring the ability to know where they've explored and in which direction they should go to uncover more of the world. Some video games, such as Red Dead Redemption 2, shown here, obscure unexplored parts of the world map. That way, players know what they've explored already. And in which direction they should go to uncover more of the world. And finally, they mentioned that they'd love the ability to quote-unquote "fast travel". Fast travel to locations they've already discovered. Many games like Skyrim, pictured here, provide this ability. Whether it's through warping directly to their destination, or through markers that act as beacons that players can follow. Now, given these three abilities, we need to understand how we can actually manifest them within a tool

that blind gamers can use. And so we ended up developing and investigating a new system called Surveyor. Surveyor is -- the definition is that it's an exploration assistance tool for virtual worlds, designed to facilitate a greater sense of discovery in blind and low vision players. There's a clip on the screen playing right now that shows Surveyor in action. But what I'm gonna do is spend the next few slides explaining what's going on here. So we treated those abilities as design goals so Surveyor responds directly to them. The first is the ability to look around, which as you might have guessed is fulfilled by NavStick. It's basically what I've been describing all along, where the player points in a direction using the right thumb stick and the game announces what lies in that direction. In Surveyor's case what NavStick also does is

it marks areas that have been swept by NavStick as "seen". So in this particular clip, as the player sweeps around themselves, using NavStick, the area they've seen is being marked as explored. That's what all those red cells represent. Everything beyond their current line of sight remains unexplored. And this directly hooks into the second component, which is the exploration log. And this is fulfilled by Surveyor's exploration tracking system that we created.

So let's just dive directly into what's going on here. So here's that same room that I've been showing. With the player's initial scan complete. Everything that was in the player's line of sight from where they are right now, and that they swept using NavStick, is marked as explored.

Everything else that wasn't in their line of sight -- for example, the area behind the large crates -- remains unexplored. Because it wasn't in their line of sight. In this particular room, the player's goal is to find and collect a key card, located behind the crates. And in this visual, I'm just gonna spin our point of view so we can more clearly see the key card. The best way for a player to discover or find that key card is to go to the very edge of the area they've explored.

That edge is effectively the cusp of the unknown, basically the edge of what they know right now. Kind of like that Fog of War mechanic I was talking about. So Surveyor's primary objective is to highlight these edges, and it does so through a specialized menu system we created. This menu system not only communicates these unexplored edges, which potentially hold items and other things beyond them, but also landmarks that the player has seen, as well as information about rooms that are within the player's current knowledge base. But that they might not have explored or they might have not fully explored so far. So players can select anything in this menu to be guided there via audio beacons. In this particular case, the player decides they want to explore the first patch within the menu. They select it. And a series of audio beacons lead them

to that edge. And when they're at that edge, they can use NavStick to survey around them. At which point, they'll find that key card. They'll discover it. So that is Surveyor. And again, I have videos with audio showing how this works. And I'll provide a link at the end that actually contains these videos. So now that we designed Surveyor, we need to understand how well it worked, so as you might have guessed, we created another video game to test our tools.

In this particular study, we wanted to see how well Surveyor facilitates discovery. And how the experience with Surveyor differs from existing in-game tools. I'm showing a couple of these Sci-Fi industrial environments. Those are directly from the game world that we created. Now, we had nine blind participants in a fully remote study, as are all of our studies, make their way through three of these game levels, using Surveyor and two other tools, representing the status quo. And we've seen these tools before. One of them is the simple audio menu, which represents those simplification techniques that I talked about. And the other is a shock wave utility, which represents the spatialization techniques, such as the ones found within The Last of Us. Now, as for the findings from this study, participants generally thought that they had a stronger sense

of agency in exploring the world with Surveyor. Participants felt encouraged to explore the levels more. And we could see this actually in the raw paths they took. For example, in this particular visual, from the starting segment of level two, all three participants that used Surveyor actually explored the entirety of these rooms, including the side rooms, off to the left, before moving on to collect the key at the top. Whereas only two of three participants -- only one, I should say -- participant used the audio menu to explore the entirety of the area, and no participant used the shock wave to explore the entirety of the area. Participants who used the menu were able to just select where they wanted to go, and just went straight there. And the shock wave didn't provide any information about the side room, which is exactly how The Last of

Us's enhanced listen mode worked. With the simple menu, participants found the simple audio menu to be much simpler to use than Surveyor. As you could probably tell by the name. Participants found this aspect of the menu to lead to a more relaxing experience, as some participants called it. Where they could easily traverse the level. And this explains why a couple of participants actually preferred the menu over Surveyor. But some of them also noted that they would need more time to get used to Surveyor as well. This whole study session was only 90 minutes to use all three tools to get these levels. But most participants found the menu less interesting to use. In fact, one participant used the words "sterile and soulless" to describe their experience with the menu. Because the game

was just basically reduced to a simple point and click task. And they just had to go from point to point to finish the level. And finally, with the shock wave, every participant felt they could complete the level very quickly with the shock wave. In fact, the shock wave facilitated the fastest level completion times of all three tools. But most participants, I should say, disliked the lack of exploration. In fact, many disapproved of the enhanced listen mode. The spatialization techniques, within The Last of Us.

With a couple of participants admitting that they didn't even finish The Last of Us, because they couldn't do much within them, with the tools they were given. And so that is Surveyor, where we created a tool that facilitates the exploration of discovery. Within game worlds. And all three of these projects, the NavStick project, the SAT project, and Surveyor, are all efforts at following that approach of looking at the experiences we can grant users, and building around those. Instead of just looking at the accessibility of those worlds, mechanically, through the lens of information. But of course, I think this is just the beginning. I think there are a lot of different areas where we can continue working along this mindset.

Both in games and beyond. And so again, the fundamental question is: What kinds of experiences can we translate over to bring that sense of joy and fulfillment to users? And so I want to spend the last few minutes talking about some of these areas. Now, obviously one of the main goals of equivalent accessibility is to allow blind players to play the same mainstream games, with just as much fun and fulfillment as sighted players. But I also think there's another layer to equivalent accessibility. And that revolves around the concept of multiplayer gaming. Or what at least the academic literature calls mixed ability multiplayer. I think that the Holy Grail, so to speak, is to create experiences that are so equivalent that both blind and sighted players can actually play together on the same field.

And this could open up a new class of games, and even social experiences. Here on the screen I have a screenshot from Valorant, which is a tactical multiplayer shooter game from Riot Games. I picked this screenshot because the area reminded me a lot of the NavStick game we created. The player is holding what seems like a laser gun of sorts, there are walls and crates throughout the levels. But you can replace this with any similar multiplayer game. But there are certain kinds of experiences that could enable multiplayer of this kind. For example, being able to strategize over a map of the field. How can we provide an equivalent experience in that sort of strategizing?

Being able to perceive slight movements in the distance. For example, someone from an opposing team is trying to sneak around you. You would want to know that they're sneaking around. You would want to be able to perceive those slight movements. So how would we create an equivalent experience for that? On the flip side, being able to sneak around yourself, knowing where you can move, without being seen and figuring out if you've been seen as well.

And being able to quickly identify allies and enemies as well. In this screenshot alone, from Valorant, there are little silhouettes that denote the locations of players as well. So how can we translate that over as well? Now, they're all valuable, and they're of course very simple. But it's things like this that make the experience of playing a multiplayer game, in my opinion. But I also think that we don't have to stop at games in thinking this way. A fourth major project of my PhD actually dealt with digital image accessibility. Specifically what kinds of tools can we create to make it easier for blind users to explore images using their finger on a touch screen, like on a smartphone. So we created a bunch of techniques, like using a menu

and beacon system, which, as you might imagine, was inspired by the video game stuff, to basically list out the components of the image and guide users toward those points. We used a neural net to highlight different areas that might be perceived as important. And we even created a zoom-like technique that allows users to blow up parts of an image, so they can more easily explore these very minute parts of the image. Obviously the default way of describing an image is via alt text. But alt text involves a sighted person imposing a description of the image on the blind user. In fact, there was a participant in our study for this project who talked about descriptions in alt text being quote-unquote "confining for blind users". When it comes to understanding images.

And it's -- if you think about it, perceiving images involves, you know, understanding not only what the image is showing, but also understanding how parts of the image relate to one another. Understanding and figuring out what parts of the image are more important. And less important. Eventually -- the goal is to allow a user to form their own understanding of an image, without someone else imposing a description. And I think there are meaningful applications for this as well. For example, in personal photos and actually perceiving what's in a personal photo, and even in education, providing views of different graphics to blind and low vision students as well. And the final area that I want to talk about is the physical world. Specifically in navigating the physical world. Now, this image right here

shows... Is a screen cap from a demo video. It is me from six years ago, actually, testing out a turn by turn indoor navigation system that I created for blind and low vision users. I worked on this project for, like, two or three years of my undergrad, as part of my research. And basically the system used different techniques, like Bluetooth beacons and even other computer vision-like techniques, to really accurately pinpoint where a person was within a building.

Now, the reason why I bring this up is because this was a turn by turn navigation system. And turn by turn gets you from point A to point B, and gets the job done. But it too is pretty confining, if you think about it. A blind user doesn't have much freedom to do what they want with a system like this. They have to follow its every instruction. So what experiences and abilities could make the experience of navigating the physical world more rich and interesting? Perhaps it's the same things we found in video games. For example, looking at... Learning about a space by looking around yourself. Perhaps it involves quickly gaining an awareness of one's orientation, one's position, where items are, or as we call it, spatial awareness. And sometimes navigation isn't

about getting from point A to point B. It's about experiencing the world. And perhaps... And so perhaps being able to discover new and interesting things, while you're going through places, can be translated over. For example, in parks or malls or buildings, or even museums, for instance.

Now, this might look a little different in the real world, but they still apply. Because at the end of the day, I believe that achieving equivalent accessibility means realizing the experiences, however simple they may be, that bring joy and fulfillment to users. And I believe that following this line of thinking can open up a lot of new opportunities for users, whether it's personal or social or even professional. And you know, this is an accessibility meetup. I presume a lot of us here work in accessibility. Many of us might even use some of these technologies.

But whatever our role, I would love for us to think about how we can move toward a future where technology can allow us to collectively share in all of these core experiences, regardless of our abilities. Thank you, everyone.

My audio... Am I... Oh, there we go. Cool. Thanks for that awesome talk. We have a lot of questions online. And I'm sure we've got some in the audience as well. So I'll start with an audience question, and then I'll kind of go back and forth.

Hi. Kevin Andrews from DC. Great talk. Just wanted to kind of piggyback off... Sorry about that. Wanted to piggyback off one of the things you talked about, which is... Use of technology. What if any role do you see AI playing in sort of facilitating greater accessibility for games? For blind and low vision folks?

Um... I think... You know what? When it comes to AI and games, I think what I'm most excited about is perhaps a lot of the vision models that have been coming out lately.

Where you can basically... You know, you take it, put an image through this, and it will tell you what's in that image. Right? And... I think... Again, this is purely speculative. But one thing that has been going through my mind is... You know... What if we use these vision models to kind of form a representation of environments within games that are not accessible? Like... For example... That Legend of Zelda, for instance. What if you put all those scenes through a vision model, and it told you where everything was, with respect to your line of sight and your camera? So I think there are opportunities. Obviously I don't know how well that would work. But I think there's huge potential in an idea like that. For sure.

Yeah. And I'm excited about that.

All right. Sorry. Question from YouTube. Does NavStick have any impact on frames per second?

We didn't observe anything. I will say these are games that we created. But the computers our participants had pretty much spanned the range of processing power and memory. And nobody really recorded any sort of issues. So... I don't think it had that much of an effect.

Cool. And feel free to raise your hand in the room if you've got a question. And then I've also got plenty online as well. So I'll go back and forth.

For NavStick, if you were to use it with an off the shelf title, would a publisher be required to kind of go in and tag every item that you would point the NavStick at?

Effectively, yes. When we implemented our video game, we were able to... I guess driving into more technical terms, we were able to tag every prefab. So, like, we tagged one Chomper, and then we just copied all the Chompers over, and they were all tagged by extension. So there are ways to more efficiently do it. But I think that in order to make it a little more, I guess, easier, having a more automatic way of doing it would be super helpful. For sure. Yeah.

Cool. From online -- and Walei, I'll get yours next. Did the findings from your study significantly differ between participants with no vision, totally blind, versus those with some functional vision?

So across most of our studies, I think the vast, vast majority of... Can you repeat the question?

Did your findings significantly differ between participants with no vision versus those with some functional vision?

Some functional vision. Okay. So I think across most of our studies, everyone was totally blind. Completely blind. I think a few folks had some light perception as well. Though we didn't find any differences in that case.

Cool. Walei?

Question about, you know... Handling sort of cognitive overload when you're using -- whether NavStick or Surveyor. And it's telling you... Well, we didn't hear how it sounds. So I'm not sure what it sounds like. Right? Is it a screen reader? Is it audio cues? How does it work? That's one question. But... Also, how did you manage cognitive overload? Because what if there's 50 things... Not 50. But let's say 5, 6 things in front of you. How do you handle that? Right?

In realtime? Where users are trying to just flow through the game and do things kind of quickly and not just sit there and you know... Take in one object at a time, so to speak.

Right, right. So in terms of how NavStick works, it basically... Again, you can customize this, but how we implemented it for some of our studies is that it read out a description of the item. So the instant you point to it, it starts reading out that description.

And then the moment you point away from it, it stops reading that description. It cuts off at that point. And if there are a lot of objects around you, then you will hear all of those objects. Though if they're all in front of you, then they basically get clustered together into this sort of... There are three Chompers in front of you, or things like that. So yeah. So I guess in terms of cognitive load... I don't think anyone from our... At least, I don't remember anyone from our study specifically mentioning that. I think they all liked it. But yeah. I don't know if there's... I don't know if that answers your question. I should also plug the fact that on the screen right now, there's a QR code with a link to my personal website. And this

page specifically has all the NavStick -- the demo videos for all the projects. So yeah.

Can you read it off?

Yes, I can. It's NairVish.com/playingfield, with no spaces

Another one from online. Fallout 4 has fast travel. However, if you go into the harder mode, it disables it. Curious to know if gamers feel that fast travel makes the game easier or not.

It's not something that we really looked at. I guess in this particular context, fast travel refers to just using beacons to actually go from point to point. Instead of directly warping to that location. I'm not... I'll be honest. I'm not sure about answering that question. But... I think... Yeah. We didn't really look at the concept of disabling it. Yeah.

Fair.

Hi. First of all, thank you for sharing. I learned a lot. My question... And this is always my question with any kind of assistive technology... Is...

And forgive me if you did mention it, but... What are the goals of your PhD, ultimately? And what are the results, beyond this, that could be real world? Is this Open Source as well? But I would love to find out ultimately... Is this going to be something that's available to the wider public? And what are your goals there?

Right, right. So... In terms of whether or not it's available to the wider public, I think I mentioned at the end of the NavStick project that our lab is actually working to convert NavStick into a plugin that developers can use. So that they can add it to their games. And this is specifically in Unreal. And it's almost done. It should be out relatively soon. So that's something we've been working on for a couple of years now. So... Yeah.

I'm gonna do... I think we've got time for maybe two more. So I'll do one more from online. One more from in the room. And then we've got to let our captioner go for the evening. But then... Anybody in the room can stick around for another 20, 25 minutes. Feel free to mingle and ask questions and all that. So... From online, how fast were players able to use the SATs, and can... I can see SATs slowing down the game play, especially for action games. I think you covered that a bit. But do you want to speak to that a little more?

Yeah, yeah. It really depended on the SAT. For example, one of the tools that we created was the smartphone map. That definitely slowed down people, because they had to play the game, and then they would go to their phone and kind of scrub around and go back -- a few participants actually talked about that. In terms of... You know... The other ones, NavStick's, shock wave, and the menu... I definitely do not remember the exact completion times. It might be in the paper. But I don't remember them significantly slowing down players. Especially NavStick.

Since it works in realtime, pretty much.

Does that have any impact on time-based task completion? Or is that something that you looked at?

I don't... Hm. Yeah, I don't necessarily know, to be honest. I will say that for the second NavStick study, we did add those elements of time pressure. And it didn't really cause any issues, to be honest. I think participants weren't frustrated at all with the timer and NavStick. Yeah.

Awesome. We'll do one more from in the room.

Hi. What is the best way for us to follow this work? You know, you've done it for a couple years? A couple months? I forgot. You said NavStick is going to be available as a plugin for developers.

How can I follow along with that?

So you can basically follow our lab, the Computer Enabled Abilities Lab, at Columbia. It is... And I can give you the link and stuff as well. But... It is our lab that's working on that effort, and then once it's out, we can pretty much let you know. And if you follow our lab's work, we do a lot of work in accessibility, actually. So... That should be the easiest. That'll be the easiest way.

What?

Abilities Laboratory. CEAL.

Computer Enabled Abilities Lab.

Yes.