Mobile Accessibility Tools: Camera Switches and More

Great. So... Yeah. So my presentation is titled mobile accessibility tools.

And as Ben had introduced me, I work mostly on the mobile platform for The New York Times. Specifically our games. And so our games and many other properties, in general, are on mobile and have been on mobile for a while. So mobile actually has...

Just a little precursor... Its own accessibility guidelines and standards. They kind of align with the web accessibility guidelines and standards. But that's just a precursor.

What I'm gonna be talking about is some of the things that maybe most or many have not seen, as far as some of the technologies that are mobile specific. If you're not used to the iOS or Android accessibility tools. And yeah. And we'll go from there.

I just wanted to give a little bit of a note that most of my demos are actually recorded, because it does require some amount of coordination to actually use the tools. And I wanted to make sure that I could get through both the configuration of the tools and the actual implementation and the usage of the tools. So yeah. Just wanted to...

When we get to the demos, you'll see what I'm talking about. So... Gonna give you the outline of the presentation. So an introduction to myself, again.

New York Times, working on the Games platform. So the first thing I'm gonna be talking about is accessibility shortcuts. So those are easy ways to set up your device, so you can get to the accessibility options very quickly. Quick I guess...

Just polling of the room here... How many people are familiar with accessibility shortcuts on devices? Okay. Great.

So a fair amount of people. So this is gonna be in that case more of an overall -- this portion is gonna be more of a review. But it's important, because it makes... If you're not, getting through the settings menu is a little bit tricky.

So every time you want to turn on an accessibility switch or accessibility option, it's easy to do it when you've got it set up. Then I'm gonna be talking about VoiceOver. I'm sure many of you are familiar with it here.

And then I'm gonna be talking about camera switches. Who has used camera switches here before? Okay. Great.

This is where maybe some of the new knowledge might come in. And then I'm gonna be talking about accessibility scanner. So if you're used to contrast checkers on websites for checking accessibility criteria, if they're met, as far as color contrast ratios and things like that, I'm focusing on the Android device here. But...

Accessibility scanner is the equivalent of the app-specific contrast checker and accessibility checker for Android. But I'm gonna go through a demo there. And then I'll talk about a few miscellaneous things here and there. So the tools I'm gonna be using are just basically Android devices and iPad.

Which I have here. For an interactive demo later, if anyone is interested. So the first thing I'm gonna be talking about is just a little bit of the history of accessibility and accessibility for mobile devices. Mobile devices haven't been around for that long.

They've been around for a while, but not, like, that long. And initially, with the very first mobile devices, which are really mostly Nokia devices, Blackberries, et cetera, PDAs, Palm Pilots, basic accessibility was text to speech and voice recognition. And that was what the limit of that was, originally. Because mobile devices were not very advanced, and they were not used by many people.

Early on. And so... So as time went on, the iPhone came out in 2007, and then in 2009, they started building VoiceOver into the iPhone. So we started getting the ability to do things like using a screen reader, and touch screen accessibility and other features.

And then Android really followed and started in 2008, the first really usable market Android device. So Android started in 2011, providing TalkBack and other accessibility tools, including a lot of comprehensive accessibility options. Though the difference with Android is that Android -- the operating system, the OS, is very -- it's very sparse and bifurcated, and there's many different Android devices. So it took a while to unify the Android accessibility options across all the devices.

So... And then moving forward... And then the W3C released the WCAG 2.1 standard, which incorporated mobile-specific accessibility guidelines. So this was important, because mostly accessibility guidelines focused on web, beforehand.

And the rise of mobile devices and mobile first applications and mobile first thinking gave rise to the idea that... Well, most people are actually using their mobile devices. So the standards should be very specific for mobile. I'm not gonna do too much into the mobile-specific standards, but there are a lot of very interesting mobile things, mobile considerations, that are not necessarily applicable to web directly.

So that's just something to consider. It's very interesting, if you want to look into it. I've been involved in mobile accessibility committees and things like that. So it's very interesting.

In that regard. So since most people said they're already familiar with mobile accessibility shortcuts, I'm just gonna briefly buzz over this. The purpose of mobile accessibility shortcuts is to allow users to rapidly enable and disable features like screen readers and magnification. Again, if you go through the rigmarole of setting up accessibility features and going to the accessibility menu, it can be quite complicated.

So there's different shortcuts on different devices that make it easier to access and turn on your accessibility features very quickly. And that's a very... Now the standards are pretty uniform across devices, including Android. So it can be tailored to the specific needs of the user.

Again, some of this might be review for most of you. But... What the interesting thing is -- that you can set up your shortcuts in very specific ways, per your device. So, for example, if you prefer that the accessibility shortcuts are usable through the volume buttons, great.

They're configurable in that manner. If you want them accessible through a software accessibility button that you can move around the screen, they're also available in that manner. And so on and so forth. It's very customizable these days.

And you're not limited to one option. And then... So there's lots of different advanced features like assistive touch -- is anybody familiar with assistive touch? Great.

So there's assistive touch, there's several Android variants of assistive touch. These are ways that you can, like, use an accessibility button without necessarily having to navigate through the hardware button. Again, it's a software menu, where you can see options. Like: Go to home screen.

Go to back. So all of these are -- there's physical options, and then there's software options. Which makes the whole world of accessibility a lot more accessible to people. And then...

Accessible options and accessibility options themselves and shortcuts are essentially very... Just by nature, of them being usable anywhere, and configurable very dynamically, they're inclusive by nature. Meaning that anybody with any preference and any preference that makes the interaction with the device and their lives easier, can set it up in a particular way, and as time goes on, this customizability is so open that it includes... The way that the thinking is -- it includes everybody.

Okay.

So... Now I'm gonna let the recorded me do some talking. And hopefully you can hear. I'm gonna just go through the setup of some shortcuts on Android devices and iOS devices.

I'm gonna start with Android, because I'm an Android-first person. I don't know how many are here. But... Hey!

That's who I am. Gotta be true to yourself. So...

I'm gonna show you how to do two things. I'm gonna show you how to use the accessibility buttons and set up accessibility shortcuts on your home screen of your Android device. These are important, because it allows you to have quick access to accessibility features without having to go through the settings menu. Which is a multistep initial process.

It makes life easier for anybody who needs to access and turn on their accessibility features very quickly. So the first thing I'm going to do is show you how to install switch access. Switch access doesn't come preinstalled on Android devices. At least, the ones that I've seen.

So what we have to do is we actually have to go to the Google Play Store, and we have to look up switch access. And then we have to install switch access. Okay? It'll take a moment to do that.

While that's doing that, what we can do is... Okay. Now switch access is open. That'll be for the second part of the demo.

So I'm going to go to... Now, on different Android devices, it's named differently. Some Android devices have it named as accessibility shortcuts. Some of them have it listed as accessibility buttons.

In this case, on this particular device, I have it listed as the option for the shortcuts -- listed as "accessibility button". So let's look for "accessibility...

Accessibility... Button." It has several options. Right now it's set to none.

I can turn on the accessibility button. And the first thing I can do before I turn on the accessibility button is to turn on several accessibility options. So if I go to my actions here, you'll notice that there are several menu options. There's quite a number of different accessibility options on Android devices these days.

And as you can see, we have switch access here, which we just installed. We have talkback, which is a well known option. And we have several other ones. So what I'm gonna do is I'm gonna install talkback.

I'm gonna set talkback on my shortcut menu. And you notice, by default, there's this little icon on the lower right corner that appears. This is on Samsung devices. At least...

This one. And I believe it's on several other devices as well. And then I'm gonna allow switch access.

And finally just for quick demonstration purposes, I'm going to allow magnifier. Here. Okay. So now that I...

If I click this button here, you can see that my shortcuts appear. Now... That's nice and all. But let's say I want my shortcuts always available and visible on the screen to the right.

What I can do is, instead of having the option over on the lower right, in that small icon, I can actually do floating over other apps. And then I'll have a quick accessibility shortcut menu here. With all my options. And if I had more options, then they would all be listed here.

Remember, I chose three. And again, now, I can choose the size of this. I can choose it to be very small. I can choose it to be very large.

In this case, I'll choose it to be small.

I can have it fade in when not in use. If there's an accessibility option that's not being used, this icon tray will slide out and slide in. If I always want it present, then what I can do is I can actually set the transparency, so that I can see anything that it's hiding. In this case, I'll make it completely solid.

So it's seen that it's over other apps. So now if I leave my accessibility button settings...

One I can do... I can go back to my home screen. And now I can just quickly activate all of them from here. So if I turn on magnifier...

What I can do is... I can magnify things on my camera.

This helps me zoom in on objects.

And I can turn on my flashlight and do several other things. So that was just a quick demo of -- and I can turn off the magnifier, I believe, here. I can go back. I can turn on talkback from here.

Talkback on. Talkback on Galaxy Watch 6 Manager is off. To use talkback on your watch, tap the talkback button. I can turn it on, and I can click it again.

Talkback off. And quickly turn talkback off. So that was a quick example of accessibility shortcuts. And I hope that was useful.

Thank you. Great. So that's basically accessibility on Android. The shortcuts on Android.

The same thing applies to iOS. Okay. So right now, I'm gonna show you the similar accessibility shortcut options on iOS. What I have here is my iPad.

And I have an older iPad. Slightly older. But I believe the accessibility shortcuts are the same for the later versions of iPad and iPhone, but you can double check. So to enable the shortcuts, what I'm gonna do is I'm gonna go to the settings icon.

Here. And I'm gonna go to accessibility. And as you can see, there's an accessibility shortcut option. So when I click on that, you can see that there's text on the top that says: Configure.

And then triple click the home button to toggle accessibility features off and on. So... Here is my accessibility feature, to the right. And none of them are turned on.

For this demo. So... If I triple click, as you can see, nothing happens. It goes back to the original...

The home screen. So instead of doing that, what I can do is... I'll click on several of these options. Invert, VoiceOver, zoom, switch control.

And there's many more I can try. But for now... Let me try triple clicking again. And as you can see, now accessibility shortcuts appear.

So I'm gonna, for example, turn classic invert on. And as you can see, it just easily works from anywhere. I'm gonna go to my home screen and turn classic invert off. Okay.

Great. And so... As you can see, the accessibility options are available real quickly. Thank you.

Okay. So that was just on iOS. The goal is for me, by the way, to do both iOS and Android demos. Because there are differences in the way they interact.

So I just wanted to clarify and make it, again, more accessible to more people with more devices. So... So now the next thing I wanted to... Many people here...

Let's see. Did this turn... Okay. So...

Right now, I'm... I'm trying to go back and forward. So... I'm gonna skip this for a second.

And I'm gonna go... Many people are familiar with talkback here. Right? If I'm not mistaken?

Most people are familiar with that? So I'm gonna go actually to switch access right off the bat. Camera switch access specifically. So...

What's interesting about camera switch access is that the actual device can be configured to read your facial expressions. So that you can use your facial expressions to interact with different options on your device. And that includes things like... For example, clicking on an app, interacting with an app, playing a game, going to your settings, going to your navigation options, and what-not.

And it's probably one of the most configurable options on the device. And the great thing is: It comes with almost every device these days. Most devices' cameras are set up to be able to recognize facial gestures. So you can assign specific actions to particular interactions.

So that you can do things in a very customized way. I'll show you in the demo. Because I think the demo will do best to explain what I mean. But like...

It's again: All configurable to the accessibility settings. So it might look a little involved. But it's... And there's a lot of options.

But once you get used to setting up the switches, the camera switch, in the way that you want, it's actually pretty straightforward to really get the flavor of the switch that is most comfortable to your interaction pattern with the application. And so... Again, this is really important, something like this. Because a number of people can only interact with the device through...

They can't interact through their hands. They have motor disabilities. And they have other disabilities, which may -- vocal disabilities -- which may prevent them from vocalizing their way through an application's interface or the device interface. So something like being able to read facial gestures is very tantamount -- a tantamount option to many people who can't interact with the device in that manner.

And I think it's probably one of the best features of the actual device, because again, cameras are accessible on every device. And this is one of the good users of cameras. So...

Okay. Now what we're gonna do... Just a little precursor. I'm setting this up on Android first.

But you'll see... I'll do the setup on Android and show you the differences on iOS. So... So...

I'm going to focus on the camera switch today. Because that's the most interesting element of this particular presentation. Though there are many more interesting switches as well. But the camera switch is available on any device that supports it.

And that has a camera. And virtually all devices have a camera these days. So... If you can't obtain a hardware switch at any point, you can attempt to use the facial recognition of the camera switch to attempt to navigate around the device.

So I'm gonna click on switch access. I'm in my settings menu. I'm not gonna use it just yet. I'm just gonna go to [audio drop].

Okay? You already have your shortcut menus. When we're ready to use it, we'll click on it. So if we go to settings, there's many, many options here.

So... What I'll do is I'll focus on the prime options for the settings. So first I'll focus on the camera switch settings. Okay?

So it says... To control your device with camera switches, first you need to turn on switch access. I'll do that in a moment. But before I do that, I'll demonstrate to you that there's these facial gestures.

I can open my mouth. I can smile. I can raise my eyebrows. I can look left.

I can look right. I can look up. Okay? And I can configure this in any way that I want.

But see, here's a preview. And...

If I look left...

See, it's detected that I've looked left once. I've looked left again. And so... If I go back, and I try the open mouth gesture -- I'm gonna use the main gestures here for navigating -- it's detected that I've opened my mouth.

And so on. So now, in order to configure your mouth gesture more specifically, there are some options here. For example... If I want to make the gesture size smaller, if I make it smaller, you can see it can detect every time I'm talking as an open mouth gesture.

You don't necessarily want to do that.

Because it'll be harder to do this demo and you might not actually want to indicate that you want to interact with an item when you're just talking. So... What I can do... I will make it very big.

And so it will require that my mouth open wide -- much wider than just talking -- in order to allow my mouth to be detected as an action. Okay? And then the gesture duration is important. Because if I want the gesture to last long enough, in order to be tracked as a gesture, in case I speak rapidly, and want to indicate that a gesture is associated with a very deliberate mouth motion, I will go and make it one second.

Okay. And I can even make it longer. I can make it two seconds.

Okay. So this helps you do a little more... So I'm gonna leave it at 1 second for now. So I can have more control of my mouth motion for the demo.

I'm gonna use the mouth motion. I'm gonna use smile.

I think that's a real smile. Smile is another action. And again, smile has a size associated with it.

So if I make it very big... I also have big teeth. It's probably detecting everything. But it's...

Yeah. I'm gonna leave it... So now, again, it can be a little more sensitive or a little less sensitive. But anyway...

So I've configured a smile gesture. And I want this gesture to last for one second also.

Before I count it as a gesture. Okay? Great. So...

I'm gonna go back. And I can do the same with "raise eyebrows", and look up, and look to the right. So what I'm gonna do is now I'm gonna assign switches to different things.

There's a pause camera switch, just in case I want switch access to stop for a while. I'm not gonna assign anything to the pause camera switch. I'm going to use the select option as a smile. I'm gonna use open mouth to navigate around the screen.

And I'm going to... Yeah. That's what I'm gonna do. So let's see if I can work on something with that.

So... Let's go back and activate our switch access.

Okay. I'm gonna go to my camera switch. I'm going to go back to the home screen. I'm gonna turn it back on.

And now switch access is on. And how do I know it's on? Because the little icon on the very top middle of the screen is now visible and indicating that my face is now being detected.

So basically you can check my mouth gestures and you can see as I'm using the gestures, I'm navigating around the screen with my mouth gestures. Yeah. And so... I used the open mouth gesture...

So you can see the blue bars on the right indicate how long it is before the gesture is recognized, and that's the length from start to end, before the gesture is registered. If I don't finish the gesture registration, it won't go to the next item. So I'm gonna help it along. I want to go to the DuckDuckGo app.

It takes a while to get used to this. Power users are very good, being able to navigate very fast with the camera switches. But I actually deliberately went slow, to kind of demonstrate. So as you can see, I selected the DuckDuckGo app with my smile.

So now I used my smile... So now I opened the keyboard with my smile, as I move to the keyboard with my open mouth. And then I go to the next options. On the actual keyboard.

Now, the keyboard is a little tricky, because there's a lot going on, on the keyboard. And if you're trying to navigate, there's many groups and subgroups of things. So... It takes a while to be able to navigate to all the subgroups.

If you're not used to how much is going on a keyboard, you might... I'm gonna help it out a little. So now I'm trying to actually type with my mouse. Gestures.

Which actually is kind of in my opinion... Kind of fascinating, at first. And I'm sure there's an optimal pattern that people can get used to, if you use the camera switch to do this. But you have to select rows.

And then once you select a row, you have to select characters on a row. There's a lot of interaction that's required. Gonna go back to the row. Select it.

And so I don't know what I'm typing. I'm just typing something to get it out there.

So I went to a different option by smiling. And so, as you can see, I can navigate around. And if I want to change my gesture, to be a little more reactive and faster, I can go back to switch access. I'm turning it off again.

I'm going to the settings.

Switch access. Switch access settings. Not that switch. Switch access.

You're gonna have to play around with it. And then if I go to the settings again, now I'm going to go back to my camera switch settings. Going to my open mouth motion. And I'm gonna make sure the gesture duration is only 25.

And... I'm going to then use "raise eyebrows" as well. For previous. Okay?

So this should make it easier. And I'm going to make "smile" -- I'm gonna leave that -- I'm gonna leave smile big, and I'm gonna leave it at 1 second. So let's try this. Let me try playing a game with switch access.

All right. So... I'm gonna open the Games app.

Switch access is on.

So this is what I work on. And so we have... All of our devices and all of our applications... We try as hard as possible to make them completely accessible.

So it should be easily adoptable to switch access. And the gestures that you're choosing to navigate around. So...

One thing that's also a matter of getting used to is to make sure that you're not getting too wrapped up in... You're figuring out how to avoid the icons on the left, when you're trying to select things. Because those are your accessibility icons and your options. So that in itself is a little bit of...

One of the kind of tricks about getting this right. Because those are there to help you. They can also impede your ability to navigate using your facial gestures through the screen. So there are ways around that as well.

But they require some more advanced configuration. Ideally you have somebody who is experienced with this, to help you with the navigation, at first. But there's many good tutorial videos out there. I'm gonna close this.

So this is me attempting to play the crosswords. I don't know how all this went, to be honest. But I did my best in the short time that I had to record this. But I was able to use my facial gestures to get to the crosswords screen.

So with that smile, I was able to open the game.

Now I'm going to...

Also, as far as what letters I'm selecting -- also, again, the menu options -- it's rather slow, even going through the menu options. But that's a matter of configuration. So the idea is to get to the squares. And at some point, I get to the squares here.

And I'm gonna see if I can... Like... Speed this along a little. Because eventually...

I'm almost there. Then I'm going to select.

And so essentially I just revealed a word. Because it's gonna take a while to actually interact with the word. But I'm doing this, again -- the point is to do this all through your facial gestures.

And so... There. There's the word. All right.

That was just an example of switch access helping me out through the camera in playing this game. Thank you. Cool. So that's camera switch access on Android.

I could do the same thing on iOS. With iOS, you also have the same ability to do the facial recognition and what-not. But the way that I had my iPad set up was that it was actually only tracking my head motions. So I actually had to move to the left and the right to do the same thing.

I'm gonna just quickly go through that. Some of the pieces there. But essentially... It's the same thing.

Sound. I can have the selection as an entire word. Like... Select.

Or something like that. But this is just... To help me out here, what I did was that I used... I used the select gesture to make a sound.

As the thing that actually selects something on the screen. So I'm using the camera, plus I'm using a vocal switch as well. To do multiple things simultaneously. That's one of the really great things about switch access.

It's essentially multimodal. So you're not relying on -- you don't have to rely only on camera gestures. You can rely on vocal gestures, visual gestures, the camera gestures, or even your manual interaction as well.

For demonstration purposes... I can mix switches together, and controls. So let me turn this on. Switch control.

And so... There's also a switch control playground as well. If I want to practice. But for now, I'll just demonstrate.

So as you can see, this left a menu section for the options in the settings selected. So if I go to the right...

I do want to emphasize the playground portion. Because as you want to get better at this, you can use the playground, which is this test bed, to really... Left access. Get better with your gestures.

And basically it's kind of like... An automated system to help you refine your interaction. So now you can see what I'm doing. I'm using my head motions to navigate.

One thing... Oh, I see the problem. Yeah.

So my camera was this way. So only one gesture was actually working. But it's okay. If I go...

So I'm using the gestures to interact with items. And then if I go...

Back I'm gonna go backwards. So now I'm gonna try to select the icons. So... I selected that box down there.

Shhh. It looks a little odd, honestly. But...

Shhh. These things do help. So I'm back on the crossword screen. So...

Let's see. Okay. Now it's picking up my actual words. As I talk to them.

It's actually filling in -- So let me exit out of here.

So enough navigating with your neck can actually cause a neck cramp. I'm just letting you know. Shhh.

Okay. Now I'm in a game session. I moved to games on the selection. So if I go downward...

Shhh.

So if any of you are a fan of Connections... I'm gonna attempt to play connections... With my head and my mouth. Let's try it anyway.

It keeps going up. If basic facial gestures aren't working, these are the defaults that always work. Let's click on play. We'll get used to it.

[inaudible] It does, but for some reason I couldn't for the life of me get it working. There was an option to configure it. It wasn't working with this iPad. I don't know if it was a software thing or what-not.

But yeah. I'm sure it does. Like... The options are there.

But as you can see... I select the board. Yeah. And then I have to select further items within the board.

So it does take some time to get used to, but if I'm good at it, I can do this pretty fast. I'll go to the next -- Some people are super experts at this, and they can play this as fast as anybody who is using their motor skills. And so it takes a while. Yeah.

So all these times can be reduced. Part of this was being deliberate. I'll put accept. Shh.

Sorry. Can select things using my voice. And so on and so forth. So you get the idea.

Again, like any skill, it just takes some time to get used to. And if you get the gestures working, you can... And so on and so forth. Shh.

So... Cool. So those are camera switches. And they're, again, very...

I think they're the most accessible switches, because they're available to everybody. Has anybody ever used a hardware switch before? Okay. Great.

So this is a hardware switch. This is by a company called AbleNet. And the hardware switches are for people who... They have limited motor ability, when it comes to interacting with a device.

But they are still able to use at least a device that has very prominent interaction capabilities along with it. So... There's many different kinds of hardware switches. They're mostly usable on Android and iOS.

This is a dual switch. There's something called a jelly bean switch, jelly beaner switches -- those are individual switches. This has two options on it. These switches are a little bit pricey.

But they do make interaction with the device pretty easy. Also, if you want to... I don't know why it keeps going on and off. But...

You can... There's also switches that you can use your mouth to... Put like... Air pressure into.

And that will allow you to interact with the device and move around. And so on and so forth. There's a wide variety. I've seen people connect video game controllers also and use them as switches.

And you can configure switches -- again, these hardware switches are configurable using the same procedure that you use to configure the camera switches as well. So that's what's really great about the API for these. Is that it's very interchangeable. So if you need to mix and match different camera actions with hardware switch actions with vocal actions, the sky is the limit at this point.

So what I'm gonna do is show you a demo of the hardware switch in action. Now I'm gonna show you a truly awesome demo. I think it's awesome. That involves dual screen mirroring here.

I have an Able Net switch here, which is a hardware switch. And in my Android device, I'm using it to take a video of the switch, as you see. Using the camera to demonstrate interaction with the switch. Okay.

And I'm using the iPad here to demonstrate the effect of the switch. I could also use one here as well. But I felt it was nice to be able to have a clear view of what's going on, on the iPad. All right.

So... Let's get to it. First, in order to configure the switches, I go to settings. Okay.

And I go to switch control. Well, firstly, firstly, before I do any of that, I need to go to Bluetooth. Now, this is already connected. I don't know why...

I don't know what happened there. Let me go back.

Yeah. Let me try this again.

So you have to pair this hardware switch to the iPad. Once you've paired the two devices, then you can actually activate and discover the device.

So we've gone through that. I've connected it already. This is already set up. There's no need to do any of that anymore.

But... If you want, you can forget the device and then you can re-pair it. So now it's paired. Right?

And that will take a little while. But it's very easy to pair. Once you pair... I'm gonna not do full screen.

Because... Yeah. I think full screen is causing an issue here. You have to make sure this device is on, too.

The power is on as well. Then you can go to switch control.

And so, in switch control, if you saw my camera switch video, you saw that in order to add switches, you have to go to the switches option here. And I have my camera switches here. So let me just remove those for now. Because I don't want to confuse anything.

And... I have my two physical switches here. So these are switches I added. So basically, whenever I add a new switch, I want to add a switch, I click on the add new switch.

In this case, I'll click add external switch. And what I will do is I'll press an option. Right? And so here is the...

These two switches are being used right now. So what I'll do is I'll forget these switches. I'm going to delete them. I'm getting rid of them.

And what I will do is... I will... Now add a new switch. So first I'll detect a switch.

I'll call it...

I'll call it 1. So let me go to the end, actually, where I'm using the switch. Looks like we're stable now... Ish.

So I'm using the hardware switch to play Strands, which is our newest game. Now I'm gonna select the board. So the entire board group gets selected.

And now I can select rows. It's almost like a video game controller. So, again, something that you get used to. It takes a little while.

But I'm going through each of the rows. I'm selecting each of the rows as a group, and then selecting each of the letters. I'll select the next letter. The point is that I've configured the switches to move left and select.

But I can also combine these with additional switches to make it more dynamic. Let me try T. Now let me go...

Yeah. I guess we can do that as well. I think... I'm doing most of the talking here anyway.

So it should be fine. So...

Again, the whole thing is the interaction with the device is just me. I'm not getting the game that well right now. But the whole point is that... This is just one example of a switch, a hardware switch.

I can use a one-button switch too, or could use multiple one-button switches, and get the same effect. However you want to do it. But this AbleNet switch is good. AbleNet has been around for a while.

And it specializes in interaction. And application to sites, mobile devices, anything that has accessibility protocols already assigned to it. So our app already has content descriptions and other things that make it easy to navigate. And therefore it makes the switch able to navigate through the screen very easily.

So anyone who is doing game design, game building, anything -- should also keep that in mind. That they should consider their accessibility labels and other things on the device. And on the game. To be set up properly for these -- for both talkback and for switch access.

Okay. So... I'm gonna get off. I have to go to YouTube, to see if that would do any better with the...

Okay. So the last thing that I want to show is... I guess there's a few more options here. Is anyone familiar with Live Captioning here?

Has anyone used that before? So what's good about Live Captioning is that for instance... The audio is muted on there. Do you want it up?

Yeah. In this case putting the audio up might be good. So basically what I did is... I put a demo of Live Captioning.

Live Captions is a technology which reads captions from any source of audio on your device. So it's not necessarily tied to the video, your video option. You can select captions on your video option. And it'll show captions.

But if you're getting sound sources from other places on your device, for example, audiobooks, or some kind of background music or something like that, Live Caption will just put the captions right on the screen, anywhere. To me, it's even... As a non-accessibility concern, it's actually very useful, for example, if you're on a train, or you're somewhere where you need to actually see captioning from an audio source, and it's not easy to access the captions and see them in the audio player that's presented.

So we'll see if this demo goes... I'm going to show you a quick simple demo on how to have the overlay captions available for accessibility. So those are basically... Let's say that every video that you might watch or piece of media might have captions.

But the overlay captions directly translate any source, even if it doesn't have built-in captions, into an external box, where you can see what is being output in text to the screen. So it's an accessibility feature. And I'll show you. So if I go to this video.

For example, here... And I can make this turn on, by pressing the volume button down. So I'm pressing the volume button all the way down. And what I can do...

If you see this little button here -- sorry. This little chat box-like button -- so basically what's gonna happen is that we're going to have Live Caption turn on. And again, it doesn't matter what video I have. Some videos naturally don't have captions.

It doesn't matter. If I open this video, it may or may not have captions. But what's going to happen is that the video itself, if it's outputting sound, it will... You will hear via Live Caption, or you'll see via Live Caption the output text from that sound.

And that's great for not only YouTube -- it's great for anything that outputs sound. If I have an audiobook that does the same, I would have the same functionality there. So that's it. And...

Yeah. Thank you. So again, I could minimize that screen. And the Live Captions would be on my home screen, and I can move them around.

It's just very useful for so many reasons, because there's many times where the captions are embedded in your video player. Or embedded in your video player application. And you don't necessarily want to have that application always open. So again...

Any audio source, it'll -- Live Captions are directly connected to your audio source. So that's what makes them really cool. And then the last application that I want to show... I do have some notes on the accessibility scanner, which is a tool which is, again, a color...

It's the equivalent of a color contrast checker, but it's more advanced, in the sense that what it does -- it checks a lot of different accessibility features for a device. I'm just checking on my time. I still have a little bit. Right?

Okay. Okay. Cool, cool. Okay.

Cool. All right. I'll go two or three more minutes. Okay.

Cool. So also these will be posted, too. So you can actually check out these demos. But the accessibility scanner is a tool which...

If you have a mobile application or anything, you can download the accessibility scanner, and you can click on it. There's an option to. And it will allow you to check everything accessible on the device. And whether it has accessibility violations or what-not.

So again, very useful for mobile applications. You're also able to correct accessibility violations, using the accessibility scanner, as well. So I'll save these demos for later. And again, you can check these demos online.

But I created an application here that had several accessibility violations. I put the accessibility scanner on. And I was able to be like... Oh, okay.

Well, the color contrast is wrong here. So let me actually live code the change here for the color contrast. And the accessibility scanner will be like... Okay.

I'll change it, and then the violation is fixed. So that's really -- it's a great tool. And you should be using it pretty often. And okay.

So I think... What I'll probably do... I'm just gonna end up with this last video, and then we'll go to... So has anybody heard of Be My Eyes before?

I just wanted to dovetail into AI. Because that's the world that we live in today. But... This is just -- the Be My Eyes application.

I'm about to take a journey that may change lives around the world.

Vision is such a dominant sense for most of us. Our eyes feed us an endless stream of information. But it is somewhat different for me. I'm blind, so I've had to learn new skills and use workarounds to live my life without sight.

I still think visually, but I have to build the images in my mind through other means. Today, though, I'm experiencing something new. I'm using OpenAI's newest model and voice first interface to imagine the world through a conversation with an AI.

Sunglasses. They definitely add a cool factor.

Oh, thank you. What is in front of me at the moment?

You're looking at a beautiful sunny day by the river Thames in London. Right in front of you is the Tower Bridge, one of the most famous bridges in the world. With its majestic twin towers and blue suspension cables.

That's perfect. Can you tell me if the bridge is up or down at the moment?

The bridge is down at the moment.

As the AI is speaking, my mind is building a visual image. I'm now getting a real appreciation of the majesty and scale of some of London's sights as they're described to me.

Tell me what you can see.

I can see the top of Big Ben peeking through the trees. It's got that classic neo-Gothic design with the recognizable clock faces on each of its four sides. It's such an icon landmark, isn't it?

Is the king is residence at the moment?

Yes. It looks like the king is in residence right now. The royal standard flag is flying above Buckingham Palace, which is the signal that the monarch is present. How exciting to be there while the king is home.

Try and tell me exactly what they're doing right now, please. I'm gonna pause that. Because you can get the idea that it can read everything from the environment, the current version that I have on the iPad, that if you want to try it out... It doesn't do the voice output, but it does the actual reading of the text and the descriptions, and it does it very well.

And then you can do what's called visual Q and A. VQA. To the actual output of the device. So you can ask more questions.

About what's in the environment and what-not. So... That's pretty much where I'm going to end. I do have some statistics that you can take a look at.

There's a lot of interesting things about accessibility switches, mobile devices, and other things that are very useful, in terms of how many people are using mobile now. And how these are becoming more and more relevant in everyone's everyday life. But yeah. I'll leave it at that.

And the slides will be available later. So... For now, I'm just gonna leave it at the Q and A. So...

If there are any questions. Thank you, Shafik. Thank you so much.

Great to see so many demonstrations today. And especially on switch technology. We're going to be taking questions both from our YouTube audience and here in the room. So first I'll just start off...

Are there any questions here in the room? All right. You're first. If you'll speak into the mic...

Hi. My name is Camille. I was just wondering... Do you happen to know if there are people at the Google Store or the Apple Store, who are trained in helping people navigate the different things that you showed the videos on?

My suspicion is no, actually. And I will say that, because in order to ask about accessibility devices, so I could do some research and perhaps get one, I had to call the Apple Store. I went to different stores. Hardware stores.

I went to Best Buy. Ask many people didn't actually know what these devices were. So the staff knew about phones. They knew about anything that catered to the audience that wasn't necessarily looking for accessibility enhancements on devices.

And so many people don't even know about these. And so... That was part of my motivation. I was like...

How do you not know? There's a large audience that's very... That needs these things. That needs these devices.

That needs somebody to help train them to be able to... There's many options. And they need somebody to help train them to be able to configure their devices and use these options, so they can effectively use their devices. So I haven't asked everybody at these stores.

But based on my limited sample size, I believe that's not a training that's there. And that's a training that should be there. For sure. I would echo that too.

I think it's part of our advocacy as a community. It's kind of great to go in there and... Put that to the people. If they don't know, I think in a best practice...

They'll probably want to learn. So it's great to, like, seed that question into the environment too. Let's take a question now from YouTube. So the first question: Any recommendations or best practices for how the code is structured to allow for switch use?

You know.

Separate from keyboard tapping. For sure. That's a great question. I actually didn't have time to get to it in the demo today.

But in the slides, I did have an entire setup of a button to be accessible. And so part of that is... On the Android side, and also on iOS, I said you had these ideas of content descriptions and accessibility labels. Some of this is outlined in the WCAG guidelines.

But also, the providers, like, for example, Google is very focused on accessibility for Android now. They have entire video series on how to set up your buttons to be accessible. And it's actually... It's actually not...

It takes a little bit of practice. Because you need to... Again, the guidelines are available. But you need to make sure that the accessibility...

That you're applying to a particular button... Is really relevant to the use case and the flow of the user interacting with your application. For example, if you have an image, and I'm gonna use the word "content description" for now. Or...

"accessibility label". If you have an image that's not interactable, then it doesn't necessarily need an accessibility label. But if you have something that's interactable, and the video I have in here was actually a great description -- by default, the label on that item is generic. Right?

For example, if I have a text box that I want to enter something into, the text box is usually just by default labeled "text box". So when the screen reader goes over to "text box", it will say "text box". That doesn't make sense. For example, what if it's a login screen.

I want to know if I'm entering my username or password. So you have to be very contextual, when designing your application, to make sure that the item that the user selects is very descriptive of what the actual intent of that item is. For example, the text box should actually be: This is the username field. Enter so and so.

This is the password field. Enter so and so. So again, strive to be as descriptive as possible. About the interaction.

As you would use it. And as a developer, who is also putting themselves in the shoes of the user. If that makes sense. But again, there's a lot of guidelines now on Google's forums, on Apple's forums, on how to do accessibility well for mobile devices.

So... And those links are shareable, too. Great. You had a question here?

Hi. My name is Mira. I have a follow-up question to that. So if I don't have access to these switch devices that you just showed, and I want to test my software, can I just...

It's like... A good practice to try it with a screen reader and keyboard navigation? And that should also cover the similar functionality that you just described? Yes.

Screen reader is actually a good baseline, I think. Because screen reader, talkback, VoiceOver -- those are good baselines, because they test your accessibility architecture in a familiar way. Most people are familiar with screen readers. So it will be able to...

Assess your accessibility labels. Because you can use the screen reader to navigate to them. Once you have the screen reader set up, what I found -- but again, I'm actually not an expert in, like, transitioning a screen reader accessible device to a camera switch or a hardware switch accessible device. But I would say that if you are, if you have set your device up for screen reader, then it should actually be pretty straightforward to set up for other switches.

But I think there's a matter of nuance into that. So it's a matter of just like... Trying it out and practicing it. But I think that's a good question and a good baseline.

Yeah.

For Android specifically, if you look at the way that the accessibility APIs are structured, if you test with both screen reader and switch access specifically, you can use just the software switch version. Like, you can set up a tap on the screen, or a... You can set up different gestures to leverage the same behaviors as what you would get with the hardware switch. But if you're gonna kind of think about a test matrix of how to approach your testing, for an Android app, I would at least recommend screen reader and switch access as your first kind of pass.

Because those test slightly different aspects of the accessibility APIs. On iOS, if you test with VoiceOver and voice control, that also gives you a similar kind of representative sample of coverage of the different ways that the accessibility APIs are structured. So we kind of approach our testing for iOS and Android a little bit differently. Because of the ways that the underlying APIs are structured.

Keyboard navigation is gonna give you a slightly different sample set of the ways that things are structured. Happy to talk a little bit more about that in detail. But... Yeah.

Great. We'll take one more question from online and then one more question here in the room and be wrapping up. So I'm gonna add a little to this question as well. This is my question.

But are there certain kinds of considerations more unique to switch access, to keep in mind with developing apps? And I just want to add on to that. Because you have done such extensive work in this area. Do you have a recommendation for the switch app of anything that could be changed?

To make it easier as a developer? So... I guess maybe the second question first sounds... The second part sounds actually more interesting.

I think that by necessity, configuration of the switches is actually fairly complicated. I didn't actually go through all the options there. But I do have a portion where I'm reading out all the options. And there's a lot to consider, when configuring a switch.

For example, how long is the time between switches. Should I use manual scanning or autoscanning? Should I have the gesture autoselect once something is selected? Or should I have just completely...

Completely have the user select everything? I think one thing that would be useful is that if there were some built-in presets already available, that I could select and be like... Okay. This is the most common preset one.

Would you like to try preset one and see how that works for you? If that doesn't work, then try preset two. And if not, then go along with customization. And if there were more on-device customization modals or something like that, that helped navigate...

There are a few. But like... They're sort of, again... The more nuanced configuration is left up to the user.

Where it's a little challenging. So I think, again... Presets would probably be a good initial option. I think that addresses the second part of your question.

The first one, I'm not sure if I got it. The first one is just related to... Anything for developing... It's kind of related.

Any tip for developing specifically for this use case. Or for switch access. I would say that I think the best thing is actually... And this is...

I don't know how satisfactory this is. But... It's actually really to get a variety of switches. And try developing.

Because this is a dual... These are two switches in one device. Many people only have one switch, and one switch per option. And so they, like, have two devices connected.

And they're using it ambidextrously. So I can use this with one hand. But a lot of people use switches with two hands. So I think...

Again, I don't have enough experience to actually translate this to something super actionable. But I would say is... That the baseline is to have your test facility or test lab... Make the...

Put the purchase in. And put the funds towards purchasing several different switches and attempting to have those used against your developed application and see how that works for different users. And then change labels and things based on that. Great.

All right. Who wants to be our last question here in the room? Questions left in the room?

Okay. Here we go. Speak into the mic, please.

From a user perspective, is there any, like, best practices? Because there's a whole bunch of features like loops and all these things, that someone can find a resource on YouTube or something to actually optimize for them. Is there anything out there like that? Best practices to optimize for a particular...

Like... Is it a nuanced feature?

Like a control. There's a ton of controls you can do on here. And everybody is gonna be different, the way they use it. Is there, like, some best practices for someone who has...

You know... Can't function a certain way, so they would like to use their app a specific way, and maybe you have to like... A different like... A couple loops, and then a wait time, and then a couple other things that really works well for a specific app?

Has anyone kind of figured out a bunch of different ways that you can just kind of look at their templates and copy them? I haven't seen like a repository of good templates. I imagine that something probably exists. But it's probably a bit informal.

I'm gonna look, actually, and see. But I guess because there's just so many different manifestations of interaction difficulties, like... It might be hard to... It might be hard to come up with, like, the large...

A large repository of templates. I'll take a look. One thing I would... I think, though, that's kind of helped me a little bit...

And I think this is one of the good uses of, like, something like a perplexity or something like... LLM or something like that... Is to actually work and ask the LLM nuanced questions and do some prompt engineering about the specific setup. That's far from being the solution.

But with some of the more advanced models, you can ask -- well, if a user is enabled or not abled in a particular capacity, and if I have these switches available, these are my options. You can post an image of the switch options, and be like... Can you suggest an optimal pattern? An optimal setup for these switches?

And it may actually come back with different options for the switches. For that particular person. And this is really good with the multimodal models, where you can share an image of the settings. And again, especially with the iPad, there's so many of them.

That it can help narrow... I think it can help narrow it down. I haven't tried that extensively. But from my basic demos, I did a little bit of that.

Like... Can you give me some better configurations. And in my basic testing, it seemed to be okay. But that has yet to be stress tested.

But that's the first thought that comes to mind.

Cool. Thank you. Appreciate it. Thank you all for coming out here in person or joining us on the YouTube stream.

We're very happy to have you all here. We do our event every month. Our next event in August is still TBD. But you will see an announcement on our meetup page very soon.

We actually have been doing pretty good, though, for the rest of the year. And we'll have the lineup... We have most of the events scheduled for the rest of the year. And we will share that out on our meetup page and social media.

We want to say thank you to our event host, AKQA. Design agency here in New York City. We're here in SoHo. We're very gracious to be in this space and to be able to host our events in person.

And we say thank you to Joly MacFie from the Internet Society of New York for always broadcasting our events live on YouTube. So if you're not able to come in person, you can always join us and participate online. Thank you to Mirabai Knight for our amazing human captions that are provided for every event that we do here in New York City. We also want to say thank you to our sponsors Equal Entry, an accessibility consultancy that I own.

Evinced, an automated accessibility solution to identify accessibility issues, and Fable, an accessibility platform powered by people with disabilities. With our sponsors, we're able to do these events and have them be accessible. And we are always looking for feedback. So if you have feedback on how we can improve, or expand our community, and make it more accessible, we're always open to hear that.

Thank you again and thank you to Shafik for the excellent presentation tonight. Thank you.