Designing for Assistive Tech in Adobe XD | Accessibility NYC Meetup Recap

I named this talk designing for accessibility in Adobe XD. You'll have to bear with me for a little bit, because it's not gonna seem like that's what we're talking about, but I promise it is. I'm a painter by training and by education. Professionally, I'm a software engineer.

I'm not a designer. In either aspect. So that might be a surprise to you. But it's okay.

I promise. It'll be good. And educational. Because my background as a painter has informed my practice in accessibility and my practice in voice design and my practices...

Well, I guess, as a designer. Does anyone here... I'm gonna describe this image. It's a pixelated picture of a man in a blue beanie.

Does anyone know who this is? Zelmann. Yes. Jeffrey Zelmann.

He is a pioneer in the web standards space. I read his book in 2002? Something like that. And it really turned me onto the ideas of semantic web, accessibility, and progressive enhancement.

So hold that thought. Also, further down the line, through a variety of careers, and having moved from art school into sort of basic HTML coding, to more recently voice design and voice technologies, I started getting into devices. Like Alexa. And Google Assistant.

And thinking about... How does that fit into visual design? So two things so far. Right?

Progressive enhancement. And voice UI. The final piece is where I've come down. I'm a software engineer on Adobe XD.

I work on the voice features on Adobe XD. Voice and speech playback. And I had kind of an ah-ha moment, after going through... I mean, that was basically, like, 12 years in a two-slide format...

Through that journey, I kind of came to a realization, and it culminated with my work on XD. The realization is... It's related to progressive enhancement. It's related to painting.

And it's related to voice technology. And to illustrate what I'm trying to get at, I want to show you a little video. This is me painting with Adobe Fresco. It's a screen recording of me painting a portrait.

And so... I'm gonna pause for a second. So this is a very classical approach to painting. And what you do classically, like, from a Flemish painter standpoint, in painting a portrait, is you have a ground.

There's a whole production. You could spend days just preparing a canvas. But you have a canvas with gesso, you paint it with a ground, you have, like, a neutral color, and then you start to do a line with charcoal, a light illustration of what you're trying to capture, and a stiff brush really rapid sketch with a dark color, and you start to build up... Not piece by piece, but all at once, you build up this form.

And so this is my background. This is my background. Having this experience in painting has informed where we're gonna go today. And so to describe here, what I just described, doing the lay-in, doing the sketch, starting to put in the darks, the lights, as a general form, starting to put in the background, blocking things in, and building up...

Like, globally in loose slowly more refined aspects. So I did promise this was a talk about Adobe XD. And a talk about design and a talk about accessibility. So let's see how all of what I just showed you fits together in a live demo.

So what I want to try to do here is show you a process, a framework for thinking about designing for accessibility. Accessibility here means to me... Maybe more inclusive design than compliance. It means making content available in a multimodal experience to a broad range of sensory experiences, and the way -- the framework I've kind of come to think about that is progressive enhancement, and that layering technique.

So starting with a blank canvas, we want to kind of just map out a flow. So we're gonna start in prototype mode. I'm gonna do like a little arcade game. Right?

Like a dungeon crawler. And see how the principles that I just described can be used in XD to build a multimodal experience. So the first thing that we're thinking about is multimodal and sensory-based... Driving this out from a sensory-based experience.

From an accessibility standpoint, most design starts from visuals. From an accessibility standpoint, that doesn't always add up, because I would say at a baseline, and I'm gonna elaborate on this more later, but at a baseline, really it's about text. Text either as auditory text, so read aloud, text as a command or an action, text as captions, you need the seasoning of visuals to make that come alive. But at a baseline, we're talking about text.

And naturally that flows into voice. So if I wanted to design a dungeon crawler game, naturally I'd want to start from a landing page. Like, sort of a... You open the app.

And thinking about this from a screen reader user's standpoint, say if I have very low or no vision, what would the experience be like? I would be using a screen reader. So we need some sort of speech playback. So I'm gonna add an interaction.

And can you all see that? Moderately well? I'll describe what's on the screen. Can I...

I'm gonna actually change this screen resolution real quick. A little bit better, right? A tiny bit? I'll describe what's on the screen as well.

Because that would be in good taste and good practice. I'm working in Adobe XD right now, and I'm in prototype mode. I've got a few canvases up. I just duplicated a handful of canvases, and in prototype mode, you can attach interactions to objects.

And those interactions have a trigger, like tap, voice, time, drag, and the actions are like outputs of that trigger. So more recently, we have speech playback. There's also autoanimate, transition, et cetera. So what I've got here is my first art board...

And I'm attaching an interaction that's time-based. And I'm gonna have it have speech playback output. I'm just gonna go through each of these canvases, and add what I want to hear as a screen reader user. Kind of like I was describing before.

Like, doing the lay-in. Right? That's what we're doing right now. The first one I put the title of the game.

Dungeon Crawler. The second one I'm kind of imagining... Okay. I would probably go to a character chooser screen.

Okay. So choose your character. I can kind of start to imagine this coming together as a screen reader user. Next you probably want -- so you've got your character.

You probably want some sort of exposition or description of the landscape. Okay. So we're starting to get more of an arc. I've written: you find yourself in a dark room.

There's something else here. Sort of a prompt. To... Do something.

So we'll leave that. Because we don't really know what that prompt is gonna be yet. Now, if I just play this right now...

Dungeon crawler.

CAMERON: Okay. There's nowhere to go from here. I need to link these together. Right?

This is where you start to think more creatively. Visually, yeah, you would have a button. But what does that mean for someone who doesn't use a keyboard or mouse? Well, you've got voice commands.

Start. Druid. And search. Okay.

So I've written in kind of the general buttons, if you want to call them that, or commands that I want to apply to get the result, which is what I just sketched out. And I'm gonna wire these up to have a voice command that corresponds to the visual action on the page. And so what this means from an accessibility standpoint is you're starting to think about... You're starting to drive the visual experience from the experience of someone using assistive technology.

In this case, you can think about voice control, or speech recognition dictation. Speech.

Dungeon Crawler.

CAMERON: It says Dungeon Crawler. Oops. I'll try that again.

Dungeon Crawler.

CAMERON: Start.

Choose your character.

CAMERON: Druid.

You find yourself in a dark room. There's something else here.

CAMERON: Search. So I haven't put anything on the last page. But this is starting to come together as like... If I didn't show you any of this, it starts to come together as a coherent experience for someone using, for example, an Echo Dot, Alexa device, or someone using a screen reader or voice control.

This is not sufficient for a visual display. I mean, it works. And that's something. But...

The idea here is... Well, let's dig into that. Why doesn't it work? For one, these need to be clickable.

If I can't vocalize, if I don't communicate verbally, voice control is kind of out the window. I'm probably using a visual UI. And according to our handy-dandy authoring guidelines, there are very clear specifications about how a button should kind of work. Right?

And one of those things is that buttons should all kind of look the same and they should kind of communicate their affordance. We've all kind of had a picture in our head what a button looks like. I think... I'll illustrate accordingly.

Rounded corners.

JOLY: You could move the stool so it's in front of you.

CAMERON: I didn't want to pull out any cords. Yeah, that would pull out cords. I'm definitely gonna pull out cords. Okay.

And then... For placement, we never got to the last step of... What did we find? Let's see.

I'm gonna put a time trigger. So that basically means that when we land on this art board, it'll immediately trigger the action. All right.

Dungeon Crawler.

CAMERON: So the same kind of interaction works now, with a tap. See? I've got a tap action. Transition.

Tap action. Tap action. So suddenly we've got...

Dungeon Crawler. Choose your character.

CAMERON: Druid.

You find yourself in a dark room. There's something else here. You found a staff.

CAMERON: So I think you're starting to see how this could progress into a more sophisticated UI. A couple of caveats. This is exceedingly difficult to do live in front of an audience. And also, visual design is not my strength.

So I'm not gonna belabor the point. But I did want to show you this process of building up. Right? Building up from a core experience, which is basically the experience of someone who has some sort of disability, and generally speaking I start with voice and text.

Specifically text. Because that has the most outputs. And build up from there. Oh, so let's talk about color.

Again, I've got a couple of swatches. That, if you don't know about this website, Adobe.com... Dot io.

Spelling of color!

CAMERON: Oh, thanks very much. This is Adobe's color swatch creator. You can export color schemes from here to your Creative Cloud libraries, and use them in XD. So I wanted to touch on this, because it's a very common go-to for designers, is to think about...

I think often in the same way that developers think about alt text as the first thing, when someone says accessibility, they're like... Oh, alt text! Designers think color contrast. And that's important.

It's kind of like multiple steps down the path that I was just going. But I wanted to call it out. Creating a swatch, like a series of assets, and then vetting them against a plugin like Stark, so, for example, I have this button, I'm gonna make it... Purple.

I'm gonna make the text kind of brown. I'm suspicious this will not meet color contrast guidelines. And... There's a hot key, which I think is...

Command-shift-H? Yeah. So that just opens the action in Stark. It's a deep link.

But they have a couple of handy pointers here. Which -- these map to WCAG guidelines. WCAG AA. Requirements around contrast.

So normal text for AA, standard, 4.5:1. So what I would suggest is... You know, right now we're at 1.26. Use this plugin to sort of -- once you've laid out your general palette, apply it to your design, and use that plugin to then evaluate your contrast and adjust your palette accordingly.

So if you use it in the assets panel, using components, you could say... Okay. I want this brown... I'm gonna add this brown to my colors.

It's really hard to see. It's over here. That's just how bad the contrast is. Add to colors.

So using this color as an asset, you can go back and say like... Oh, this is clearly too light against that green. I'm gonna change it globally. Like a variable you would do in programming.

So that's my recommendation for strategies around color and color contrast. So I'm gonna pause there for a second. The other approach or I guess thinking you have to consider is developer handoff. So this is like as a designer -- you're sort of building from scratch.

You're doing a green field design, and you want to build foundationally on accessibility and inclusive design as a principle. What happens when you hand it off to a developer? There's a couple of ways that you can go about doing that. One is...

Using annotations. I've shared this asset out in the group. What we're looking at is a group of identifiers, annotations, little boxes, that have modifier keys, tab order, letters, so that you would use in a screen reader context, assistive technology output, and regions. Also, there's tab order direction arrows.

And you would actually pull up your design and annotate your visuals accordingly. So, for example, I'm gonna show you. If I wanted to show that command-F5 is opening VoiceOver, then I could actually use those annotations adjacent to the design. Command.

This is a bad example. This is assuming this is already open. If I wanted to go right, I could use tab and the tab order. And what that looks like in a finished design...

Is something like this. This is a little bit of a variation on what I just showed you. It uses the spirit of those annotations to simulate a screen reader experience.

VoiceOver tutorial. VoiceOver is the built-in screen reader on MacOS.

CAMERON: This is all designed with native features in Adobe XD, including voice commands, speech playback, and key commands. I've applied a keys and game pad trigger, and you can specify a key. In this case, the key that VoiceOver supports for the rotor is... Control-option-U.

That's right, right?

Welcome to MacOS. VoiceOver on. Headings menu.

CAMERON: All right! Headings.

Press the number 2 to filter the headings list.

Level 2 headings.

Use the down arrow to focus the next menu item.

Billing info. Credit card info. Billing info.

Use the down arrow to focus the next menu item.

CAMERON: So what we're doing here is actually more of an experience... Like, an illustration, or a mockup for developers to understand how the assistive tech should behave with a design. So, like, the designer is not actually building those modals. That's part of the assistive tech.

And I'll show you... If I hit command-F5, it'll turn on VoiceOver. And I get that experience. So by...

VoiceOver on... Full screen.

CAMERON: Command-option-U. Control-option-U.

Response menu. VoiceOver off.

CAMERON: So illustrating how the assistive tech should behave will give you a point of conversation with developers on how to execute on your design. It's, I think, more of an illustrative exercise. It's -- annotations are sort of like core -- just tell me what I need to do. Versus an assistive tech illustration of: This is kind of how the experience should work.

So when you build with annotations, that would be for things like accessible name, role, properties. If you want to show how someone should use it, you can use this template. And all the components are here. In that Dropbox download.

So like the cursor. Toggle VoiceOver, et cetera, et cetera, et cetera. And I'll post some more assets to that Dropbox directory after this. I think I'm missing one or two of them.

One other thing in the spirit of developer handoff and annotations. You might have caught a glimpse of it. There's a plugin... That we use to define how components should behave.

This is sort of like a plugin that encapsulates a design system. And... Let's see if I can articulate this. The design system dictates what a button should look like.

How it should behave. And ideally its accessible name, properties, behavior, role. And we're starting to do that here. I have a way old version of this plugin.

We're starting to do this with our internal plugins by adding an accessible name field. Which should generally correspond to the actual title of a button. But if you have something that's just like a little edit icon, like a pencil icon that's a button, you'll need to specify some sort of alternative textual content. Right?

We know this. It's like the alt text I talked about earlier. And having a plugin-based system, which sadly I can't show you right now, but having a plugin-based system where you can annotate those elements, is powerful. And that's kind of the direction that we're thinking about.

With that work. So... A couple other things. I wanted to touch on deaf and hard of hearing.

This is... I hinted at this before when I was doing the sort of lay-in exercise of... Okay. Voice is great, but what if you can't hear?

What if you don't communicate verbally? Then you want some sort of textual content that corresponds to the audio you're hearing. And that was... It's related to captions.

Thank you, Mirabai, by the way. The voice content and the audio content should have a corresponding textual content on the page. You see how all of these things... Like, it kind of makes sense, and then if you learn the WCAG principles, then you start to understand how design and usability and inclusive design come together with accessibility and compliance.

Another point I wanted to make about Deaf and hard of hearing is that Alexa actually has devices that have displays. So designing for a voice experience shouldn't necessarily be voice only. Designing in a multimodal way so that you can basically progressively enhance your design will support a wider array of users. Also, I've pulled up the picture of an Echo Show's accessibility features, including captioning and tap to Alexa.

We'll talk about this after. So generally, what I want you to take away from this is that when you're designing for accessibility, it's not about the tools. It's not about the plugins. It's not about the guidelines.

The rules. It's about creating an experience that works for everyone. Or at least the broadest range of people you can manage. And voice lets you do that.

Speech lets you do that. Text lets you do that. You just keep building on top of all of those components, and you'll end up with something pretty good. Thank you.

(applause) So I'm happy... We're here. We have this space. I'm warmed up.

So I welcome questions. If you have, like, very specific technical questions, I'm happy to answer. If you have thoughts, if you have ideas, let's hear it.

And there's a mic that I'll bring around.

CAMERON: Yes. Please wait for the mic so we can get captions and livestream goodness.

So the annotations that you're showing us earlier, all of the, like, icons and stuff -- do you put them in your wireframes?

CAMERON: Yeah. You guys... Well, you have a mic, so I'm not gonna repeat the question. Yeah.

The annotations that I showed earlier would actually be in the wireframes, adjacent to the designs. So that's sort of the approach that you can take. It's frankly a workaround right now. But it's the best that any of the design tools can manage right now.

And I think some of you as designers working in accessibility have some variation on that, right? Like breadlines, essentially, for accessibility. I look forward to sharing more about that soon.

I'm excited to hear it. Thanks.

CAMERON: Thanks.

Hi. I was wondering if you could talk a little bit more about either the possibilities or limitations of using what you showed to prototype and user test. With blind users.

CAMERON: Yeah, sure.

Just general screen reader users.

CAMERON: Absolutely. So the features that I just showed will work with a couple of exceptions on web. So you could share that VoiceOver experience that I just showed with a customer. Who is blind.

Who uses a screen reader. They have to turn off their screen reader when they're going to use this. But they could kind of smoke test your tool. I would recommend doing it in person.

And walking through it with someone. This is not really... My opinion, my take on design tools in general, and XD, is that it's a communication tool. You're communicating with stakeholders, sort of pitching an idea, selling something.

Right? You're communicating with developers. Here's how... Here are the specifications.

Right? I don't think that that aligns super well with user testing, at least for disability. Because disability user testing has a lot of parameters that you need to put in place. And it's hard to do that remotely.

There are some tools... Correct me if I'm wrong here... Loop 11? They still do user testing.

Usertesting.com has communities that you can tap into. So that's... If you want to talk about that afterwards, I'm happy to... You don't need to see my Twitter handle.

But okay. If you want to talk about that afterwards some more, come find me.

Here I come with the mic.

I just wanted to make sure I'm getting heard. I saw this button. It's the power button. I do have one question.

So I see that Adobe XD is compatible with Apple VoiceOver for making your prototypes. What about the equivalent technology in Android? Can it be used for that as well?

CAMERON: Let me break that down. So everything that I showed I designed in XD. Including those overlays. So...

The components... Basically, the short answer is yes. You can do the same thing for Android. But we don't have a UI kit, per se, for Android Talkback.

That could be something that could exist pretty easily with the speech playback features that I showed. Yeah. Does that make sense? You see what I mean with the components?

So like here... These are actually components that I can edit. So if I wanted to say, like, make this red, I could. This is where it gets really interesting, in my opinion.

Is not just recreating the user experience, but what if we could use XD to create new assistive technologies? What if we designed the next screen reader in Adobe XD? I think it's entirely plausible. And I'm excited to see what all of you do with it.

I might beat you to the punch. But we can try! Yeah?

I do have another question. So... Maybe this is a bit simple. But let's say you want to make an interface, and it uses speech recognition or screen reader to read the different parts of the components.

If you're creating this prototype in Adobe XD, how would you switch it so that it also reads out in different languages?

CAMERON: The language support is across all XD-supported languages. So if I set my operating system to be in France, and I switch my XD language to French, it'll give you a variety of French playback voices, and it will recognize French speech. So it's globalized in that sense. Question behind you, Shawn.

So this made me think of Google Home as a software solution. But I can't think of the top of my head of a really good example. What's your favorite solution of voice tech that you've recently used, that I can go and explore and see how they've done it really well?

CAMERON: Solution using voice tech? Voice control?

Yeah, outside of like a Google Home --

CAMERON: So there was a big announcement at WWDC a few months ago showing voice control. It's a feature that exists in -- has existed in Windows as well. I think that the marketing around that is a really interesting illustration of how voice control can be meaningful for someone with a disability. Do you know which one I'm talking about?

It's a man who is using a wheelchair, who has a motor disability, who is opening messages, he's opening maps, he's creating an adventure where he goes out into the world, and I think that was a really good illustration of how voice control can enhance someone's quality of life, in addition to being just strictly a utility.

That's really cool, yeah.

CAMERON: Yeah, it's exciting. The voice control stuff in general is super exciting to me personally. And there's more there. But I need to keep my mouth shut.

(laughter)

Thank you.

We have a question from the stream. Can you show an example with the annotations in wireframe?

CAMERON: I don't have one here. But I'll share one on the Dropbox. Tomorrow. I need to get some food.

And that'll be on the Meetup page.

CAMERON: Yeah. The Dropbox link is on the Meetup page now. It's in the comments. So that has the annotations file.

And all of the other components, the VoiceOver components, that I showed.

Any questions here? Yes?

Thank you. Hi. You mentioned a lot about prototyping accessible experiences using XD. But I was curious what kind of support XD itself has for using XD...

If that makes sense.

CAMERON: Yeah. Good question. Really good question. We have a couple ways of answering that.

There's like the official answer, which is like... You can go to Adobe.com/accessibility whatever and see our official metrics. Which would be appropriate. There's also more creative answers.

Such as... You can open accessibility inspector and check it out. I encourage you to do that. But...

I'm not gonna take a position on that. Try it out. And tell me what you think. We do have a user group set up for prerelease for people who want to try voice control with Adobe XD.

So if you want to be on that prerelease, let me know.

Thank you.

CAMERON: You're welcome.

Hello. I'm kind of new to designing for accessibility and inclusion.

CAMERON: Yes.

But my question is: If I had a prototype that was already built out, if I need to work backwards to make sure it's accessible, what would that process look like, using this method?

CAMERON: This approach that I just illustrated is very much a green field design. Right? You're starting from scratch, essentially. If I were to inherit a design, which is, let's be honest, more often the case than not, I would do a couple of things.

I would try to build -- like, extract components from that design. Sort of create a design system, a coherent set of components, and then start to break those down, and you can still think about it holistically as an interaction, but by chunking the components, you can start to break down the problem. Also, think about the kind of user flows that you're accommodating. Focus on the things that are high conversion.

Focus on the things that are core to the application. So, for example, you could have a great product page, but if you can't check out, what's the point? I think this is where thinking about multimodal comes in handy. Because, well, let's think about that for a second.

And this is not the official accessibility answer, but what if we made an Alexa skill where you could check out? Or Google Assistant skill? Where you could buy something from the website? Does that accommodate the requirements?

I mean... You will have to decide. And it's a complicated question, actually. Amazon, I guess, famously, notoriously, has a separate "accessible" website.

And it's problematic for reasons I think you can imagine. But... I think it's more interesting now that we have voice experiences that are first class. And I think that's where voice assistants become interesting as an assistive technology, because we're designing for them at the forefront.

It's not an afterthought. It's not something that's gonna go and get stale or forgotten about. If you build a voice assistant ability, you've got to own that. Because it's not something you can write off as like...

Well, only a fraction of a percent of our users ever even look at this, so if it gets out of date or is broken, maybe we don't prioritize it. If it's something that is more broadly used, then you don't have a choice. It forces the issue. Sorry, that kind of didn't...

That veered off. But to try to bring it back... If you inherit a design, what I would say is like... Try to break it down into user journeys.

Identify the flows that you want to really focus on. And then within those flows, identify the behaviors. And identify the components. And then just kind of create a narrow path that you can accommodate.

Even if you had to hold someone's hand, even if you had to get on the phone and show them how to do it, create that path and then start... Get a wedge in there and start expanding outside of that. Like, how do you go from accessible to actually inclusive?

Thank you.

CAMERON: You're welcome.

Other questions? All right! Well, thank you so much! (applause)