Voice Content, Multimodal Accessibility, and the Story of Ask GeorgiaGov | Accessibility NYC Meetup Recap

Preston So: The first thing I want to say is thank you so much to Cameron for that wonderful introduction. I'm just going to jump right in because I know we don't have a whole lot of time. I'm hoping to make it through all of the slides today. But if you're looking for a dedicated book that is about this topic, specifically voice content, and usability, with mentions and coverage of multimodal accessibility voices role in accessibility overall, as well as of course, the Ask GeorgiaGov case study that I'll be working through over the course of today, please feel free to purchase my new book. It's now available right now at bkapart.com. And it's the first ever book about voice content, strategy and voice content design that exists in the world today. I also, by

the way, have another book coming out later this month, called Gatsby The Definitive Guide. For those of you who are web developers, or Java developers, this is a really handy book for anyone who wants to learn about Gatsby js. It's published by O'Reilly and has a foreword by Jason Lang store if you can get more information about that, at my website, preston.so/books/gatsby. So today, I want to talk about a few different things. And I don't know if I'll get to all of the content today. But this is going to be really focused on voice and how it can really contribute to multimodal accessibility, and specifically how we have made that work within the realm of our work on Ask GeorgiaGov, which is the first ever voice interface for residents of the state of Georgia, as

well as among the first ever content driven voice interfaces in existence. The first thing I want to talk about is what exactly is in a conversation? And why is it that we are so interested in conversational design and conversational interfaces, and of course, voice interfaces, even though there's still a lot of discussion to be had about accessibility with regard to voice that I want to talk about. Why it is that voice has really captured the imagination of a lot of people who work in accessibility and accessible tech, especially those who have struggled with a lot of the previous foregoing paradigms that we see in web technology and web experiences across the world.

I want to talk about the differences between delivering content through the web, and delivering content through voice interfaces. And as many of you will expect, a lot of that has to do with some of the comparing and contrasting that we have to do between screen readers, and pure voice interfaces that really revolve around speech. I'll talk a little bit more about some of the ways in which we built voice content through Ask GeorgiaGov and this whole project. And then I'll end today with a little bit of discussion about some of the accessible outcomes or some of the accessible outlooks that are available for voice content here to come in the future. So what exactly is in a conversation? Well, Erica Hall in her book Conversational Design -- she also incidentally wrote the foreword for my new

book -- had this to say about conversation conversation is not a new interface, it's the oldest interface. Conversation is how humans interact with one another and have for millennia, we should be able to use the same principles to make our digital systems more intuitive to us, by finally getting the machines to play by our rules. Now this quote is really interesting, because I think it really is tugs at one of the most important distinctions that happens today between what I would call natural interfaces or natural interactions, or organic interactions which revolve around human conversation, whether spoken or signed, and also the artificial forms of interactions that we constantly do on a day to day basis with some of the more artificial or mechanical interfaces that we use today, namely keyboards and mice and video game controllers, and refreshable

braille displays, all of which are very different from the sorts of natural conversation that we have through sign language, or that we have through spoken language. Spoken or signed conversation is inherently messier than written conversation. One of the reasons why spoken interfaces or voice interfaces are so much more challenging than some of the chatbots that we see today, or some of the written interfaces that we work with today is that it's inherently more human. It's inherently more organic, it's inherently more spontaneous, it's inherently more extemporaneous. Which of course, means that it comes with all of those foibles that come along with the problem of human speech, and human speech. We have all of the issues of body language of social cues of arms and hesitancies. And all of these things that happen in spoken or signed conversation, which really don't occur when it comes

to a written conversation. That is just bits on a page or bit In a textbox. According to the book, The Conversational Interface by Michael McTear and several others, we engage in conversations because of three specific reasons. We need something done, like conducting a transaction, we want to know something like information of some sort. And of course, we're social beings, and we want to have someone to talk to. And these are, of course, prosocial conversations for conversation's sake, where we might speak or sign with somebody across the deli counter or speak with somebody across the way within a bar or within a restaurant. Of course, we distinguish between voice interactions or conversations. Conversations are, of course, an exchange of ideas that lasts an arbitrary length of time, they might accomplish some of these things, or multiple aspects of these sorts of things. But voice

interactions are very different, because they realize an outcome for the user that falls into one of these categories, whether it is a prosocial voice interaction, or an informational voice interaction, or transactional voice interaction. So, all voice interactions are conversations, but not all conversations that we might have using ASL or that we might have using, for example, a text conversation on SMS. Those are not necessarily voice interactions. Voice interactions are really about realizing some form of an outcome for the user that falls into one of those categories. Here's an example of a transactional voice interaction. In this dialogue, we have two distinct people. One is named Alison and one is named Burhan. And Alison has just come to visit Burhan's pizzeria. Alison says, Hey, how's it going? Burhan says, Hi, welcome to Crust Deluxe. It's cold out there. How can I help

you? And Alison responds, can I get a Hawaiian pizza with extra pineapple? Which by the way, is the best kind of pizza there is. Burhan asked her what size? Alison responds, large. Burhan says, anything else?

Alison says, No thanks. That's it. Burhan responds, something to drink? Alison says, I'll have a bottle of Coke. And finally Burhan ends the conversation by saying, You got it. That'll be $13.55 and about 15 minutes.

Informational voice interactions are very different. In this conversation, we're actually going after something very different. It's more of an investigative journey or more of a really a sleuthing process to understand some sort of information rather than conducting a pizza order. Alison begins the conversation the same way. Hey, how's it going? Burhan says Hi, welcome to Crust Deluxe. It's cold out there. How can I help you? Alison, instead of responding with some sort of an order, she says, Can I ask a few questions? And Burhan says, Of course, Go right ahead. Alison says, Do you have any Halal options on the menu? Borhan says, Absolutely, we can make any pie Halal by request. And Alison responds by saying, What about gluten-free pizzas? And Burhan says, We can definitely do a gluten-free crust for you. No problem. Anything else I can answer for you? Alison

says, That's it for now. Good to know. Thanks. And Burhan ended the conversation by saying, Anytime.

Come back soon. So, voice content involves informational not transactional interactions. We're not necessarily here to order a pizza. But as you can see, the conversation between Alison and Burhan could have gone for much longer, there could have been more exchanges. And eventually she could have ordered a pizza. But voice content in its very atomic form involves an informational voice interaction that is really about asking those questions to understand the truth are they're gluten- free pizzas available? Are there halal pizzas available at this restaurant? But one of the differences between voice content and, of course, the content that we typically find on a web site, is that voice content involves the transmission of information across time, not space. And this is one way, as a matter of fact, how accessibility really ties into the notion of voice interfaces, and this

transmission of information across time. And I'll get to that in just a little bit. But I want to first turn to a little bit of discussion about voice interfaces themselves, because one of the challenges of voice interfaces is of course, that they are not necessarily the most accessible things themselves. And they also have certain issues that tie into a lot of these experiences that we've built for, for accessible reasons or not.

Erica Hall writes, The ideal interface is an interface that's not noticeable at all. And of course, this is known in user experience as a zero user interface. Unlike a lot of the interfaces that we work with on a daily basis, like keyboards and mice, touch screens and dial pads on a smartphone, a conversational interface that resorts solely to human speech and conversation is a zero user interface because there is no visual or physical component to interact with, apart from the power button or some sort of other button that might do a reset. And of course, zero user interfaces are very interesting from the standpoint of science fiction, from the standpoint of all of these experiences that we interact with on a daily basis. But there's a subset of voice interfaces that are really

interesting for several reasons. The first reason is really the most common reason, which is pure voice interfaces are interfaces that have no screen have no physical component, and only allow interaction through speech. But, of course, the second reason why pure voice interfaces are interesting is because, well, as a matter of fact, there are multiple kinds of speech. And it's not necessarily just speech that isn't signs that we're worried about. Pure voice interfaces, in a certain sense, especially when you think about the notion of spoken interfaces or human speech interfaces should really be able to allow interaction through sign speak as well. So pure voice interfaces are interfaces that have no screen only allow interaction through vocal speech, but of course, are inaccessible to the deaf community, are inaccessible to people who are really people who need to have the ability to interact with

a voice interface through sign language. And I'll get to a little bit more about the implications of this towards the end of my talk today. There are, of course, several voice assistants that are part of this kind of realm of pure voice interfaces that we interact with on a daily basis, like apple, Siri, Amazon Echo, Windows Cortana, and Google Home. A lot of these voice assistants are very spartan, they're very austere when it comes to their design and when it comes to their ability and capacity to engage in rich user interactions. And as a result, we have certain interesting historical patterns that have emerged over time that really have culminated in the development of voice assistance. Back in the 1990s. interactive voice response systems or IVR systems were the first true voice interfaces that engage in authentic conversation, whenever you call, a phone

number, or a phone hotline for an airline or hotel, for example, chances are those initial interaction you have with them are engaging in that sort of conversation that revolves around interacting response systems. Of course, this came in around the same time as text to speech devices as well, which is another very important way in which we conduct phone calls with various organ sensations. But voice assistants are very different. Unlike written chat bots, which are [inaudible] same amount of emphasis, voice assistants are nano channel, which means that they're hardware specific, whereas chatbots are monochannel. What I mean by that is, generally speaking, you can use a chatbot on a smartphone, on your browser, on a Slack channel, on a Facebook Messenger window.

Whereas voice assistants, you really can only use within a certain setting, whether that means an Alexa device that is connected to your car, or an Alexa device that's sitting within your bedroom. And that is an example of a voice assistant that's really monochannel because you really need to acquire the hardware in order to be able to use it. There's also omnichannel frameworks, which I'm not going to mention to a great extent here today. But Dialogue Flow, Amazon Lex, a lot of these frameworks are now emerging that blur the boundaries between monochannel and omnichannel frameworks. But this taxonomy here glosses over one very important category of voice interfaces, and in my opinion, possibly the most interesting to study category of voice interfaces. And that is, of course, the example of the off the shelf voice content implementation that exists today, the screen reader,

which is, of course, both exalted and also demeaned within the realm of accessible technology. I think one of the key questions that a lot of us ask today, especially those of us who work in the realm of accessibility is, do screen readers today truly deploy the content that people with disabilities need to access as efficiently as they could. And there's a lot of different examples of this. There's a lot of different discussions that have taken place about this. And that's where I turn to next. But first, before I go into some of this information, you can see more information about some of the things I just mentioned here in chapter one of my book, Voice Content and Usability. So, let's talk about voice and its role in multimodal accessibility. And I'll talk about multimodal

accessibility towards the end here. The screen reader is very interesting because it's really one of those older voice interfaces that in many regards proliferated much before the advent of the interactive voice response system or IVR system. Back in 1986, the so-called Research Center for the Education of the Visually Handicapped at the University of Birmingham developed this Screen reader for the BBC Micro, and the NEEC portable, which was the first ever screen reader for a user interface or a device that had a screen. And that same year, Jim Thatcher over at IBM created the first IBM Screen Reader for text based computers that was later rebuilt from the ground up for computers that had graphical user, interfaces better known as GUIs.

But of course, one of the things that I think is really tough about the screen reader is that they are fundamentally about transcribing visual content and visually readable content, and for blind and visually impaired website users. Screen readers that transcribe visual content in this speech are the predominant method of interacting with text and multimedia and format elements and sitemaps, and all of the things that make the web great today. Of course, thanks to the introduction of semantic HTML, and especially ARIA roles, beginning in 2008, screen readers started facilitating interactions with the web pages that we interact with today, allowing users with disabilities to traverse the page as an oral and temporal space, rather than a visual and physical one. As Aaron Gustafson writes in "Conversational Semantics" in List Apart magazine, screen readers for the web provide mechanisms that

translate visual design constructs, proximity, proportion, etc. into useful information. At least they do when documents are authored thoughtfully. And I think this is one of the key problems and the key tenets that many of us who work in accessibility deal with today, is how do we make sure that a lot of these visual design constructs do not obfuscate or obscure the information that we need to provide and deliver, because there is one significant issue and one huge weakness when it comes to screen readers, they're difficult to use for many users, and unremittingly verbose for the vast majority of use cases, even if you skipped the main content, even if you skip navigation. For many screenreader users, working with web based interfaces exacts a heavy cognitive toll, it requires a huge amount of

work. And it really results in a great deal of fatigue. Screen readers can be less efficient. And I'll use georgia.gov as an example of this. This is how georgia.gov looked back in 2016, 2017. We first built the voice interface that would revolve around georgia.gov. Now georgia.gov has been at the very forefront of all of these concerns that come about with the Web Content Accessibility Guidelines, WCAG, with, of course, making sure that there's alternative texts available for all images. But a lot of these things are not really very convenient for screen readers -- links, navigation bars, nav bars, calls to action, form elements, multimedia, a lot of these things require a lot of spelunking up and down and rappelling up and down through these elements within HTML to understand, for example, what the alt text is

for an image, or what the caption is for a video element, or what makes up a navbar, if you need to go back and click on a link back there. So screen readers are very inefficient when it comes to working with websites that are very much visually routed. Chris Maaury is a blind technologist who works on voice technologies, and he wrote in Wired magazine about his experience as a blind user of screen readers. And he writes this, From the beginning, I hated the way that screen readers work.

Why are they designed the way they are? It makes no sense to present information visually, and then, and only then, translate that into audio, All the time and energy that goes into creating the perfect user experience for an app is wasted, or even worse. When I read this quote for the first time, it really piqued a lot of my imagination, which is that we spend copious amounts of time, as people who work on accessibility, into making sure that all of our websites adhere faithfully to WCAG guidelines, that we adhere faithfully to a lot of the tenets of accessibility on the web that we now understand. But what Chris Maury is asking here is a much deeper and much more fundamental question -- if the use cases that we're trying to solve, with the requirements that we're trying to enable, are things that are

fundamentally broken when it comes to these visual underpinnings of the website. Why shouldn't we go into a completely different direction? And this is one of the reasons why, of course, Chris Maury is now working on voice technology and working on voice interfaces to solve this problem. Because one of the things that we've seen, not only from our work on Ask Georgia Gov, the first ever voice interface for residents of the state of Georgia, but also with other voice technologies, is that often well designed voice interfaces can rapidly get users to their destinations faster than long winded screen readers can, and a lot of that is simply because of the focus, the change in focus away from building a visual foundation for a website towards, instead, building an oral and verbal foundation for that same content. So,

why voice interfaces plus screen readers? And this is one of those things that I always mention, voice interfaces should never be alone in terms of your strategy for accessibility. And as we know, multimodal accessibility involves the provision of accessible interfaces, assistive alternatives, across multiple dimensions of user experience. This doesn't just mean screen readers for web users. It also means voice assistants for blind users, refreshable braille displays for deaf blind users. other means of interacting with our content and our transactions on a regular basis. Of course, we're seeing a lot of those ideas begin to percolate through the World Wide Web Consortium, for example, with their new WCAG guidelines. and beginning to understand that it's not just about these visual or physical elements that we have to manipulate, it's also about a lot of these things that go well beyond what

screen meters have been capable of from the beginning. And whereas visual interface users can have more areas irrelevant to them in the viewport, blind users who are using screen readers have to listen to every single utterance, and voice interfaces, for that reason, really should exist in an accessible world, in parallel with and alongside screen readers for content delivery, and conducting transactions, and so on, so forth. So, with that, let me move into some of the conversation about Ask GeirgiaGov.

And I really wanted to share the case study here, because this is a really interesting example of how we endeavored to use our user research, to use our demographic understanding of people in the state of Georgia and people with disabilities in the state of Georgia, and also senior citizens and the elderly in the state of Georgia, to really understand how we can answer their questions at home. One of the things that we know about state government and, of course, local government, city government here in New York City, is that a lot of people are unable to access physical offices. They might not necessarily be as comfortable calling a phone hotline or using a website. So, our goal with this project was to really look at how can we make a lot of the Frequently Asked Questions content, which is really the most

visited website content on georgia.gov, available through Amazon Alexa, so that people could have a separate form of access to this sort of content for everyone who needed that across the board. And this really involved, of course, looking at some of the demographics, looking at some of the analytics, and analyzing what are the things that people need from this website content, and how can we deliver it to them in a different way? That makes much more sense. But here's the big question. And this is where I move a little bit more into the realm of content for content strategists, and many of us who are on this webinar or this zoom call today, this meetup today, we interact very often with content that is on a website, and we have to think about how do we make this accessible not just from the standpoint of the

WCAG guidelines, but along other axes of accessibility that really enable and enrich and offer more nuance to people with disabilities, and their experiences of our content. But the key question here is, how do you juggle parallel content across screen reader and voice? And there's many articles, and many sessions, that have been given about how to actually write content for screen readers, not just from the standpoint of audio roles, and from semantic HTML, but also from the standpoint of how to write content, period, how to write content in and of itself, in the form of copy. For most organizations, unless you have better reasoning -- and I share a lot about that within Chapter 2 of my book -- the really kind of, like, you know, foolproof answer, or the most

important and sort of the most graceful answer, is to adjust the existing web content you have, so that they're amenable. All of those pages are amenable for both screen readers and pure voice interfaces, in addition, of course, to the web users who do not use screen readers on your website. And there's two key aspects of this, that really also apply to screen readers as well, when it comes to usability of screen reader driven experiences. The first is content legibility. Is it understandable to a listener, rather than a reader?. And the second, of course, which I think is even more important when it comes to voice interfaces, as opposed to screen readers, although it is still very much important to screen readers, is content discoverability. Is content discoverable for users who need to interact with

conversational flows or dialogue driven flows, rather than clicking on links or following links within the context of a screen reader? In Ask GeorgiaGov, we decided to really mirror the same structured content and the same organization of content, the same hierarchy of content within the Georgia.Gov website, in order to accommodate the expectations of users who might be using both properties. As an example of this, Georgia.Gov at the time it was back in 2017 had a popular topic section made up of multiple Frequently Asked Questions pages. You can see there are three examples here on this screenshot right now name changes, saving for college and voting and all of them had a summary section at the beginning after the topic header with a What You Should Know bulleted lists and we took the introduction paragraph and the what you should

know section that bulleted lists together as a essential What You Should Know section for voice interface. After that What You Should Know section we had Frequently Asked Questions, items that were organized into question and answer couplets. So you'd have a question And then an answer. An example of this is, one question could be, how do I transfer my out of state driver's license? Once I moved to Georgia? That's a question that makes up the first part of a frequently asked question. The answer would start with the sentence. Within 30 days of moving to Georgia, you were required to visit the nearest department of driver services, customer service center, and so on, so forth. So we wanted to make sure that this structure was something that could also be implemented and reflected within the

Ask GeorgiaGov voice interface. So what we did is we performed a first ever what we call omnichannel content audit, to examine cross-channel legibility, and discoverability. And I'll show just a few examples of what that looked like. This voice content audit that we did for Ask GeorgiaGov was really one of the first times that I worked on a project where we did something that had never really been done before in the industry, we looked for a lot of other examples of this sort of project came to content audits. But generally speaking, content audits are really more focused on regulatory compliance, they're really more focused on things that relate to editorial voice. They're also related, of course, to making sure that that accessibility is followed in the form of accessibility audits. But there really wasn't necessarily an audit that was oriented towards those

who might also be using a voice interface, just solely using the voice interface, in order to interact with this content. So, we had a table here, as you can see, on the page, the document that we used, that had a certain legend with certain items like, Keep intact, Reword surrounding sentence per suggestion, Rewrite phrasing of question, Needs further discussion, Other, and No action needed. And over the course of evaluating all of the content on the georgia.gov website that we planned to implement within the Ask GeorgiaGov voice interface, we discovered a lot of different issues. And, like countless other editorial teams, as we know, especially here in New York City, Georgia had limited funding, they could only manage one single source of truth for content. So, in some ways, they were already pushed towards this

direction of saying, well, in order to maintain all of these different manifestations of this content, one for screen reader, and the website, one for voice, one for of course, their written chatbot that they have today as well, as well as, of course, other forms of content that may emerge in the future, like virtual reality or augmented reality, Georgia wanted to make sure to have only a single source of truth for all of that content. So what we did is we audited for voice content legibility and voice content discoverability. What that entailed was, well, let's talk about links and calls to action, for example, and I think this is one of the ways where screen reader usability dovetails nicely with voice interface usability. As we all know from interacting with ChromeVox, or

interacting with our own screen readers, links can be a little bit of an interesting thing to deal with, an interesting element on the web to deal with when it comes to screen readers, and voice interfaces are exactly the same way. One of the problems with voice interfaces that screen readers do not have is that with voice interfaces, it's very difficult to follow through with a call to action or click on a link, there's no way to color text blue and underline it, or identify a link element within a voice interface unless you build that in as a custom piece of functionality. And then of course, you might as well be rebuilding certain aspects of the screen reader. So, addressing links and calls to action were very important. As an example, I'll read out just a couple of these example cases.

One of the questions that we dealt with was about Medicaid, and one of the answers that was on that page was Medicaid has several programs that can help you and your family, then a link, Learn more about these programs. Of course, one of the issues with this whole endeavor is that if you hear the phrase, Learn more about these programs, not in your screen reader but rather in your voice interface, you're going to be very confused. And you can imagine a lot of other calls to action that have the same issue, like downloading a PDF, or read more, or find out. These phrases are really not something that's very meaningful to a voice interface audience. So, we changed that text to actually recontextualize the link within the sentence that came before. So instead of Medicaid has several programs that can

help you and your family, link, Learn more about these programs, we instead said Medicaid has, link, several programs, and, link, that can help you and your family. And a lot of these approaches really helped us to reduce the amount of verbosity that we had within our content, and also to help reduce the numbers of calls to action. Because one of the things that has been found over the course of a lot of content strategy over the years is that calls to action really aren't that great for what Aaron Gustafson calls headless UI users, or user interface users, with where those interfaces do not have a visual component.

Verbosity was also a big issue. I think one of the things that we know from working with screen readers, as well, is that screen readers can be very verbose and they can read out these Russian novels of content and, oftentimes, by recontextualizing these links and getting rid of calls to action, we were actually able to reduce the length of a lot of this content. Here's one example. The before state of one of the answers was, The Georgia Lottery Corporation only raises money for the Georgia Pre-K and HOPE scholarship programs, it does not administer those programs. Read more about the, link, Pre-K program, end link, and the, link, Hope Scholarship program, end link. Now, after we recontextualize those links and got rid of that Read more call to action, the content not only read smoother, it actually shortened

the text so that we could link only the first mentions of the Georgia Pre-K and HOPE scholarship programs. So, adressing verbosity is, of course, another very, very important issue for a lot of not only the ways in which we can improve the experience of voice interfaces, but also for screen readers in the long run as well. Voice interfaces have a bit of a different problem, though. And that's, of course, revolving around context and ambiguity. One of the biggest issues of using a voice interface is that you have no screen and, of course, a lot of ambiguity surfaces from context that is settled elsewhere on the page. One example of this is a FAQ topic that might deal with unemployment benefits. But later on the page, you have a question that asks, How long can I receive benefits?, without any further mention

of unemployment benefits. Again, our voice interface for Ask GeorgiaGov involved a search of content, where you would search for a topic and then traverse a tree of content in order to access your question and your answer. So, if you're searching for unemployment benefits, there's a lot of ambiguity that gets introduced. Because if you don't have a question that's phrased, How long can I receive unemployment benefits, as opposed to, How long can I receive benefits? You're going to ask the question, Which benefits? Here's another example. One question that we found on our FAQ pages was, How do I start my loan application? But, of course, the State of Georgia administers multiple kinds of loans, and this was on the page that was titled Applying for Environmental Loans and Tax Credits. So, we changed

that question phrasing to, How do I start my environmental loan application, to be very, very clear about disambiguating those things. Of course, one of the final things that we did in our audit was also to evaluate some of the cross-channel interactions. One thing that we found was that a lot of voice users will interact with not only one, but multiple devices over the course of their work with voice interface. For example, you might be using a voice interface. Later, scrolling on your phone, you might be using a voice interface, while you've got your iPad next to you. You might be scrolling on your phone while you've got some other connected device next to you as well. So, to facilitate these sorts of cross-channel interactions was very important. And one example of this is tha, in those

sorts of calls to action, where we knew that there was useful information on a website that was not owned by the georgia.gov property, we allowed for people to understand that you could go over to that website on your other device, in order to potentially get to that device. And this is an example of how we can enable more cross-channel, more flexible, multi- dimensional, and ultimately, of course, multimodal interactions across all of our devices. One example of this is the before state of this answer was, See the differences between a, link, LLC sole proprietorship and Corporation, endlink. Now that call to action was replaced by a little bit more description, and a little bit more of a pointer out to that property.

So, the after state was, Not sure which business type meets your needs? Learn the differences between a, link, LLC sole proprietorship incorporation, end link, in the georgia.gov blog. A lot of these sorts of compromises were things that we had to evaluate with our team, and work directly with the Georgia editorial team, to really understand how these sorts of changes might implicate some of the advancements in user experience that you wanted to see. So, for more on this topic of voice content auditing and omnichannel content audit, see chapter 2 of Voice Content. And, of course, this really dovetails very nicely with a lot of the accessibility audits that many of us do on a regular basis. So, eight months later, in 2017, we found that we had some really interesting pieces of data that emerged. We already

knew from the very get go, based on the previous data that we had already gathered, with the help of Georgia that a lot of elderly Georgians, and a lot of Georgian's with disabilities, did not necessarily interact with the website the same way is as other users did. And we found, of course, that we had 79.2% success rate on all interactions on his voice interface, which was a very, very high success rate at that time, because this was at the time when Amazon Alexa had just been released. And, of course, it ended up with 71.2% of all these interactions, getting to the point where a phone number was provided. We had an additional feature on our Ask GeorgiaGov voice interface that allowed for somebody to also acquire a phone number at the very end in case they wanted to call that agency. And, of

course, one of the very interesting things that we found was that the most popular keywords that were searched were very different when it came to voice users versus web users. So, whereas web users would search for a different top three list here, our most popular search keywords for a voice audience were totally different. They involve vehicle registration, driver's licenses, and state sales tax, which was a very, very different set of outcomes from what we expected. But, of course, what I wanted to end with here in terms of this section is that the only sustainable approach is a single source of truth for content that's reused across channels. And, of course, the real kind of kicker, though, here is, well, that the screen reader, you have all of these issues settled in some sense,

right? One of the disadvantages that we identified earlier of screen readers is that you've got this visual foundation. But, of course, it comes up as an advantage, because you no longer need to worry about how to navigate across these HTML elements, because the HTML elements themselves, and the hierarchy of the page itself, dictates how you navigate and go through all of the content on the page. But the question when it comes to pure voice interfaces that deal with content is, how do we address issues like navigating spoken content, not written content, and for our audiences, not visual audiences, it's a very, very different kind of approach. Now, I'm not going to have time to get through all of this content here today, but I will encourage everyone who is here to check out my book Voice Content and

Usability, where you can find in Chapters 3 and 4 a full examination of how to write dialogues and how to create effective flows. Let me just skip ahead a little bit here. I see Cameron's already on back on video, probably to give me a time check.

Cameron Cundiff: A few minutes, but you're good.

Preston So: Yeah, let me go ahead and skip to the end here. I think this is going to be the more interesting portion. Here we go. So, let's talk a little bit about the accessible future of voice content, and I'll leave plenty of time at the end for questions. I'll stay a little extra if we have a bit of time. But I want to just talk a little bit about the accessible future of voice content. As all of you have probably felt over the course of my examination of screen readers and voice content, there's a lot to be gained from the enriching of our content on the web, and the transactions we conduct on the web, from providing another alternative to the screen reader that might potentially perform more

efficiently than the screen readers that we build from the get go on a daily basis. And I want to talk briefly here about the accessible future of voice content framed within the backdrop of the inclusive and equitable future of voice content. And here's the question for those of you who use voice interfaces, When you hear Alexa or Siri speak, who is the person you imagine in your head? Who is the person that you picture in your head or the voice that you can hear in your head? Because when you hear Alexa or Siri speak, a lot of the issues that come about from conversational interfaces, from voice interfaces being the most human interface of them all, and those biases, and those forms of oppression, percolate into the voice interfaces that we build. One example of this is that we don't

normally personify our websites -- unless you remember, of course, the old Ask Jeeves. We don't personify visual and physical interfaces apart from gimmicky interfaces like Clippy. We do necessarily personify and anthropomorphize our voice assistants like Alexa and Siri. But what are the implications of this? And what are some of the issues that surfaced from this? Well, there are several, of course, which is that Alexa and Siri are fundamentally, to the vast majority of their users ears -- Well, they're they sound like cisgender heterosexual white women who speak with a general American dialect. There is no room for those who speak in other dialects. There's no room for those who might speak using African American vernacular English or AAVE . There's no room for those who code switch between LGBTQIA+ and straight passing modes of speech, or those who code switch bilingually

between English and Spanish. And of course, there is no room for those who sign, and there is no room for those who also are using other dialects and other forms of speech that we all know within the accessibility community. And so, this really generates a huge amount of concern on my part, and I do this in Chapter 6 of my book. But, voice interfaces must represent the richness of human language, and I don't just mean from the standpoint of how human speech works, from the standpoint of spoken dialects and general American, and all of the American dialects that we have, and the ways in which immigrants use language in the US, and the ways in which queer communities use English in the US, and the ways in which communities of color use English in the US. I also mean the fact that it must

represent the richness of human language from the other forms of human speech -- of course, sign language and how we use gestures and physical forms of language to communicate much more meaning than voice interfaces. So, what I want to conclude with here, and I'll jump into a couple of takeaways here, but before I do, one of the things that I would love to see from voice interface manufacturers today, especially those who are building pure voice interfaces, is not necessarily just a means for the Amazon Echo Show, or some of the ways in which we provide screens that are attached to voice interfaces, but we also need a means for voice interfaces to be able to read, interpret, and speak in sign language as well.

There's a point in the future that futurist Mark Curtis calls the conversational singularity, where voice interfaces will be indistinguishable, they won't be discernible from natural human conversation. But I think one of the questions that we can all agree on is, well, at what cost does that come, and this conversational singularity is a singularity for whom? Because, for all those who suffer from systems of oppression, for all those who are marginalized, for all those who suffer from ablist systems and structures in society, for all those who suffer from transphobic, and homophobic, and misogynistic, and misogynoir-rooted systems within the context of our society, the conversational singularity will not reflect their forms of speech and their forms of interaction.

Until we see that full dimensionality represented there in as well, I have some major concerns about the conversational singularity, not just from the standpoint of all of the things that we've discussed so far with regard to the ableism and oppression rooted within this notion of a single monoculture for all voice interfaces, but also because it puts at a huge disadvantage all of those who live in the Global South, all of those who are frontline customer service workers and operatives, who handle our phone conversations on a daily basis. Because, if you think about it, who were the first to lose when corporations begin to outmode and outsource a lot of these conversations that people have with customer support teams to voice interfaces that are automated? You lose, of course, the ability to speak

with somebody who represents the dialect of English spoken in the Philippines, or the dialects of English spoken in Indonesia or in India. And at what cost do those losses, and income of course, come as well. One of the things that I found interesting is that usability research shows that some users perform some more of these artificial sounding or uncanny valley type interactions, because they're more rehearsable, they're more practical, and they don't really root themselves in some of these problematic notions.

So, a few takeaways for everyone here today. Voice interfaces are the most human of interfaces, they are the most primordial, the most primeval, the most organic of user interfaces that we have today, because they reflect at least one form, of course, of human speech that we have within our society, voice content itself. If you want to actually present voice content excessively, you have to root that in verbal language and oral flows, not visual structures, although visual structures do provide that other alternative for those who want to be able to use and interact with content on a website. Voice content cannot stand alone alongside the web content, it has to reflect an omnichannel strategy that really encompasses the screen reader, other assistive devices, many other forms of user interfaces as well, especially some of the other devices that we see coming over the horizon, like

digital signage, IoT devices, Internet of Things, devices, augmented reality and virtual reality, wearable technology and, of course, haptic interfaces and other devices that we'll see emerge here in the future as well. So, with that, I'll share some resources here, I'll share these slides out and ensure that I get all this content out to everyone who's attended today. Thank you so much to Cameron and Meryl and the team for having me here today. Last thing I'll say today is check out my new book Gatsby The Definitive Guide coming out later this month from O'Reilly. You can get that information at my website, preston.so/books/gatsby, and over at a bkapart.com. You can get my second newest book now, my second latest book now, Voice Content and Usability, the first ever book on voice content. Thank you all so much for having me today.

Cameron Cundiff: Preston, thank you so much. This is this has been enlightening, and also in ways for me challenging, so I want to sort of prompt a discussion here and hopefully you can take another 5-10 minutes. Do you have a hard stop?

Preston So: For 15? I'll have a hard stop. But yeah, 5-10 minutes.

Cameron Cundiff: Okay. All right. So, one of the things that many of us here, as accessibility professionals, might have tuned into, or cued on, with your description of screen readers was this idea that screen readers are basically an abstraction on an abstraction. Right? They are an oral interface on a visual interface to some content or interaction, and that's true, and it's the best we have. So, I think the question is, what do we do with that? How do we bridge between a world where things are visually presented, and have some mapping, to a screen reader, and a fully voice driven experience? What are the middle grounds ?

For example, there are things that screen readers do really well, that Alexa may never do well, like interaction, interactive interactivity, like rich interactivity with links. You touched on this a little bit, but I guess it -- and this is maybe your point about being cautious about the singularity of voice --it's a question of multi- modality, like how do we reconcile all these different modalities? When in fact, we're very limited in the resources that we can we can apply to, to these these efforts, right?

Preston So: Yeah, that's a great. So it's a very broad question. I don't know that I can give a very good answer to that question.

Cameron Cundiff: Maybe I can guide the question a little bit around, about single source of truth, right? So, single source of truth for for content is what you described, it's auditing written content for applicability across a voice experience and a visual experience. How do you do that with interaction? What is the the shared source of truth for a checkout flow, or a signup form?

Preston So: Yeah. So, that's a great question. I obviously didn't have time to get into the usability research section of the slides today, but I think one of the things that we need to examine as practitioners in this area is, especially those sorts of checkout flows, or those sorts of more transactional interactions, which are not necessarily the focus of my book, or the topic today, is to conduct really robust usability testing, and ensure that a lot of the expert -- and this is really where I think content audits dovetail nicely with this distinction between informational versus transactional voice interfaces, also with a distinction between transactional interfaces versus content, is that the content audit is, in some ways, like a usability test for content, right? You read copy, you read text, you've interacted with media, you really want to evaluate

how that content sounds, reads, and is printed, whereas usability testing is really about, okay, how can we do a usability test that's also omnichannel, is also multimodal. And my expectation, and this is what we actually did for those cross-channel interactions, I wish we had gotten to that point. But those cross-channel interactions that I mentioned earlier, where people often have, let's say, a smartphone plus an Alexa device at home, or an iPad, plus a Google Home at home, those were some of those interactions that we really played a lot with during our testing, because that's one way that you can get a more holistic perspective and a more comprehensive look at how not only those individual experiences performed, just like we saw with the content audit, but more so. And I think, more importantly, how similar or how interchangeable are those experiences with each

other? Because the last thing you want, for example, is for somebody who has a certain mental model of how a checkout system should work from their interaction with the website to be completely befuddled and confused, because that same flow does not in any way, shape, or form, mirror or reflect what their expectations are on a voice interface or a chatbot. So, my answer would be, yes. consider usability testing to be kind of that other analog, and to really look at through your usability testing your accessibility testing. How much do these experiences match up with each other when it comes to user expectations? And, of course, this requires a huge amount of research beforehand as well to understand your users very deeply.

Cameron Cundiff: Yeah, thank you for that, Preston. I want to give you time for your next conversation. Doug's question is, will you be giving some sort of version of this talk in the future at any point? I know we, unfortunately, had to truncate some of the slides. Where? Besides, I mean, I guess, very clearly this slide shows your book. I would I would encourage Doug to check out the book, but do you have other opportunities that people can hear you speak?

Preston So: Yeah, so the first I would say is -- so yeah, Doug, great question, please check out my book, it's gonna really have the most complete examination of everything that you're looking for with this content. It is really probably -- I mean, there's no possible that -- like, I've tried to put a lot of this content into an hour long talk, but it simply doesn't work. I do have two upcoming talks coming up, though, that I will mention. The first is An Event Apart, which is not going to be necessarily about this same topic, but it's going to be really about how to translate a lot of those web elements to voice, and actually at An Event Apart in October, there will be another speaker who is talking about

screen readers as well, and writing content -- sorry, writing HTML for screen readers. So, I think it's going to be a very interesting set of talks. The second one is Button. Unfortunately, that's going to be a shorter version of this talk, it's going to be only about 35 minutes, that's also coming up in October, and we'll cover a little bit less about accessibility, but more on the kind of user interaction design and some of the usability work that we did, although it's only going to be 35 minutes, so about half the time that I'm spending with you all here today.