Professional Testers with Disabilities | Accessibility Chicago Meetup Recap

we um i don't know maybe my clock is wrong i have 4 27. uh nick what do you think would you like to shall we get started or do you want to give a couple more minutes yeah uh yeah we could get started um so yeah but that's what your time is at the end for q a sure great um fyi uh feel free to ask any questions in the chat and uh do feel free to continue to introduce yourselves as well until we the speakers start i won't be able to see chat because i'll be switching over to the presentation slide deck um so there we are bear with me for that uh folks did ask about live captions we have that link in the chat and karen if you can place that in the chat again i would greatly appreciate it

you can reach that at any device with a web browser it's streamtext.net forward slash player question mark event equals cda and our captioning is sponsored by mcdonald's so thank you very much mcdonald's you can of course follow along with the slideshow on your own device and look for professional testers with disabilities just go to speakerdeck.com forward slash a 11y chai of course keep the discussions going on twitter at a11ychai or using the hashtag a11ychai we are live streaming this on youtube so hello to everybody on youtube you can ask chat ask questions through chat on youtube as well and if you would like to follow along with us on youtube it is youtube.com forward slash a11y chai we do post the recordings from uh from our meetup groups on there as well so if you'd like to refer back to this

talk and or if you want to send that along to your friends please feel free to do that and i'm just screwing that up all of a sudden sorry for that just a reminder if you prefer not to be in the recording check your video and ensure that the camera is off hold questions for q and a time after the presentation unless it's a technical question of course um the co-organizers will monitor the chats for technical issues and questions oops just i don't know what's wrong with my mouse right now just want to take a pause right now for jobs if you have any jobs at your company that you're listing specifically in accessibility and or you're currently looking for a job feel free to take yourself off of mute and mention it and or feel free to drop

it in the chat so go ahead and pause right now for that all right nobody's going to speak up um you still have the opportunity to drop that in the chat and or you can mention that afterwards as well and we do also want to take a brief mention to the current events obviously um it's very poignant and concerns a lot of us and especially as accessibility professionals we need to acknowledge that there are folks in our community that are being affected so we do encourage you to check out the hashtags disabledblacktalk and disabledblacklivesmatter both of those are currently trending on twitter right now and with that let's go ahead and introduce our speakers heather burns heather's passion for accessibility started early turning self-advocacy into a student-run organization to help others gain the accommodations they needed for academic success at microsoft she worked on the

accessibility of windows and the office family of products becoming a co-inventor of ui automation u.s patents and you can look up those numbers they're very difficult and very long heather is currently a full-time accessibility consultant for equal entry a company founded by a former microsoft accessibility peer thomas logan who we i saw was also on our line equal entry provides clients with accessibility expertise through audits training and expert witness testimony john g samuel john g is an award-winning business development leader with over 13 years of experience building strategic partnerships designing profitable business models and transforming organizations around the world he is currently the head of lci tech and responsible for launching a new technologies excuse me responsible for launching the new technology services business for lci which is one of the largest employers of americans who are blind john has a proven record of

accomplishment of leading starter startup initiatives previously helping build home strings a fintech platform focused on diaspora investments and starting a highly successful joint venture in cameroon for astor a global telecom infrastructure company and without further ado heather and john please go ahead and share your screen and take it away well awesome thank you so much for that introduction and thank you all for inviting us to participate today at the chicago accessibility meetup my name is john samuel and i am the keep innovation architect at lci tech and we are a division of lci and we are the largest employer of american thor blind and our whole mission at lci tech is to create careers for people who are blind and my connection with the disability community started when i was diagnosed with a degenerating eye condition called retinitis pigmentosa when i was a

freshman in college and at the time i didn't i didn't know anyone who was blind and when i got this news it was devastating and it really shook my world and i was ashamed to tell anyone but i moved along with my career and i moved around the world and throughout that time in different places i would figure out different ways to accommodate for myself and i and i was figuring out ways with my limited emission eyesight i was using uh larger text using inverted colors but eventually my site got to a point where i could no longer to screen and i had to use and you know the challenges that i was facing throughout my career i realized the importance of accessibility and so when i launched lci tech i knew that accessibility is going to be something that we would need to

address and um that's what we're doing and uh but i'm joined here today with my friend and colleague heather thanks john so as um kind of kicking off the way that we kicked off the the morning today so it was she her is my pronouns i'm in charlotte north carolina um i didn't purchase it recently but i'm finding that a gift from christmas is a jogging leash for my dog and that is providing me lots of stress relief um as we get some jogs in and she's just a lot nicer on that particular leash than others so that is good but as kind of alluded to my journey with accessibility started pretty early in fact i don't remember a time really before i knew about my learning disability so it was very much when i struggled academically when i had some difficulties

my parents were always a great supporters to be able to say be able to name what it is and be able to tell me okay so that just means you gotta work harder and you might need some accommodations in order to be successful so throughout my academic career i was allowed to have accommodations extra time but one of the controversial x accommodations i received was the ability to use a computer now this was for i could do spell checking i could rearrange sentences and i could craft essays using a computer a lot more efficiently than by hand and at the time you gotta go back in the wayback machine we're talking apple two e's we're talking five inch floppy disks as far as storage right um that the teachers didn't have much experience with computers and they were concerned that if

cl if computers got into the classroom they wouldn't be able to identify what is unique student work so it was really breaking ground with that but i always kind of knew that a computer would help me um then when i graduated um and it was like one of the early um so july 93 the new yorker publishes this cartoon and it's a depiction of a dog sitting at a computer he's at a keyboard he's looking down at another dog and the caption is saying on the internet on the internet nobody knows that you're a dog and this was done by peter stein steiner excuse me and for me i can't remember who cut that out and gave it to me but i remember laminating it and holding on to it because it was so powerful

to me to think that you know on a lot of levels i can pass i can be social i can choose when i disclose about my learning disability and if i stumble over it before i've told somebody about it i get very self-conscious if i run into it with a colleague that fully knows my background i'm like come on you should know about this so um it became very big for me the power that a computer could provide to anyone with a disability so that all disabilities could become could be made to be invisible and allow the individual to have the privilege and the honor to know they get to choose when to disclose and i thought that was incredibly powerful so it led me to the career of computer science and microsoft and kept me passionate about accessibility so

that's kind of my backstory when we get into kind of why we're here you know um i know that some of my friends that i see on the the zoom meeting so thank you for coming um have forwarded this to others that might not have the accessibility background so i'm going to just do a quick background for those that might not be as versed as others on accessibility but really including individuals with disabilities in product testing is really the only way to truly validate that a product or a website is in fact accessible you can use automated testing tools and i love them but if you build a reliance on automated testing tools you can create a culture where developers are coding to pass a test and not to make it truly accessible if you have uh you know okay fine we'll

just give developers screen readers we'll give them voice dictation technology drag and others and the learning curve on those is pretty steep so if you don't also include a significant amount of training you end up with some unintentional consequences and by that i've seen it at least twice in the last year and a half where a developer got a hold of a screen reader and because they were tabbing between all the elements they decided that everything needed a tab stop and they were adding tab indexes to everything and they spent so much time doing that and trying to do tab index in the right order and trying to all this stuff they didn't do the request of connecting the label with the input and allowing the programmatic access so here they thought they were doing this great thing they didn't help screen

reader users at all and they just aggravated poor you know the individuals that were using the keyboard right your your power users as well as those that are using you know cited but physically use the keyboard rather than the mouse and then finally i just want to mention that there are some organizations that do a great job of hiring they include individuals with disabilities on their team we love seeing that but that's not a replacement those team members they grew up with the product they know where all the mitigations are they know where all the workarounds are and they become expert power users without even knowing it so they're not a true attestment to an individual with a disability being successful at your product so i'm going to hand it back to john but before i do

i want to give just some visuals as far as when we talk about usability um and john's gonna go into this a little bit more but i did some internet searches and i found this company uh they're based in california uh interface analysis associations the only reason why i'm highlighting them is because they were kind enough to publish their pictures of their testing room so this is very much the typical testing room uh that you would see there's a desk that is a fixed height desk there are chairs to allow you to adjust that at most they can put paper on that desk they could put a computer on that desk but visually what you're seeing is tons of cameras around the room from the ceiling from the side trying to get every angle to record when somebody enters this testing room what they do

on the walls you see a reflection so you can see a mirror across one side and the reflection on the other side of the wall there's another mirror when we flip over and look at the client observation room you realize that one of those giant mirrors or is actually a one-way mirror so you can have clients and the the facilitators sit in this back room there's a huge monitor that presumably shows all the pictures that are being uh the cameras are capturing to allow you to kind of see what's going on within that testing environment so this is you know what i used in microsoft this is a very typical kind of experience so wanted to provide those visuals but i'll hand it off to john thank you heather yeah so i mean when i got into this field i started wondering why don't

companies use people with disabilities in their usability testing and they started asking this question to people and companies and and they would tell me oh we tried it just didn't work out and you would say oh yeah we you know we just couldn't find really great test subjects and i started wondering and asking myself why why are they not finding good test subjects i started thinking about it you know they're looking out they work nine to five so they're looking for people who are available from nine to five so looking out in on the street trying to grab somebody who who has a disability or who's blind uh and uh and during those time hours and uh you know and i start wondering uh the reason why that person may be available is because they don't have a job and maybe one of

the reasons they don't have a job is because they don't necessarily have the technology skills to have a job and so when they're trying to bring them in to do some uh usability testing those folks just don't know how to use the system and i i just kind of stuck with me and and i started thinking about what does an actual test you know environment look like and i'm so happy that heather was able to describe that because that's what it is if you have to go to a test facility you have to figure out transportation to that that location and then when you get there you know you have to the accommodations may not be accessible you have to figure out that they have these fixed desks and chairs and these systems you know i know for myself

that i have my own computer set up and customized my own specifications and i know a lot of people who use assistive technology they have it the same way so when you get somebody off the street to come into one of these sterile environments and they have to set up their system to their own uh specification you're going to eat up a lot of that testing time and so i can see that that being major issues for not having um great tests and test subjects when you bring in people off the street and then i started thinking about what is it that the test like testing experience look like often the case it's a written document with click here click here and you know when we're designing these type of testing scenarios it's important to think about the input devices and not making any

assumptions because you don't know what yeah what type of assistive technology or input devices that people are going to be using but before we start to look at how to find test subjects for your usability testing i think it's important to take a step back and take a look at why was lci born and so lci was formed in 1936 by a local civic organization in durham north carolina and the civic organization saw that on the streets of durham there were a lot of people who were panhandling and many of these folks had visual impairments and so what they wanted to do was create employment opportunities for these individuals and that's what the formation of lc industries which is now called fbi but when we started to um the start of the business the first line of business was creating cane furniture if you may

remember those the wicker type of chairs and then we moved into mattresses and these type of organizations were popping up all over the country and the federal government took notice and started to procure products from these organizations and today lci had manufactures over 2500 products we have a footprint of 11 states um we have eight manufacturing facilities 50 retail locations two distribution centers and we employ over 800 people and 400 of whom are blind or low vision and so you know when we start to so when we when we start to look at this you know this opportunity of lci we really were focused on manufacturing retail and distribution but although manufacturing is a very noble profession we realized that we needed to start thinking outside outside of those because not everybody wants to go into this profession so in 2017 the company realized that

they wanted to create technology-based jobs and so we launched what is now called lci tab and so the the whole purpose of this was really to create those knowledge-based jobs and careers and when we started looking at the uh we started looking at the statistics and i think that's where it was really important that we really started making a decision based on the the statistics that we had around the black community and in 2016 7.2 million people identified with some sort of uh visual impairment and it was 78 of people had a high school degree 33 percent of people had some type of college um higher education and then only 15 percent had completed about their degree or higher and the real kicker was that only 29 had full-time jobs and these numbers were really just like staggering when we realized that

we really had to do something we had to do something and so we started looking at the barriers that were standing in the way of people from pursuing vicinity stuff and so we noticed that transportation has always been a barrier which as we all know is crumbling every day that we as we live in the new normal but we also noticed that education was a big challenge and also accessibility and so when i when i launched lti tech i realized as i mentioned my own personal chances that we had to address the accessibility issue but we knew we weren't going to be able to do this alone and we needed to have partners to do this so so why am i talking about these these stats and and these these barriers is because even though lci had such a

large number of employees we had 400 people in our organization who were blind we did some surveys and i was really excited when i joined because i thought i looked at the results of the survey and i saw that there were so many people with college education there's so many people who had great experience with computers and i thought this was going to be the great foundation that we needed to launch healthcare tech but as i started doing the assessments of these individuals did stellar on the survey i realized that a lot of these individuals had gone to college or had these prior careers in technology before they lost their vision but once they lost their vision many of them did not continue on with education or continue with uh learning how to use assistive technology and so

we realized that we had to go outside and look extremely for talent and and so today we have 10 people who are on the team at lci tech seven of whom are blind or low vision but you know this took us we had to look external and took us some time to find and so we did face challenges and we although we're always looking for a new town on the team it's something that we've had to really work out to find people who fit the skill set that we were looking for to build up our organization with that i'm going to pass it back to uh heather yeah so really i think it's what john's highlighting is the fact that finding the right participants are so key so at equal entry a lot of our clientele are looking for a specific uh persona

right when they're developing their soft their application or their website they have in mind a specific persona and i can use the example of tableau they're a company that is on our website and i work with they are an enterprise level visualization product they work with a large amount of data they are a business tool they want their primary persona is educated so they want to include participants with disabilities that meet that education bar that would be business professionals utilizing their product we need to make sure that they're proficient with computers if we have a customer even if it's a retail uh website and no their main persona doesn't require a college education but they do expect computer proficiency because if they don't have that from a usability participant then they're going to kind of unfortunately blame the user as opposed to their software

and so we want to make sure that we're giving them the most valuable feedback that they can't really argue with that they have to accept and go okay yeah really we need to address this because even somebody that is highly proficient with a computer proficient with their assistive technology is going to be it hasn't been successful therefore they need to fix it the final component i would say for a participant is really kind of willingness to try and fail and i think this is a true for any usability study you're putting people in a position you're asking them to do something that's a little off the wall or a little new outside their comfort zone they really got to be willing to try and just accept that yeah it might not work and that's okay and then the final thing that we uh feel

very strongly about at equal entry is that these participants should be paid this is not something that you should be looking for volunteers to try to come in and assist with because you are looking for an educated a computer proficient a proficient with assistive technology they have a specialized skill they're providing you with valuable insight there should be some payment monetary kind of contribution back to them so with that once you kind of find the right participants then you know like so what's worked you know so equal entry we've been working with lci for about uh um over a year now and essentially what we found is when we tried to do it ad hoc we had a lot of technical setup issues it was a how do we do this again right it's a which technology did we need to

join how do we just set up the meetings how do we get the technology aligned so we've now started kind of having weekly office hours that's cut down on the amount of china churn from a technology perspective we also solely use zoom zoom allows us to share the computer audio as well as the visual of the individual and their screen so we get great facial expression from the tester that we can pick up on immediately and we can record the audio for the screen reader as well and we'll give an example of that in just a second and then at equal entry we're very specific about the fact that we write our scripts to be as inclusive as possible we um and that's as john was alluding to rather than directions that say click on right it's activate so they can

choose which input device that they're using in order to to to do that uh we also found working specifically with lci that it was difficult for them to kind of switch back and forth between the application that they were working with and the notes or the digital information that we were providing as far as what those scripts should be on what they should do so we have found that it's better to have a facilitator verbally tell a visually impaired an individual with visual impairments to have that read to them so that then they can take those actions without having to switch back and forth between applications so with that i want to pause and i'm going to show a two and a half minute video and this is kind of an early session that we did with lci vaughn was the the person that was

helping us here and what we're going to see is she's working with the equal entry website we had just done a refresh of our about page and we had a developer and a junior tester working on it they had given it the thumbs up and then we had von do a quick review for those i'm gonna quickly put back in the chat window the link to the um full well if i can find where the chat window is can i ask somebody else to where's my chat window i wanna make sure that everybody uh here we go nope i can't find it it's lost in my screen somewhere i apologize but if somebody could resend the link to the the actual powerpoint that we have if you want to follow along in the notes section of this slide so the presentation notes

has the full transcript for what you're about to hear and if you are new to listening to a screen reader this is going to be very fast so you might want to follow along with that transcript in order to be able to understand what's going on but also give you kind of a debrief afterwards so we all cross our fingers and hope that technology works for us today why don't you move to the about page and find out a little bit more about thomas logan okay our work vertical bar equivalent banner recite visitor blog about my contact us medabot menu menu menu what belt title is about vertical bar equilibrium banner region skip to content select heading level one site navigation reach visited link home vertical bar equal entry services many are working about heading level history heading subscribe to our mailing list wrapping

the top about heading level heading level to our history blank equal entry main region site navigate search site contact us about menu menu menus hold on you we're going quickly there so yeah what just happened so i was hearing you hear about the our history uh-huh and then you went to the end of the page okay so you said go to the about page and look for thomas history over the past six years equal entry has gone into a global company with team members joining us from throughout europe asia and the americas and then i think it jumped back up okay main region end over equal entry was founded in 2012 by accessibility how did you get that main region end what key press were you using there to the main region end yeah because you were reading

you the focus was in the uh our history and you're reading that text and then i'm hearing an announcement of region end and then oh i arrowed down or um i was using the paragraph uh command and i moved on too fast that was all [Music] okay keep going i i'm just trying to follow along with what you're doing from a keyboard perspective so and here is the first time that i really lost you okay sorry about that so it looks like i am here in heading level to our history that is the heading and there are two equal entries found over the back main region and paragraphs here over the past six years describing the history of equal entry yes looks like that's it okay i'm going to say that you might have just found a bug for us

so visually what you can see on the screen right now is the equal entries about page with our history has the two paragraphs and then below it you can see that there's another title that is just barely readable of our team and then there's images those images are just the very top of thomas logan's head so and when when it was hitting that last page and it was going all the way to the end you could see that there was another image and it wasn't me so there was clearly content in there that she was missing and what with a little bit more investigation was identified as that we had a misuse of arya hidden that had misfired and was hiding the rest of the content to assistive technologies so the automated accessibility testing that was done passed no problem and our tester

who was using a screen reader was tabbing to active elements and was able to find and read all the active elements on the screen but for a native screen reader that is a screen reader user that is just trying to navigate a page clearly they were hitting some significant problems with the page so it was a quick and easy fix it was only up for a little while i promise but it was fascinating to me and i thought it was a great example of how you can do some very basic testings you can and get a good sense of what an individual would find but then actually working through just making running through some very specific scenarios the big key issues and making sure people are successful is as i said that gold standard of making sure something was accessible so

over why don't you move to the uh started playing again sorry about that so really our lessons learned one sharing the screen with the computer audio is absolutely key we've tried this with other technologies where i couldn't hear what the screen reader was saying and it was just all on the participant to kind of communicate what they were experiencing in their own narrative uh and it just wasn't as effective so i'm pretty verse in uh screen reader so you could tell that i was allowing vaughn to use her own speed of how she was using it or you know how she wanted to use her screen reader as allowing her to do that and i could follow it with experience that's pretty easy to do and be able to understand where she was on the screen but you really need to have that computer audio

individual volume adjustment is something that we would love to see because sometimes when you want the screen reader to be slightly less um and pump up the volume of the individual that is participating so that you can hear what they're saying when they're talking over their screen reader we'd love to be able to do that independent volume adjustment but that's not there yet so that's i'm putting a plug for everybody making that request to their technology vendors um and i've kind of alluded to this as far as zoom allows me um the in the recording all you saw was vonn's screen but when i was participating in that i could see her facial expression and that is pretty critical from surprise frustration confusion right you heard her kind of go uh huh right oh right in in seeing the facial expression help to understand

that those those you know voicial expressions um so if those facilitators aren't used to the speed of the screen reader it is important to ask the participants to slow down their screen reader and again it's just that critical ability for the the facilitator to be able to hear exactly what the participant is hearing to be able to make some judgments over what might be the problem uh and being able to articulate the other thing as we've kind of talked about is like uh or you heard in the video understanding what keystrokes are being used how they're choosing to navigate at that point why they're choosing to do that so in general our biggest takeaway and i don't think we hit on this but when we're talking about like why companies don't do this type of testing a lot of it has

to do with the fact that they require that physical testing environment so we're big believers that we want participants using their own computers we want participants using their own assistive technology that they've customized and they set up so really to host your own usability kind of component and include individual disabilities we really suggest investing the time to find the right participants that meet your requirements for your product your website you know that what's a normal persona or what's the what's the target development persona and then look at it from okay if the individual was with a disability what would the requirements look like right are you willing to have somebody that's brand new to assistive technology or not right um there's various times when there's we've had clients they're like um we are working on keyboard accessibility we're not ready to work with screen

readers yet we're taking a interim step in our accessibility plans and we're starting here and then going to move on so making sure you match the right participants and then we're hitting on this right plan on the remote participant and that means that the development environment has to allow for an individual at their own home to be able to access it using their personal equipment so and then it's budget the right amount of time and i'm going to say multiple sessions and then if you need to cancel because you never know that first session you could have total technical difficulties with just getting everything set up right to be able to get good data back so if you have multiple sessions then you don't feel rushed to be able to get through anything and you can just cancel further sessions then

so and then the final one is set expectations up front and that is expectations with um the product development team that is asking for this as well as the participant so it's oftentimes we would recommend doing this type of usability study after you've done an accessibility a stat a traditional accessibility evaluation to determine where you think an individual is going to be successful and where they might have problems setting that expectation up front letting a participant know okay in general this is an accessible website there are a few areas we're going to have some gotchas so and i will try to warn you of that or avoid those or just letting you know we're going to purposely hit some of those and just having that expectation up front allows the participant to feel a little bit more comfortable

about when they do hit a stumbling block so and with that i think we have made it to our q a awesome um and folks can feel free to take themselves off mute to ask as well um i'm switching over right now to our slack instance i know that we did have some questions cropped up in chat so bear with me while i located that we had a question from joshua barr are you testing on mobile this way as well so we do have uh so lci has participated with doing android as well as ios testing with us i will also say that i have traveled to lci i have the advantage of being in north carolina with them so i have traveled and been able to sit next to them honestly i find the remote just as effective once we've gotten through all the

hurdles with making sure that they have the right technology to cast their mobile device to the computer screen and then share out that computer screen we're off to the races um so yeah so we've done the ios android as well as websites web applications um i think we did one windows client so we've done a whole host of things with him thank you we also had a question from steve i realize this is very bespoke and everyone is different but is there a typical pricing range for facilitating testing in this way if we wanted to make the case for uh to do this i fear the first question is is going to be how much that is an awesome point we did not discuss ahead of time if we were talking pricing at this type of event so

i think i'm going to have to say contact john or i after i don't um i don't know what protocol is for this so you know i'd say just one thing i think one of the main things is we've always tried to make at least from the from our usability perspective we wanted to make it attainable for everybody because at the end of the day we want to make sure that people are not doing it because of it's cost prohibited so we've been trying to make sure that you can do the disability that that's uh reasonable for people that's on the usability side we try to do it as much as we can great thank you folks feel free to go ahead again put questions in chat either here on zoom when we are or on the youtube live feed we'll go ahead and

answer those questions either way i'm catching up on the chat looks like we have a question from andrew how do you balance leading testers versus letting them organically way-find the task it seems there is a good amount of prep to even have the conversation that's an excellent question um so in the video that i showed is kind of go to the about page and look up about thomas logan right that's a pretty kind of broad i also typically start off in a session with a kind of a just general explorer right i'm giving you a url and i'm telling you that this is what we're going to be working with today why don't you just take a look around and figure out what this thing is right so you give somebody a url you don't know if it's a web application you

don't know if it's a website so i often let them just explore for a little bit and again you're hearing the screen reader and going and kind of hearing how they explore which i find pretty valuable but it is specifically when you have a site that's a site or an application that you know is not as accessible as it really should be then you end up having to guide a lot more and we warn clients about that right as far as you want a level of accessibility to be there to be able to be successful and be able to show a solid scenario of somebody being successful um is what we try for and what what what it is this is kind of a carrot to really put in front of clients like do good and then you get this right

you'll have somebody be very successful and you'll have a recording of it so it kind of goes both ways but if it's a very accessible product then the less leading is needed is the best way to say that right awesome um next question have you ever tried visiting the subject at their home or a neutral location so i've met with lci at their hou at their place of work um as far as going to the person's home i can say yes in my past so the i i gave the pictures of the usability testing facility um the first time that i as a program manager requested individuals with disabilities we ran into lots of problems getting people to the testing site the testing sites accessibility getting the accessibility software on the computer and essentially it was an abysmal failure so the next usability study

i did at microsoft i said i want to give people computers i'm going to preload them with our beta software that's not out to the public it's going to be locked down so that they have access to it but we're gonna install in their house and to my manager's credit they go okay how many computers do you need i mean it's microsoft okay fine so here i load up like brand new machines still in their box putting them in my car i'm like somebody's going to stop me nope so i did put them at individuals homes they were people that we had found through advocacy organizations for the blind and honestly we didn't get great data um as far as of the participants only one had really advanced computer skills to the point where they felt comfortable utilizing

this system and the idea was that they would have it at their home and because of the ability to pre-load that beta software it had to be a pretty powerful machine at the time it had to be locked down from a microsoft privacy perspective but we said hey you can install your own assistive technology you can customize it we're going to give you lots of time with this came back later and no like again only one participant had actually used it everybody else was just kind of like yeah i turned it on didn't really yeah didn't you use it in any way um so i have been to participants house i don't really see a huge difference between somebody's comfort level with being in their home versus their comfort level being at their work environment um and then working remotely

they could be at work or at home doesn't matter to me yeah actually i have a quick question for heather heather you know i'm i'm actually um you know i know that our schedules have been kind of a little wacky the last several weeks when we started working from home but have you worked with the team um at home you have how was that experience how was it different pretty much the same right i mean so i've worked with vaughn and tristan and several others both remotely and face-to-face when they're at home when they're at lca the key is having it be their own system having it be their comfort level with their assistive technology and their wide as you throw at them something new from a usability perspective kind of building off that we have a question of how do you overcome

problem when uh testers excuse me how do you overcome a problem of testers getting nervous when observed that is a little bit of time um and it's just rapport um i think so hopefully i'm a welcoming voice and it's not too painful to listen to me as we go through this webinar but it's really um you know part of that is setting expectations early on with the participants as far as you got to be willing to to try and fail because that's what that's what we're looking for right if you succeed beautifully that's interesting data but it's really interesting where and how you fail so that's what we're kind of going for so they got to just be willing to do that and comforted by knowing that hey every failure that they make is a failure that the actual general public won't be making

because we're going to be able to fix it i just want to pause uh for uh for a thank you that i just saw in the comments heather your excitement and passion is contagious thank you for this information and sharing your experiences this is great and i agree with that so i just wanted to make sure that you knew that that was happening in the chat another question that we have how long do you recommend your testing session be per participant we hold it to pretty much an hour or less and that is at that point it gets draining um so the shorter sessions to typically work better and our block allows us to get through any technical issues and still get valuable time so shorter you might end up just having technical issues and not getting to any data

but so we block it for an hour and if we get through we can call it early and you know thank everybody for the great insight and we're going to go run off and go fix things or use the full hour but then cut it off and say okay we'll rejoin and again it's the having multiple sessions really does help us to be able to pick up and revisit things and one thing sorry i want to comment on that i think one of the good things is that you also have a consistency um heather you kind of you have us get set schedule every week you're going to be doing this testing also which makes it consistent for the team which i think is great yeah definitely cuts down with the technical issues yeah do you have some more questions

that are coming through from chat but i see that dan has his hand raised so i just wanted to pause and allow dan go ahead take yourself off from you and ask your question my question is thanks uh do you do braille testing in my current job which ends monday i've been advocating that people need to test with braille displays because i know when i've done ios testing of an app or a website i've caught things where something would might work with voice over speech but not with braille with a braille display i don't think we have done any specific braille testing with lci but again the participants that i would talked about where early on in my microsoft career um the one that was the most computer proficient was a braille uh user so he used a combination of

um jaws as well as his braille display yeah that's what i do jaws nvda voice over top it's multiple inputs right the more the more advantages you have to get the information you need so yeah i love it but i i don't think we've done specific um and i think it would be interesting to look at a usability study where we just did braille without the assistance of the screen reader i'm not sure um what it would be interesting to get some data points as far as the general public uh as far as how often a braille display is used in conjunction with a screen reader or in an in isolation and then be able to build test space usability tests based on that you could look at the web beam website they might have statistics there yeah and actually everybody on the team

i think everyone on the team except myself actually knows braille and they have uh multiple braille displays on the team so that's definitely something you can turn on so they might have been using it without me realizing it and that's fine right if that's information that they have access to but they haven't specifically pointed out that they're using it thank you another question we have how many testers with without disabilities do you recommend and do you analyze the result together i didn't see who asked that question for analyze the ques the the test the results together that's um i think he's referring to equal entry and lci analyzing the test together i wonder um c1 feel free to take yourself off mute to clarify um i read that is the people with and without disability analyzing those results together

um but that's just my interpretation of somebody that's yeah that's my question so like um i've never have experience uh testing with um testers with disabilities so i just thinking like you know in the future if i you know incorporate that and how would i analyze you know data you know from different groups of testers so from my perspective the best user scenarios are going to be the ones that you're doing with both sets of groups right so if you have a a key scenario right logging into a web application being able to search for something specifically and find an article read a full article right those are very basic tasks that you can give to any user um and you should see the same success rate between an individual that's part of a general usability study and participants with disabilities right the

the goal would be to be able to say okay this scenario was well not successful for anyone this is confusing and we need to go back to the drawing board from a design perspective or okay this class of user had no problem with it but individuals with disabilities you know be it vision impairment mobility impairment cognitive something along those lines was having failure issues here so as far as the number of participants that really kind of depends on the product um and kind of what data you want to get out of it but i from a cost perspective somebody brought that up before right there's always cost time uh and the ability to take that feedback in so if i can get you wonderful results with 25 participants but it's gonna take me three months and we just missed the ship window

so you can't actually make any changes based on that feedback then that's not a good study so you're really trying to narrow that down to well how much time do i really have to be able to influence and be able to make changes so sometimes it's one right one's better than none this is the way i look at it if i can get more i will but it's really they when can i get them in how can i do the test and get it back so that developers can make change okay thank you great question next one we have a series three questions um so do you also engage user testing at design or prototype phase how many test scripts do you normally start with at initial stages and what what's the best practice during excuse me what's the best practice duration

first session are there any privacy consent requirements for recording video and audio okay three parts let me i found it in the chat so i was gonna say if you'd like me to repeat any part of that while you're answering happy to do that as well i appreciate that but i think i found it so um i i will say prototype phase is difficult because normally you have some very rough images that you can't interact with a screen reader with um so it's a more theoretical kind of conversation at that level and reliance on traditional accessibility consulting knowledge to guide a development team so i would say that usability studies with individuals with disabilities are a little later in the process there are usability studies that are done with tests or prototypes but in it's normally paper-based type of stuff or image based

um so that is one kind of area that you just kind of have to have more of a conversation with a participant if you're going to do that how many testing crypt you normally start with the initial phase again it's how much time do we have and where the developers can be influenced so if if you're just starting with a client we typically want to say that a minimum of four scenarios really give them a taste for what this would kind of be like and get them through and then you build from there but there are other clients that start off with no no there are 15 key things that we absolutely have to make sure are right and we're going to have the different teams all dedicated to lined up once you give us data we're going to go and fix it

and it's like okay well we'll we'll get through as much and then it's setting up those timings um so i think that covered the two the last one being privacy uh consent absolutely yes um so we have non-disclosure agreements with our clients and we have a partner agreement with lci that includes that non-disclosure we have the agreement between lci and us for being able to record those videos and share those with assets with our clients so that they can see the full experience or we can cut various video cut various videos down to a specific issue for them sound great john this is one this one's directed specifically to you do you have testers who can test with speech recognition technology yeah so we do we also have people who use um the dragon uh you know natural speak so we do have people who

use that as well and we do offer that as well great uh nikki karen that's all i'm showing right now that i see with the questions from chat um do you see anything from twitter or any in chat that i've missed or any on youtube nope i haven't seen anything else great we'll go ahead and just pause again thank you for your wonderful insights and thank you for taking the time to answer the questions and also for your wonderfully prepared presentation this is very insightful and helpful just to learn about usability testing with folks with disabilities what would you say are some of the biggest challenges in communicating discovered issues with developers um honestly once we get to having a user with disability and a recording of them struggling it's an easy conversation for developers when it's a issue that was identified

through an automated tester a testing tool or an accessibility consultant i mean i consider myself very very educated and very adapt but they're going to still look at it and go really is that true i'm going to try to argue right when you have an individual with disability that is truly struggling and it's so obvious they're very motivated to fix that um so really i haven't had a big challenge once it's identified by an individual with a disability and i think that's one of the most powerful components of this so with that john do you want to yeah yeah yeah for sure you know i think you know our hope from this presentation is that you know you leave here um seeing the value in including professionals with disabilities in your in the in the usability testing process and then also

uh some of the best practices that you may have learned from here you know i think one of the ones that that heather really pointed out was the importance of allowing um subjects to use their own technology and in their own uh environment of choice so i think those are in those remote environments so those are some of the key takeaways right there and then also there's organizations like lci and other organizations like ourselves across the country who i urge you guys to tap into to access the professionals and then also you know it's important to partner for this uh standard testing so hopefully you guys uh can take these away with you guys and uh we really appreciate you guys giving us this platform awesome i think we'll go ahead and close it out with this last

question we are almost at time for final thoughts um we have a last question do you have any tips for neurotypical people who to get used to testing with blind or low vision subjects john do you want to take that one or you want me to okay okay so honestly i i don't know if it's any different than any other usability subjects as far as i would encourage you to be honest about yourself as far as disclosing letting the participant know what is what what has worked well for you in the past what hasn't um what might be some areas of concern and and being very open um specifically if you're talking with somebody that is raising their hand and asking to help improve the accessibility of a product they're already eager and they want to be there to help

so if there is ways that they can help communicate that right then just be upfront and honest and have that dialogue i don't know if that was really helpful or not but that's how i would approach it i think that answer heather answers a lot of things right but it's just being as authentic and honest with others that we can do absolutely um just want to echo what everybody's already saying in chat this was a great session thank you both so much heather and john we really appreciate joining us and sharing your insights with us yeah i have truly enjoyed being here and getting to know the chicago alley meetup and while we're dealing with the corona 19 and everybody is changing the world right this will always be part of my memories with chrono19 and one of my big hopes is really that

as we're seeing more people work remotely that this is going to become more accustomed and that society is going to accept it more and then it's just going to make lives for individuals with disabilities so much more better so great thank you all appreciated thank you thank you all right everybody just a reminder we will go ahead and post a recording on youtube once we close this out we'll also be adding a transcript to this as well so thanks everybody for joining us from across the globe um thanks for joining us accessibility chai and again follow us on twitter have a good evening