Captions: Humans vs Artificial Intelligence: Who Wins?

So... Rather than just speaking to fellow captioners about what our profession might look like going forward, this talk is more addressed to users of captioning, and people who might be in the position to arrange for captioning, either for an event that they're hosting, or for an employee that they're hiring, or perhaps they work as a disability director of the university, and their responsibility involves coordinating captioning for students or employees or public events.

That's going to be more of my focus. I think, specifically, because there has been so much changing recently in the last six or seven years, in terms of accuracy of auto-captioning, in terms of the accessibility with respect to just being able to turn on your phone and push a button, the ability to get real-time captions from an automated source has gotten much easier. At this point, it is very affordable. We might talk a little later about whether the business model involves creating the market and then starting to charge for it.

I think right now, very few companies are making any money at providing automatic captioning, but we'll see what happens in the future when it comes to the actual market. I'm going to get started primarily addressing the differences, just the actual technical differences both in the production of human captions versus auto-captions and in the results and what each might be used for in which circumstances.

Just a little about myself. I have been a professional steno captioner since 2007, both here in New York City and remotely across the country, both on-site and through the internet. I've got my RDR, which means that I've been certified it up to 260 words per minute. I prefer doing really thick thorny medical and technical jargon.

My colleagues and I, at White Coat Captioning, do a lot of tech events, programming conferences. I also really like medical stuff, so I've done medical, pharmacy, dental, veterinary school. That stuff is really fun for me. I also founded something called the Open Steno project back in 2010, which was an attempt to break machines monography away from the proprietary locks upon it that had concentrated exclusively in the professional court reporting industry, because I really love steno as a technology, and I feel like it has so much potential to be used for so many more things than just professional court reporting.

The Open Steno project was an attempt to make steno more usable by amateurs, or honestly, anyone who spends a lot of time in front of a computer on a daily basis, by making the software free, the hardware less than $100-ish, and have a suite of fun, free gamified training materials to help people to learn this technology and increase the ergonomics, and the ease and fluency and speed of their text input.

This talk will first address the current state of live captioning by humans versus live captioning by automated systems. Then I'll go a little bit into speculation of how I think it might progress in the future and what I think the industries might look like overall, and how they might intersect.

If anyone here has seen the talk that I gave at Accessibility Camp, New York City, some years back, I addressed auto-captioning, but at that time, it was really, basically unusable. People referred to them as auto corruption, because YouTube was getting maybe a 70% to 80% accuracy rate, which is really three errors per sentence more or less, and people were having all sorts of fun making caption-fail videos. It was terrible. It's just legitimately terrible back in 2010, 2011-ish.

It has absolutely legitimately gotten better, that must be acknowledged. Throughout this presentation, I will acknowledge my own bias, as a human captioner is a professional captioner, who gets paid to do what I do, and who really enjoys my job and wants to keep doing it. I will make the case for why humans, I think, will always have a place in the real-time captioning industry, but it would be disingenuous not to acknowledge that to a significant degree, auto-captions have improved.

The question is, first of all, as they are right now, what situations are they appropriate for? Then the much more difficult to quantify question is, where are they going in the future? I do think that it's still an open question. I think a lot of companies are banking on auto-captions, basically just getting better than humans in all circumstances, and being cheaper and being easier to disseminate and being much easier to turn into a viable business stream, and they're just thinking that human captioning will no longer be necessary in, who knows, 5, 10, 15 years.

I think that's unlikely. Again, perhaps this is my bias speaking, but I think that might be a little bit over-optimistic. The question is really, what is the trajectory of auto-captioning? We're going to have to wait and see, but I'm going to talk about some of the reasons why human captioning has certain advantages over auto-captioning that I think are going to be very difficult to supplant.

First, let's address the indisputable advantages that auto-captioning has over human captioning. Scaling and commodification is, of course, the big one. Just, logistically, being able to press a button and instantly have an algorithm transcribe your speech for you. You don't have to coordinate with human.

You don't have to arrange payment. You don't have to vet and filter and figure out which captioners are good versus the ones that are mediocre. It's something that can be done 24 hours a day at basically any point in time, and for as long as you want, you know, humans don't humans get tired, they have to switch out they have to rest, they have to eat. Robots don't need to.

They won't make little random finger slips. They're not going to make what human captioners call misstrokes. Their output is going to be very consistent no matter what. Now, as I'll address a little later, the input is really what we're talking about, but let's leave that for a second.

Just the fact that you can have it accessible on your smartphone, on your laptop, anywhere that you're connected to the internet, is a huge, huge, undeniable advantage, and also a very powerful business model for companies that want to make money at real-time captioning.

These are more, the sort, mythical, or dubious advantages that have a lot of power in the narrative around auto-captioning, but I think it's important to confront the fact that they might not be entirely realistic. One thing that is pretty famous now is what's called the 90% equals 100% fallacy, which means that humans are really bad at understanding probabilities that are not either 1%, 50%, or 100%.

When you tell someone the accuracy of this captioning is 90%, it's really difficult for a human not to round that up to 100%, and be like, "Wow, that's practically perfect. I mean, where can you go from there?" The problem is that language is so granular. When you really think about it, in an average sentence of 10 words, 90% accuracy means you've got one wrong word per sentence.

That might not be a big deal depending on the word, but if it's the most important word in the sentence, it can be a very big deal. When you magnify that out, so you've got 10 wrong words per paragraph, it can suddenly escalate to a point where the captions might be either useless or worse than useless, and that's at 90% which some auto-captioning algorithms under some circumstances can get substantially higher than that; some get substantially lower than that.

90% accuracy is one wrong word per sentence. 99% accuracy is one wrong word per paragraph. 99.9% accuracy, which is what I try to hold myself to, if I get less than this, I feel like I'm having a bad day, is one wrong word or omission per four pages. Even though to the average person, 90, 99, and 99.9 might sound like rather similar numbers that you can sort of group together, they're actually not at all.

They're quite different when it comes to the actual accuracy of what you're trying to understand. There's also what I call the Star Trek Effect, which is that people have been talking to computers in an easy conversational manner for the last 50 years. They're always perfectly understood. The computer understands their commands, and their conversation, and whatever they want them to do, and just carries it out as a matter of course.

We are starting to sort of replicate that in real life. People are talking to their phones more. They're dictating text messages. They're instructing their smart speakers to play music.

I think people are also starting to realize that even a slight margin of error in these kind of conversations can be intensely frustrating, and sometimes to the point where certain people might feel like it's not worth it. I don't think that voice as an input method is quite the slam dunk that it necessarily was thought to be five years ago.

This is the one that always stymies me because I've been doing steno now for, what, 12, 13, 14 years, and people still come up to me at conferences saying, "I thought that that had to be an AI because it was so fast, and no human could possibly type that fast." Then I saw you were making corrections and fixing errors and I was like, "Hold on, is that a human? How is this possible?".

The thing that I think stenographers have not been yet been successful in carrying across is how powerful a technology steno actually is. It's about six times more efficient, more ergonomic, faster than QWERTY typing. It was invented in 1911, and it's still the fastest method of text entry that's ever been invented. I can basically write at 200 words a minute with a really high degree of accuracy.

I would say, at that speed, probably over 99.9% for most subject matter. I can do that eight hours a day without a break, and my hands don't even feel it. That's how powerful steno is. It's really not actually the input method that's the bottleneck for stenographers, like with the AIs, it's more a matter of being able to hear things correctly, being able to parse them and comprehend them, and then being able to filter them through our steno code system to get the correct output.

It's not the fingers that are actually working hard when a captioner is working; it's the brain.

Now, I'm going to talk a little bit about the advantages that well-qualified, well-trained human steno captioners have over AI captioning.

The main thing, the really crucial, crucial thing, probably the most important part of this presentation, is that transcription is always better when you understand what people are saying, which so far, computers just don't. They are matching very, very complicated and vast databases of probabilistic and fuzzy, logically interconnected words and phrases, and sound bites, and Markov chains, and stuff that I really have a fairly vague comprehension of, but they don't really understand what people are saying in the sense that they can't guess what someone's going to say next.

They can't use what someone has said in the last paragraph to inform what they're going to say in the next paragraph. They can't fix errors. When a computer has made an error, it has no idea that it's made an error. This is what I call the sniff test.

It can't run every single word that it is outputting through a filter of, "Does this makes sense? Is it likely that this person would've said this? Does this fit in with the rest of what they're saying? Is this contributing to the overall message?"

Because a human can do that in a really granular fashion with every single word that they choose to output, that's a tremendous advantage. I think people, even when I explain my stenographic system to them, assume that there's some sort of fuzzy logic, or autocorrect, or autocomplete, or something that the computer is doing that I'm not doing, but that's not true at all. Steno is 100% deterministic. The input that I give it will produce the same output every time with absolutely no variation.

It's really a matter of me understanding what the speaker is saying, and then deciding what that's supposed to look like in text. That's something that computers can't really do. They can match the sound to the letters, but they can't match the meaning to the meaning. There's also secondary considerations like punctuation which might not sound like a big deal, but can be surprisingly complicated and surprisingly important to ease of reading comprehension.

When people make up words, when people use unfamiliar technology, when Kelly with a Y, is sitting over there, and Kelli with an I is sitting over there, that's something that it's really difficult for a computer to get a handle on. In some circumstances, that's not a big deal, in others, it is.

This is a little bit of a pun, and I googled a bit to see if I was the one to coin this phrase, and didn't find anything. As I said, I personally don't really understand the difference between machine learning and deep learning.

Something that humans can do that computers can't as I've seen in reviewing the output of a lot of different auto-captioning algorithms is what I like to call steep learning, which is an inductive leap from non-comprehension to comprehension the farther you go along. When you start off, and someone is speaking to a subject matter perhaps that you don't have a lot of general ground in, you can take it in, and take it in, and take it in, and suddenly, it clicks, and the gestalt of what they're saying coalesces for you, and then you're able to vastly increase the accuracy of your output because you're understanding the story that they're telling even if you're not well-versed in the material itself.

That's something that's very difficult to teach a computer to do. These algorithms rely on vast, vast inputs of training, and it takes quite a long time for them to churn through that and to come to increasingly more and more accurate algorithms. I think the argument is always, well, they're getting better and better at the core, and soon, all that will be left are the edge cases. The thing is that language has a lot, a lot of edge cases.

I don't know if you've ever tried to see if a document that you're reading is online, and you just pick maybe a two or three-word phrase, and you're like, "Oh, that seems relatively unusual." Type that into Google, and it'll be the only hit for that document. The recombination of individual words in this language can be so multivarious and so unique in each particular circumstance that it's very difficult to map that probabilistically, and it's very difficult for an algorithm to get a better sense of what it's captioning over a single half-hour session, but a human can, and that's pretty important.

Now, of course, this could be an entire other lecture and I think I probably will give a lecture on this at some point, finding a captioner who's qualified, what you need to look for, what qualifications they need to have. There is, for sure, great variability in quality between captioners, partly because many captioners are trained as court reporters and their whole training is to get down phonetic approximations of what's being said in the moment, and then go back and edit and fix and clean up the transcript for later publication and real-time captioners don't do that.

Our product is the real-time feed and any transcript is just a by-product. You need to make sure that whoever is captioning is trained as an actual real-time captioner and not as a court reporter, but that's a side note. The fact is, that if you've got a really good, well-qualified captioner with a good sort of general knowledge about a variety of subjects, they're going to be able to produce high-quality captions in a very wide variety of circumstances.

Whereas algorithms, you can pick one, you can say, okay, this has got the best overall rating compared to the other handful of algorithms on the market, but when you start tweaking the circumstances, when you dip the audio quality down just a little bit, when you start to increase the amount of unfamiliar jargon, that comes maybe 5th or 6th in its list of possible matches rather than first or second, when you introduce an accent, suddenly that accuracy can dip in a really unpredictable way, and so you're never really going to know whether the algorithm is going to be performing at the top of its game in any particular conference or even speaker within a conference and that lack of predictability can cause real problems.

The fundamental issue is that sound is an imperfect method of communication. Humans misspeak, they don't articulate, there can be background noise or music, people pronounce words in a variety of ways, depending on where they're from. There are many homonyms used in English. There's so many different ways that language is a difficult thing to nail down into a simple binary output.

Humans are so good at doing this, at listening to this lossy fuzzy analog data stream and turning it into comprehensible ideas that they don't actually understand a lot of the subconscious processing that their brain is doing and they tend to think that the signal is a lot clearer than it is and that the noise is a lot less significant than it is. If you talk to someone who has hearing loss, or if you are a person with hearing loss, you'll know that there is a tremendous amount of cognitive fatigue involved in trying to take imperfect audio and synthesize it with your human semantic understanding of what the speaker is saying and come out with a clear and complete idea stream from that lossy audio stream.

It's something that captioners know because they do a lot of thinking on the job and people with hearing loss know because they do a lot of thinking throughout their lives, but people who haven't had to deal with either of those circumstances tend to just let it slide under the radar.

These next two slides are going to be just four or five, I guess, simple questions that you can ask yourself if you are tempted, use auto-captioning rather than human captioning for any given event, especially I think if you're talking about open captions at a public event. Can you assure yourself and your audience that all of your speakers will have standard American accents? Male speakers are preferred, turns out. A lot of these algorithms are trained to the baritone male voice and have some problems with female speakers and there is some work on getting non-American accents into the corpus, but that's definitely a work-in-progress that's lagging behind the American transcription.

Then do you have someone able to give the algorithm crystal clear audio? Can you assure yourself that all the speakers have good mic skills, that they will articulate that there won't be background noise or music and that the audio stream and the internet connectivity will be top-notch? Those are the two really big ones that if the answer is no to either of those, you absolutely should not use auto-captioning, but then there's also three questions about what are the stakes of failure?

How important is it that the people in your audience or the person who you're captioning for a job interview or the person sitting in their college lecture is going to get full and accurate text? If they miss one word because the algorithm is 99% correct, but that one word in the paragraph is wrong if that's the most important word in the paragraph, does that mean the student fails their test? Does that mean they don't get the job?

Sometimes yes, sometimes no. This is a question that you have to answer. What if you're getting really good speakers and really high-quality auto-captioning and then suddenly someone comes in with a slightly unorthodox accent and the accuracy goes from 99% down to 80%. What do you do?

Can you just toss their whole speech out and say that, well, the person got 8 out of 9 talks at the event, so that one talk that was pretty incomprehensible and confusing, doesn't really matter?

Is lag a factor, is the person using the captions, using them for interaction? Are they able to answer back to the speaker? Are they expected to answer questions? Are they expected to take notes, and then maybe complete a pop quiz at the end of the hour.

It's really important to consider that real-time interactivity is something that a human captioner sometimes has a real advantage of over auto-captioning.

I wanted to make my slides really clean and pretty, but I couldn't figure a way to make this less busy or less weird-looking than it was, but I thought it was important to put it up here, all the same. This is from the accessibility Camp NYC that we did some years ago that I spoke at. This was a great guy named this Disabled Foodie. I love him. I loved his presentation.

THOMAS: David Freidman.

MIRABAI: David Freidman, yes. Thank you very much. He was talking about wheelchair accessibility in New York City restaurants, and he's a lovely, charming guy and for the record though, he does have cerebral palsy. It does not affect his speech and I would say he has your completely standard American accent with good articulation and a baritone voice for what it's worth.

If you can see the video it's a little small, I'm not going to play the video. I actually use two different versions of auto-captioning. One that I used the YouTube algorithm for about 3 or 4 months ago, and then one that I just did today and it makes different errors but about the same amount of errors. You can just see at a glance that there are some real problems here, 81% accuracy.

Now it was not 81% accuracy throughout his talk. I did admittedly cherry-pick this particular paragraph because for one thing, I thought that the hilarity of replacing special education with sexual education pointed out how auto-captions can sometimes be not just confusing, but outright embarrassing and even potentially a liability issue. If you're displaying them to all in sundry and you are considering that they're representing your organization and a human captioner who got I'm a special education teacher, correct, as did the auto-captioning earlier would know that he was not saying sexual education later in the paragraph, that's where semantic understanding comes in.

A human captioner would also probably have some knowledge about the ADA and the IDEA if they've worked in accessibility before. Hopefully, if they're a New Yorker, they would also know what Jewish kvetching was, and not transcribe it as either huge fetching or I think like fusion etching was like the newer version, I don't even remember. It was ridiculous, but it did not translate correctly.

I've got about 10 or 15 minutes left and this is where I'm going to shift into speculation about how the industry might look in the future. I think it's pretty undeniable that the more auto-captioning becomes ubiquitous on smartphones and in public places and on tablets and basically everywhere, the more people start using it, the more they will want to use it. I think that's already happened with offline captioning. I know a lot of people of my generation with no hearing loss who have the captions on all the time when they're watching TV because it's useful for multi-modal input being able to read something and listen to it at the same time can actually be more enjoyable as well as more comprehensible and less fatiguing, even if you have no hearing loss, whatever, but also people are going

to find out how useful even bad captioning is. I have a feeling that that will create a demand for good captioning.

Plus, and this is pretty significant. The babyboomers are coming into their senior years, especially since many of them have gone to rock concerts in their youth. As many of them are staying in the workforce longer and are wanting to participate in public life longer than perhaps the previous generation did and as there are just so dang many of them. I really think that as age-related hearing loss, has more and more of an effect on this particular generation, which also I think are more comfortable with technology in general and with computers and smartphones in particular than the previous generation.

I think that that is going to be a push from them to have more ubiquitous, always-on, not on-demand, but just proactive captioning for public events and in public spaces.

This is the question, what about the economic issue?

One consideration is that there are just far too few steno captioners to provide good quality auto-captioning to everyone who needs it. I think there are around 45 million people with some degree of hearing loss now, and that's going to go up by at least 10 million in the next 10 years. There are 400 certified real-time captioners in this country and far few were in the rest of the world. The disparity is just staggering, even if I was a robot and could work 24 hours a day, there's no way that I would be able to caption for everyone who wanted or needed captioning.

I am also expensive. I'm going to be more expensive than an algorithm, no matter what I need to feed my family. [chuckles] There isn't necessarily the money available to pay a human captioner for absolutely every circumstance and for every human being who might want captioning. There's also the possibility that the current trend of more places offering open captioning to anyone who might be able to joy it and appreciate it, but not necessarily self-identify as having a hearing loss or not necessarily knowing their rights under the ADA, that they have the ability to request captioning.

I see a trend of companies and events closing the captioning as much as possible saying, "Sure, you can have it, but only on your smartphone. We're not going to put it on the big screen. Sure, you can have it, but only on request, only on demand. We're not going to offer it proactively."

I can absolutely understand the reasons for them doing that, but I think it's very detrimental, not just for me as a professional caper who wants my product out there and available to anyone who can use it, but also just in terms of spreading the word about how important and useful this technology is for a large number of people and not just people who require communication access in all circumstances. People with profound to severe hearing loss, many of whom use sign language to communicate with their nearest and dearest and interpreters to communicate with people who don't sign, they know their rights under the ADA, and they ask for accommodations whenever they need them.

People who have a hearing loss, that's more in the mild to moderate range, especially if it's acquired later in life after they already learned their primary language they might not need it for a conversation between friends in a quiet cafe, but they might absolutely need it for a public event in a darkened auditorium when they're 20 feet away from the speaker and can't read their lips. I feel like open captioning is really going to be important to provide that widespread broadband access to as many people as possible.

1 in 15 people have some degree of hearing loss. If you're talking about people over 65, it's 1 in 7. If companies don't want their auto-captioning algorithm saying sexual education instead of special education and making them a laughing stock for the entire internet, they might have the tendency to try and close that captioning down and not put it on public display. That's something that I think we need to be aware of that trend, and something that many accessibility advocates and professionals will want to push back against.

There's also an even more dire and scary possibility, which is that legislatively, the ADA could just be cut out from under us. I'm not going to talk about it too much, but it's nothing we should take for granted. We always need to stand with disabled and not disabled ... deaf and hearing and hard of hearing alike.

We all need to be aware of what's going on politically in this country and we need to be very loud in our advocacy for ourselves and our fellow people. There's also something that might be good for me, but not necessarily so good for deaf and heart of hearing people. Though, who knows, perhaps if auto-captioning gets good enough, this stratification might make sense where it's something a little bit like the era demonstration that we saw some months ago where blind and low vision people can have automatic image recognition on their glasses that will help them to read medicine labels and street signs and whatever but if they're in a space where they need navigation tips, or they need to really appreciate something visual about an event that they're experiencing, they have a certain number of hours included in the service that includes a live human guide

describing the scene to them. I could see that two-level system happening with an auto-captioning service too, where someone, for some circumstances, they're fine with auto-captions, but then if the stakes get higher, or if suddenly the accuracy tanks, they click the switch that puts them to a standby human captioner, who's then able to take over and increase the accuracy. I personally wouldn't mind working in that circumstance, but I do worry that if the model becomes, you get human captions if you pay for them, but you get auto-captions if you can't.

That's a real problem where the haves get good captioning and the have nots don't, so that could be a good or a bad thing, depending on how it shakes out in practice. Another thing that I've seen again working in its way through the current trends of the industry, companies saying, "You don't really need live captioning. Just record the audio and then send out little 30-second snippets to like 4 million people on Mechanical Turk and then they'll either correct an auto transcript or transcribe it themselves from scratch.

Then you stitch all those pieces together and boom, voila, you've got a beautiful low-cost transcript." They get paid 5 cents a minute or whatever. Then you've got this transcript for cheap in an hour or two hours, but that's not real-time communication access and there are also quality control issues with that modality as well. Not as an ideal solution, but again, it's scalable and it's a really tasty business model for a certain type of company.

It's something that needs to be considered in how everything's going to fit together going forward. Or some companies might leap feet first into the whole algorithm business and enjoy it for a while and then that first big crucial moment comes and the auto-captioning fails and they realize what it's like to be working without a net and they go back to human captioning. This could happen, I think it's quite likely honestly. I also see the possibility of human and auto-captioning working to some degree together.

There are certain unfamiliar terms that I might be like, "I wish I could stop time for two seconds so I could Google that. I know I'd be able to put it in my steno dictionary if I could." If I could have a caption stream, just running as a byline on my computer monitor and be able to see what it's able to access in its vast database of words and terms and just use it for reference in producing my proactive word by word transcript.

I could see that becoming really useful. I could even see a hybrid system where a steno captioner is standing by reviewing the auto-captions. Once they dip below a certain level of accuracy, the steno captioner can jump in and save the day. That's a little bit riskier I think, for a number of reasons, but it's possible that there can be a good fusion of the two that can be to the benefit of everyone.

Especially like I said to deaf and hard of hearing people who are just not going to have full coverage if they only ever rely on human captioners for every possible thing unless we're able to get a lot more human captioners, which I'm working on, but [chuckles] it's a work-in-progress. Basically, I think the fundamental issue is that you trying to repair a broken thing after the fact is always trickier than being able to get it right the first time, and then maybe clean it up a little.

I can see if my transcript is almost perfect, but I dropped four words in there. The auto-captioning algorithm can scan through it, detect four missing words. I can quickly jump there, spot-check it, fill in the words, and have a 100% accurate transcript within five minutes of the event. I think economically and also in terms of usability, that's really powerful way of doing things rather than the current paradigm where you get the auto-captioning filling in a bunch of mistakes and messiness, and then throw a swarm of low-paid humans at it to fix those up.

It's much less efficient. I think ultimately, even if it might be cheaper in some ways that you figure it, it's not necessarily going to be as workable in the long run. That's basically what I've got to say. I've got a ton of thoughts on this subject and I would be very happy to talk to anyone about this.

Basically, you start me going and I'll never shut up. This was my attempt to boil down the most important and most salient points of how humans just work differently than auto-captioning algorithms and what advantages we are able to offer that so far. Who knows? Maybe the singularity is coming tomorrow.

Who knows? So far, we can do things that auto-captioning can't and that's my argument for why we're still relevant and useful in this day and age and hopefully, in the future going forward. Thank you so much. [applause] Firstly, thank you very much.

This was awesome as someone who relies on captions a lot of the time, it's cool to see behind the scenes. Something that you brought up about American accents got me wondering about how to make it easier for human captioners that have to deal with content in more than one language or where there's switchover. For example, I go to a lot of Jewish events where the speakers will just say things in untranslated Hebrew, or refer to things where the translation in English is really sticky and was wondering how you deal with that and how you might see technology working with that or not working with that?

MIRABAI: Yes, that's a fascinating question. I have to confess my parents speak six languages between them and I only speak English. It is the greatest regret of my life that they didn't teach me any of their other languages. There are a few working multilingual captioners and some of them have worked out stenographic systems to be able to switch on the fly.

The open-source software that I commissioned and helped to develop is able to support language switching and dictionary switching. This is technologically something that's possible to have, but of course, you need someone who has fluency in multiple languages plus stenographic ability and not all languages have as developed or coherent stenographic theory as others. There's always the question of automatic translation, which of course has many problems and I think we'll continue to have many problems going forward.

You can make the argument that if you're going to use one or the other, you might want to get the English stream perfectly accurate done by a human, and then let the algorithm translate it into another language. If you're talking about the same person translating two languages with imperfect knowledge of one or the other, that's a hard question and it's not one that we've come up with a good solution for.

PARTICIPANT 1: Thanks.

THOMAS: We had a couple questions on the live stream, so I'll do those and then keep going around the room here. The question from Travis Hopkins, "I was curious around how often do you add new words to your dictionaries and how often do you run into words that don't work in steno if ever?

MIRABAI: Like I said, I've been doing this for about 12 years now professionally, and in the beginning, for about 12 years now professionally, and in the beginning, I would add hundreds a day, honestly. These days, depending on what I'm doing, in terms of words that just aren't in there at all, it's probably fewer than five a day. Sometimes, even fewer than that. Often I find myself adding stock phrases because I don't necessarily want to type area under the curve in four strokes when I could do it in one.

I have found myself making my steno dictionary more efficient in terms of using more phrases, but I don't tend to add a lot of extra definitions these days unless I'm working in a subject that I've never captioned before. The question was whether I encounter stuff that just is not possible at all to render into steno. One thing that not everybody knows about steno is that it's not a strictly phonetic system. It's a phonetic mnemonic system which means that some entries and definitions are entirely phonetic, a fair number, but then a bunch of them are just idiosyncratic code words that the individual stenographer makes up.

A good portion of learning how to do steno well and efficiently is learning your own little inner language, your own mnemonic hooks, the ability to take a word and crush it into a portmanteau or a memorable piece of gobbledygook that you're going to remember the next time it comes up and you'll be like, "How did I write that? Oh, like this." A lot of it is knowing your own individual brain, and that's why each stenographer has their own dictionary that has some overlap with other dictionaries, but the more you use it and the farther you get into it, the more it's actually just a reflection of your own brain's inner mnemonic structure.

There's nothing that you can't turn into steno because it's just you're making up your own language.

THOMAS: Awesome. Our second question from the stream from Adam Goodkind, "Hey, Mirabai, good to hear from you again as a regular--

MIRABAI: Hello. I love you.

THOMAS: Live reaction. [laughter] As a regular CART user, the most inconvenient part is having to look at a screen rather than making eye contact with the speaker. Is there any progress being made on that front like Google Glass?

MIRABAI: I wish. That's something that I like to address in the whole demand for captioning skyrockets slide because smartphones are ubiquitous, but they are annoying to have to take them out of your pocket and there's battery issues and stuff. When everyone is walking around with head-up displays that look like regular eyeglasses and aren't dorky like Google Glass was. I did buy glass.

It was not practical for captioning and I didn't want to wear it in public because it made me look like a tool. [laughter] It also just did not work very well and was ridiculously expensive, but it is absolutely my dream that someday everyone will have screens in front of their eyes and it won't be a big deal, and then that's when captioning is just going to be everywhere.

THOMAS: Didn't want to look like a glass hole. I just wanted to take a chance to use it.

MIRABAI: Exactly. I didn't want to use the word, but yes [chuckles]--

THOMAS: Well, I did.

MIRABAI: It's a technical term.

THOMAS: [inaudible] Joly?

JOLY: I probably use AI captioning more than anybody else here. I use it quite a lot. I use Web Captioner which uses the Google translate API. People are very impressed by it.

One of the things that I like about it when I'm doing it on the streams is that it's very real-time. It just pours as the words come out of people's mouth. Even the best human captioner, there's going to be a little bit of delay and that's the one plus it has. I'm gaining more experienced using Web Captioner.

I can replace words. I can set it for a certain accents and I find that accents are the main inhibitor and I've learned the hard way to the same lessons that you put in there, that good audio and people speaking in standard accents is really what's most important, but generally, there is no budget for the human captioner, so the choice is between no captions or AI.

MIRABAI: It's a hard choice. I've spoken to several deaf friends and colleagues about how they feel. Some of them are like, "Yes, I do want to be able to use auto-captions. I don't want to have to haggle with a stenographer every time I need access, and not primarily for buying a coffee at the coffee shop situation which you don't really need captioning for, but for conversations, for meetings, for phone calls.

I think the issue though is that if people who don't rely on captions get the idea that auto-captions can work everywhere for every circumstance and that paying for captions is somehow a fool's game, that's what my deaf friends and colleagues are very concerned about because they feel like if that becomes the standard procedure, then they won't be able to convince their employers to pay for human captioning for important business meetings or for university classes or things like that.

It's a really fine line to walk. Again, it's not something that I necessarily should be the largest voice in because I'm not a caption user. I am speaking to some degree out of self-interest. I want to do my job and I want to be compensated for it.

I do worry about people deciding that this level of captions is good enough, especially as you say, it's a novelty for sure when people can just press a button and see, "Oh my God, it's getting so much right." If you're not relying on the captions, what it gets wrong might not matter to you, because you can hear what it was supposed to be, or it might just be funny or amusing. I think there are a lot of different ways that the intersection of auto-captioning and human captioning could go going forward, and I can see a pretty good middle ground where there are uses for each, but it's going to be interesting to see whether one will supplant the other.

JOLY: What I see is a real function for it and the way I do it is that people are multitasking, and they have the thing on the screen, but they're not listening, and they can follow and see if there's something going on that they want to put the sound up. Especially, I'm running stuff out on Periscope a lot and the captions come up, that's where AI really works for that. People just want to have a general clue what's going on.

MIRABAI: I agree. I actually use it myself for that purpose. If I'm trying to get a gist of the video, I'm going to be like, "Oh, that's probably an error, but I think I know what they're saying. Maybe, I'll listen to the audio later."

Also for search, it can be useful to have an auto-captioning corpus for search. It's definitely going to become more prominent, no matter what. It has made advances that I honestly did not think that would be able to make and I have to give credit to the companies that have been producing these algorithms for that. I think we just have to see what level of accuracy in which set of circumstances it's going to be able to attain in the future.

JOLY: In terms of the hybrid things, I agree with you that it's fixing something that's broken. It's more easier just do it over and do it properly. It seems to me that it's very easy to spot those errors when they're happening and that if they built into their systems, the fact that you could touch screen, anything that was wrong and crowdsourcing it even if you've got an audience of 20 people. It would be pretty easy to then produce something fairly accurate transcript out of something,

MIRABAI: I think that's true up to a certain threshold, but then when the errors multiplies so that they're just all over the place, that's when you have to scramble if you don't have a backup system in place to deal with that.

JOLY: My problem is this thing of, you can see something wrong and you can fix it so that it's fixed next time, but it's very difficult. If it's something that's common, I'm sure you have this in your dictionary too that if it's something where you know that if you put in this correction, it's going to screw up when that word comes up the next time. It's a very difficult thing to do. I think that's all I've got for now.

MIRABAI: Thank you, Joly. I actually really appreciate your input because I personally have not used a lot of auto-captioning in live situations. I've seen some after-the-fact auto-captioning, but it's useful to hear your point of view on that.

JOLY: Can I say that if you go to my live stream channel, which is livestream.com Internet Society, you'll see a whole bunch of events, and you can look that where I've used it, good and bad [chuckles].

MIRABAI: That's definitely useful to have.

SHAWN: This is Shawn. For those who don't know me, full disclosure. I work at Google, and my team that I was on for a number of years made voice typing in Docs as well as more recently auto-captions and slides. That said, having you do the captioning for our meetup and the other folks at White Coat Captioning has turned me into an absolute captioning snob [chuckles].

The question that I'm eventually leading to is, I pay a great deal of attention to captions when I go to events now. By and large, most of them are not great. Some, I can see why. Some are the systems where it's using predictive typing.

It's people typing on QWERTY keyboards, and it's using corrections, and then they're selecting that which leads to just massive amounts of emissions, massive amounts of errors. I'm also seeing people who are using steno, but they're just not very good at it.

MIRABAI: That's a very important issue to acknowledge for sure.

SHAWN: That's one of the biggest challenges that I see in making the argument for live captioning with the human being is that most people are exposed to lousy captioners. When the error rate of the average human that they see is worse than the auto-captions, it's really hard to make that argument. I wanted to get your take on that and your vision of how we can fix that.

MIRABAI: I could not possibly agree with you more. My stock phrase with respect to this is that my best competitors are my best friends, and my worst competitors are my worst enemies because people who try to do this job and do it poorly poison the field for all the rest of us. People who do it well, there's plenty of work to go around. I'm not worried about people poaching my work.

I just want them to increase the prominence of captioning and increase the demand for captioning, increase the standards that people hold captioners to so that the really top good quality captioners can get as much work as they want, and people who need captioning get good captions. I absolutely agree with you. As I said, there's only 400 of us with the certification and the certification is 96% accuracy which is not good at 180 words per minute, which is quite slow.

There needs to be higher standards, and there need to be more good captioners. I'm trying to make them with the Open Steno Project. By the way, this slide, @stenoknight is my main Twitter account, @plover is my Open Steno specific Twitter account. The Open Steno Project is trying to get more people to use Steno for whatever they want in the hopes that a few of them will catch fire and discover that they love it, and rise to the top and become highly qualified professionals, but it's a slow process, and we need more exposure.

Please get the word out however you can.

SHAWN: Props to the project. I downloaded it and installed it on my laptop. It was free. It was super easy.

I got a 3D printed case for a steno machine, and I soldered it together and it was a lot of fun. You should try it too.

MIRABAI: Sweet [chuckles].

SHAWN: Other questions?

JOLY: Oh, this is something I've always wondered about. What happens when you get an itch when you're--

MIRABAI: [laughs] Funnily enough, about a dozen or so of the most common words in the English language can be done with just the left hand. You just wait for one of those words to come up and you type out that and you scratch with the other hand, and you're solid.

JOLY: Seriously though, do you think this, eventually if it becomes common, might affect how people speak that people will learn to speak more slowly recognizing that there are stenographers? I've seen this happen. I do global conferences, IGF conferences, where they have translators and people learn to speak slowly and firmly so that translators and so on, do you think that might happen?

MIRABAI: It's possible and that would be great though. Again, to the mic skills question, I think it's hard to teach to every individual that you're going to need to hear from, but in the aggregate, I hope so.

JOLY: There's a feedback loop?

MIRABAI: Yes. Some people are just not going to care or not going to be able to. I think I was fairly successful in modulating my natural rate of speech. Lindsay, did I do okay?

It took a conscious effort to do so. Anyway [chuckles].

JOLY: Your accent is [crosstalk]--

MIRABAI: Sorry, Lindsay. Thanks.

THOMAS: Adding on to being aware and actually totally making sense, this idea that a human can learn in real-time in the domain of what your particular presentation is about, I'm curious for etiquette and efficiency as far as for someone, that's myself, to be aware that it's a human captioner and communicating, if you saw a mistake, any points or practices on just pointing that out quickly and getting that.

MIRABAI: That almost never happens, but it's absolutely welcome when it does. If you say an unfamiliar word that gets mistranslated, giving the spelling of that word would be fantastic. I can't think of many people who do that but please, more of that.

SHAWN: I usually keep half an eye on it, just because my name is Shawn, and that spelled in five ways that I know. I usually keep an eye on things like this and say that mine is the way that's S-H-A-W-N as opposed to your S-E-A-N.

MIRABAI: Indeed. That's very helpful.

PARTICIPANT 1: To add to that, something that's been done in Jewish communities recently when you request CART or an ASL interpreter is to actually provide a list of terms that might happen. We've been talking about doing it in CART in a Facebook group I'm in, but I know that when we've had ASL interpreter at Jewish events, I've actually occasionally either given them translations or hovered in the background and fingerspelled the Hebrew terms and transliteration for the person using it which is not the most efficient way to do it.

I don't know if that's something you'd like to see more of or not.

MIRABAI: Absolutely. When we can get stuff in advance whether it's getting PowerPoints or someone just coming up to us like I did the Polyglot Conference some years ago and a guy came up to me and said, "Hey, the first sentence of my talk is going to be, welcome to the Polyglot Conference. I'm going to speak to you about Yiddish in New York, in Esperanto." He wrote down the Esperanto sentence that translated to what he just said and I put it in my dictionary.

He had it and I was so happy that I was just able to like voom. It all came up perfectly. I was like, "Yes." I still have that in my dictionary.

He's fabulous. Anything you can give to us, very, very helpful for sure.

THOMAS: I think this is probably related to that. Travis Hopkins had a second question on tips for getting good audio for machine translation and if you hire real-time steno captioner, best ways to accommodate for them?

MIRABAI: I'm not an audio tech so I don't think I could speak to the details of that, but just as clear and clean as possible, please. Joly might know more than I would.

JOLY: People should understand cardioid mic [inaudible]

THOMAS: People should-- Cardioid mics.

MIRABAI: Cardioid mics? I don't even know what that means. Sorry, not my field of expertise.

JOLY: The mic that you're speaking into is a cardioid mic which means that it picks up in this kind of-- I don't know why it's called cardioid, but it picks up directly in front of it for a limited area. If you're holding a mic like this, it'll just pick you up and nothing else. A lot of people you talk about mic technique, you see people who stand like this and they hold the mic like this, especially when they have speakers in the louds in the ceiling, and then it's the bane of my life.

Crestrons, I hate them. [laughter]

MIRABAI: If you're in that situation, a human captioner is usually going to do a better job than an auto-captioner in dealing with that. Not that we like it, not that we're happy about it.

JOLY: We've had people here who've been speech impaired that you've done and blowed my mind, Mirabai.

MIRABAI: Thank you.

PARTICIPANT 2: Hi.

MIRABAI: Hi.

PARTICIPANT 2: This is fascinating. I learned so much. I'm just wondering, has there been any study done coming from a user's point of view, how they feel about the auto-captioning and human captioning either from the audience or from some larger data?

MIRABAI: Nothing formal that I'm aware of, though I would love to read about any research that someone could find on that subject. As I said, in preparation for this talk, I've just been asking a half dozen or so with my deaf acquaintances. The consensus has basically been, I'd love to be able to use auto-captioning when practical, but not if that means that I'll never be able to have human captioning paid for by employers or organizers of events that I attend which seems entirely reasonable to me.

JOLY: I'd like to give a plug for the dynamic coalition on accessibility and disabilities. It's part of the ITU. It's part of the UN. I posted something to them which I'd auto-captioned.

I got grief back because the default on Web Captioner is capital letters. They pointed me to stuff which says lowercase is easier. I found that there is actually a dyslexic font in Web Captioner, which I've now switched over to. Do you have any comments about why.

The default on television is all capital letters. Why don't they change do you think?

MIRABAI: With respect to the dyslexic font, my understanding is the advantages of that have not actually been proven by research. I've heard people saying this.

JOLY: I can believe that.

MIRABAI: It's less advantageous than it was originally purported to be though. Again, not my field of expertise. In terms of all-caps versus mixed-case, I would always recommend mixed case under every circumstance. All-caps, I think are very tiring to the eye and less comprehensible.

JOLY: I like them because they show up better on Periscope when they're small. You know what I mean?

MIRABAI: Maybe.

JOLY: I've gone to the bigger wisdom and switched to the lowercase.

JOLY: I've gone to the bigger wisdom and switched to the lowercase.

MIRABAI: I'm sure that there are different preferences among different caption users, but by and large, I think people prefer mixed-case.

PARTICIPANT 1: Just in response to the fonts thing, the research is largely just use Sans-Serif fonts that are straight lines, but there's not the specific dyslexic fonts have themselves not been proven.

MIRABAI: That's what I heard.

PARTICIPANT 1: This font is fine. I think that's Verdana.

MIRABAI: It's Verdana. I usually use Inconsolata, but it depends.

MIRABAI: If I can make one more comment about something you can do to help captioners. Know that when you read something, you usually increase your speed by 30% to 40% versus when you're speaking off the cuff. It can be very difficult to caption material that's being read aloud as opposed to just spoken conversationally. It's very useful for captioners to have that material in front of them if possible and for people reading pre-written material to consciously slow down their rate of speech.

SHAWN: Thank you. On that note, thank you for that awesome talk.

MIRABAI: Thank you. [applause]