Accessibility NYC Meetup Recap: Inclusive Web-based Virtual Reality

Great! Well, welcome, everyone, to Accessibility New York City. This is our August Meetup. We do the meetup every month.

The first Tuesday of the month. And always having really broad, diverse topics. First off, I want to thank our event hosts, Tyson and Stephanie here from Thoughtbot. They are our host.

Thoughtbot is where we have our meetup every month, and we really appreciate having this space to be able to do these presentations. On the captioning up here, on the screen, this is being done live, remotely, by Mirabai Knight, from White Coat Captioning, and we appreciate every time we have our events, we're always gonna have communication access, realtime text translation, so you can count on that for the events that we're doing. And we would also like to acknowledge and thank Joly MacFie from the Internet Society of New York.

That's how we live stream these presentations, so for people who aren't able to attend in person, they can participate over the stream, and also all of the streams of our events are archived on YouTube. So if you want to go back and look at previous ones, they're all archived and available. Without further ado, I'm very excited about this presentation. I'm always into the latest technology, and Roland and Ian are very advanced in this space of thinking about these new technologies.

And I think there will be a lot of great discussions here. As you might see at the front of the room, there's a lot of different devices here too. So I think at the end of the presentation, there will be some time to discuss and maybe do hands on. With that, Roland, take it away.

Thank you! Yes, hello. Thank you, Thomas, and thank you to Equal Entry and to Thoughtbot for hosting us this time. I'm going to present to you accessibility for web-based virtual reality.

It's going to be a two-part presentation. One is more about the theory and about the space of virtual reality, and the second part is what we're doing in the web-based virtual reality space, and where we can actually find some interesting angles to make this a more accessible space. First off, I'm Roland DuBois, a UI/UX developer, and over there is Ian Petrarca, a sound engineer who works with me on the project. Very talented.

First I would like to see some hands. Who knows what virtual reality is and has used it in the past? Hands?

That's a two-part question.

So who knows about virtual reality in general? Okay. Now everyone that has not used virtual reality yet? Anything?

Anyone? Okay. And who knows what web-based virtual reality is? Okay.

A few. I need to know, because I'm going to talk tech, so I need to elaborate. So I need to know how deep I can go. Okay, cool.

Well, let's start. When you think about virtual reality, the next thing -- what you're seeing on YouTube is probably something like that. And I'm gonna play that video. Oops.

Does it work? Where is my mouse here? Yeah. It's a clip.

Right? Small. It's in South Africa, but it doesn't matter. (inaudible) (screaming) (laughter) So we're seeing an elderly woman, let's say, in her mid-50s, 60s, in South Africa, sitting in a lawn chair in the garden, watching something on a VR headset, and she's going crazy.

She doesn't understand what's going on. And everyone makes fun of her, and that's how it is. Right? Virtual reality is pretty much entertainment for others.

When you're immersed. But... This is a bigger topic, though. Because virtual reality being used to entertain us is just one part.

And if you think about the medium of immersion and the medium of virtual reality, augmented reality, mixed reality, I would like to go through a few... Through the different kinds of... I have to find my mouse again. Sorry.

Yeah. I would like to show you the spectrum of what we have, between the digital world and then also the computer- generated world. On the left... On the right you have the virtual reality, which is completely built and constructed in three- dimensional space.

And then piece by piece you add real elements to it, and you go to augmented virtual reality, augmented reality, where you place a layer of digital representation on top of the real world. Mostly with AR glasses that are basically like see-through glasses, so you can see something augmented on it. And then you have the mixed reality. Which is integrating virtual elements into a real space, with the knowledge of omission.

Which means you can hide something behind a table. The virtual reality glasses understands the space you place the objects in, and can crop out objects to make them immersed in the space. And understand the light and the shadows of the space and make them mixed reality. So just to give you a little overview of where the spectrum of immersion plays in, we're going to focus on virtual reality in this talk.

But it's very important to kind of get the vernacular right. So when you look at the industry, the virtual reality industry, especially in the US, is a big crowd of companies popping up. Most of them are focused, obviously, around tech. About the headsets that you're wearing.

And input devices. Wearables. How to make things more immersive. How you trick senses into believing that you are immersed in VR space.

But if you see the pie chart on the top left, it shows you the market share right now. And you see a clear focus of light blue. That's the HP Vive, the 360 dimension headsets. Does anybody know what the difference between 3 dimensions of freedom?

3 dimensions of freedom means you have rotation information on the X, Y, and Z axes of the head. If I rotate like an airplane, I can move up, down, left, right, and side to side. But the position is fixed. It's only 3 dimensions of freedom.

Three pieces of information that give me the rotation of the space. The eyes, the head are centered to the scene. And that's what you get. Most of you probably have seen 360-degree photos.

Those are made with 360 cameras, and they're inherently 360 -- inherently three dimensions of freedom because the position of the camera is fixed in space. Six dimensions of freedom means you add position tracking to it. Not only XYZ rotation, but XYZ position. If you are immersed in a 6 dimensional freedom space and you go to a virtual apartment, you can look behind the table, beneath the couch, beneath the chair, because you can change the position of the camera in virtual space.

And the focus of the industry right now is very much on those contraptions. They have light towers. They have sensors in the corner of the room to create a space where your head set is tracked in, and therefore it will understand what your human, eye level position from the ground is, and kind of gives you accurate translation of where you are and where are you going. Also where your controllers are.

Your hands and your feet and everything that you're tracking with your input and with your variables. However, that in the chart on the right is changing. And obviously every new technology comes up with the hardware first and then tries to sell that medium, that hardware, with content. However, it's all a bit backwards, because without the content, the hardware is nothing.

And therefore you see here the green pie and the yellow pie towards the year 2021. That's a projection chart -- will be mostly focused around the content. And so the VR headset market and the device market is kind of fading, because you need to sell the hardware with the content in it. So that's just a nice, interesting observation that we see in all different kinds of markets.

It doesn't have to be VR. It's actually everywhere like that. So here's a little overview of what the VR headsets that we're seeing right now in the market. Some of them are in China only.

But most of them are available in the US too. On the top left, you have the HTC Vive, the top headsets, and on the bottom right, you have the lower end of the VR headsets, where you can use your cell phone as a medium to immerse yourself in virtual space, thanks to the big screen of the phone that you split into a stereoscopic viewer, and the gyrometer of the phone understanding the rotational space, making a VR headset of your phone just by Cardboard.

That's very cheap. And then there's the transition, between the phones that are a little bit enhanced with Gear VR headset with the remote that is attached to it, these two, the Gear VR and the Day Dream, those are cell phone-powered VR headsets that are a notch better than the Cardboard, because they have specific technology that's added to it, with a VR headset, or just a certain level of phone power. Yeah. And like I said before, I'm seeing a lot of history repeating.

It's always the same story, basically. If we remember back in 2007, Steve Jobs announced the iPhone, and then we had to wait two years until it turned finally accessible. So all the people that had vision disabilities or physical, motor disabilities had to resort to the Blackberry and their tactile keyboards, or flip phones. And it just wasn't there.

But you can see that the direction goes from the early adopters, the people that really buy the devices, they're mostly focused on the hardware. And then it takes a little while until everyone is considered. And I also see a little bit of a browser war style going on, where the market share is under attack, by the big players. Obviously you have the different VR headset makers that are fighting for the space.

They want to make sure that their software is being used exclusively. And they're trying to bring in... They're buying up companies to produce content for them. So there's a little bit of that going on.

And then also clearly console wars, because right now, VR is mostly entertainment. And mostly gaming. So if you think of all these different game controllers -- I can zoom in here -- you can see none of the actual interfaces are comparable or understandable. They're not the same.

So you couldn't switch between NES and the Playstation. It would be a different kind of user interface. All that changed later on with network gaming, but we don't go down this web at all. And understandably, like I said, the focus is not to make it work for everyone.

The focus is on making the technology work. This tweet is a little bit -- it's from 2015. It's a little older. But it's John Carmack, the CTO of Oculus, which basically said: Stick yaw control is nothing.

We don't need it. We want to make sure we have the VR so that 6DOF VR is high latency and potent. But stick-yaw VR is basically the synonym for inclusiveness, in my opinion. And there's a lot of talk about the idea of growing VR legs, of people having issues with latency, and getting seasick because of using VR headsets, and it's a big misunderstanding.

There's always a lot of discussions on Reddit going on, about the idea that if you use it, just use it as much as you can, in order to get used to it. There's no real understanding about vestibular disorders or the fact this 35% over the age of 40 actually have that in the US, and it's not something you can train away. There's not that awareness, because it's not being tested on people in all different age groups.

Mostly tested towards target groups that are going to buy the devices. Yeah, here's the example of the VR controllers that we have right now, the 6DOF devices. They're pretty much all fancy-looking 6DOF tracking devices which assume you have ten fingers and both hands and full range of motion and can jump around the room and are a happy fella. Yeah.

There was an article in Able Gamers that I read a while back, and I'm linking to it at the end of the talk. There was -- talking about the key points which make VR inherently inaccessible. So I already talked about some of them. I'm gonna fly through them really quickly.

The emphasis on the motion controls, obviously. 6DOF tracking. That's really exciting and new. And obviously you need to have certain positions if you want to play games in VR, so you have to actually use your body and use physicality in order to use the devices.

It's also very focused towards hardware and software. Very much one ecosystem at a time. It's not really a shared experience. Then you have...

It's a very complicated setup. Most of these 6DOF VR headsets are actually extensions. Screens with tracking on them. They're running on high powered computers running Steam.

That means if I would be in a wheelchair and I would want to be in the center of the game place, or the tracked area, I would have to have someone else launching the device for me, because it's on the other side -- I wouldn't be able to actually maneuver it. It's not really an easy setup. Which also causes a lot of friction in the adaptation in the market. And then gamification.

Like I mentioned before, games are the seller. It's more like console market right now than actually a VR content market. And games are having a very different kind of input and user experience to productivity software. So it's a very different kind of approach.

But they're dictating right now the market. And that's why it makes it a little bit inaccessible. And then audio design, the thing that we're actually touching on mostly today, it's used mostly as an afterthought right now. Because VR is not really focused on making the sound spatially accurate.

Mostly some effects in there, and some good Dolby, and that's it. And so we have definitely some space to catch up there. And then there's not enough testing with a broad range of people. Yeah.

So also what I'm seeing, as far as it goes with the technology, is that we're going more and more towards... Oops. What did I do? Okay.

Yeah. I'm skipping a slide, sorry. Yeah. I'm flying through this really quickly, because most of you know already these things in the space of accessibility.

This is like... How people with visual disabilities overcome challenges in user interfaces. With zoom, contrast, color inversion, and they use tools like screen readers. And keyboard inputs with Braille.

And then we have people with motor disabilities that are using customized custom mapped controllers, rigged controllers, the way they feel best for them, and use voice controls, even Alexa. With the Internet of Things now, we have things that are accessible now that weren't before. And there are single click input, the sip-puff breathe switch. It's a very limited control.

And there are more examples. I will share the slides later so you guys can dive deep. What did I just miss? How many did I miss?

Okay. So what I'm seeing is a trend towards the fact that we're actually losing controllers completely. And that we're going to really just understand and capture hands and then translate those into VR space. Not manipulate the VR space.

We're gonna have questions after this. So in order to... If we're having a look at the controllers that I have found that are actually rigged and focused around people with motor disabilities, and the need of some customization, there are some charities like Special Effects, that create custom controllers for game consoles. There are some makers and engineers that rig those devices for kids.

So they can play with their other fellow kids, and it's a very inclusive, important kind of act that they do. And we can learn from this kind of rigging, that the controller is basically key to using the content. And it's very important to open the customizability towards any kind of technology. Especially if you're having already a controller that works in an environment to manipulate content or explore content.

You should be able to translate that and just use the things that you used to use, in order to dive into the space. And then there's Bluetooth connections. And some other controllers you can buy on Amazon. And then there are dedicated VR controllers already.

So what we're seeing here is -- slowly it starts. People come up with a solution, because the technology is failing them. Or the technology is not supporting their needs. And therefore you have...

Like, a controller, it's a facial expression VR controller that has a band around your VR headset, and it understands muscle contraction. Therefore you can navigate the space just with your facial expression. Or the 3D rudder. It's not yet supported with JavaScript API, but it works with Unity and all the others.

And you can navigate it. It's like a joystick that you can work with your feet. And there's a driver solution, like Walkin VR. Which is basically for people that are missing an arm, that you can actually use both controllers and switch back and forth between the controllers and manipulate interfaces as if you had two arms.

It's pretty innovative and pretty cool. They have some really good videos on YouTube. I suggest you check that out. So if we're looking into accessibility in the VR industry in general, there is stuff going on.

It's not being ignored. But it is... It's still a little bit of a side note, understandably. Because it is complicated.

And it does kind of slow the experimenting of crazy ideas. But it also is very important. So we have... We see that software, like, 3D engines, like Unity and Unreal, are already integrated natively.

Like colorblindness settings. So you can actually build your environments and check if a person has some kind of colorblindness. So you can see if contrast is good enough. And here are a few talks from OC4.

This is from last year. But most importantly, the last link is the Game Accessibility Conference, that just started last year. Which is actually really focused around accessibility and VR. So that's something to look into.

And then I just found an article like a few weeks ago, about Microsoft's showcasing 6 dimension of freedom HTC Vive environment, where a blind user is using a 6DOF controller with a contraption that blocks him or her. So she actually has a virtual blind stick to navigate the space and explore virtual trash cans and virtual streets and different sounds. So you see interesting approaches to how to make VR for people without vision exciting.

If you saw the first video, it was all about vision. Now it gets more into the actual special experience. Yeah. So here are a few components about accessibility in Unity, which is a 3D engine.

And we also see here it's mostly around the idea of visual problems. Visual disabilities. Font sizes and contrast and selectivity. So it's not really focusing on audio at all.

Which is interesting. Yeah. So let's talk about the WebXR device API. Now that I showed you all this development and what's going on in the space.

Who knows about the WebXR device API? Yeah? Anyone? So that is an experimental API.

It's basically a way for the browser of any device to understand the hardware behind the browser. So my cell phone has a browser. My desktop has a browser. And my VR headset has a browser.

And I'm loading up a headset and it will tell me if I can use a stereoscopic camera, have three dimensions of freedom, six dimensions of freedom, understand the input of position information in the space. So I basically can use any kind of browser to augment and progressively enhance an environment that I can describe through the browser. And you can see all the browsers supporting that if not natively, as polyfill. If you have an iPhone, you can use WebVR on your iPhone with the polyfill.

So it's basically cross-browser at this point. The benefits are you don't have to go through an app store. You can just load up a website and experience VR, whether it's on your desktop, tablet, phone, or on a VR headset. You can switch between VR mode and non-VR mode.

You can use other JavaScript libraries, like text-to-speech, or some kind of data visualization library or Google Maps or whatever you have. The power of the web is at your feet. Because it's just another API that you can order from. And since there has been so much done for accessibility for browsers already, it's just in my sense -- it's just the logical next step to bring all these kind of beautiful things into a VR environment and make them immersive.

And here's a small video from my friend Arturo. He actually created an Explainer video of what progressively enhanced VR means. So you go from a desktop with a magic window, to a tablet, where you can touch, and from a phone, where you have the gyrometer showing you the directional... And then just a Google Cardboard device, and then the DayDream headset with remote.

And then we go further into 6DOF devices that also have browsers. It's not in this video. And one tool to create those VR experiences... I have to find my mouse again.

Sorry. Is A frame. I'm running the A-Frame New York City Meetup, so anyone who wants to join me and build stuff is very welcome. A-Frame describes content the way HTML5 would describe a website.

So if you have a link, you would have a scene, a light, a camera, and a box. You would place those in a three-dimensional space and create a virtual environment, just semantically, like a website. It's pretty powerful. And it's built on other libraries that are much more complicated.

But it's a very good tool to prototype and test and just explore nice things. All the demos we're going to show today are built with A-Frame. And part of using semantically-based description of VR is that you actually can use the browser -- like the metatag, like reduced-motion can make a VR environment less busy, in order to not trigger any seizures. That's very important.

It's powerful. So you can actually borrow from that kind of environment. Here's an example for an API that uses the game play API. It's a component working with A-Frame.

A-Frame has a lot of components in the registry, and people like me are submitting components to the library, in order to enrich the experience and make it more powerful and more -- everyone can basically plug and play stuff together. It's much easier. And here we show an example that we have all these different keyboards -- from a keyboard to a controller, across a board. To make a standardization of an input possible.

And I did look into this. I looked for... I was searching for commonalities in the accessibility as an input. And went down the list from keyboard to game pad to touch pad, from HoloLens gestures to touch screen gestures.

One of the most common denominators, what are the lowest key inputs that we would need in order to build across-the-board accessible inputs, and I basically came up with the stick and the four buttons. Remember the tweet? Anyway... So these are some components that I wanted to showcase that are built in A-Frame.

On top is the A-Frame GUI, graphical user interface, where I basically create a user interface with buttons and text that is machine-readable or screen-readable. And ARIA-enabled, and also can map controllers, so you can create interfaces in VR. And then we have Ruben's A-Frame ARIA component that makes elements readable. If you say this is a box, you can read it out with ARIA "this is a box" so screen readers can enhance your VR experience.

You can have keyboard support for interactions, and you can have some experiments about reduced field of vision when you move. So that you don't get sick when it's too fast. With teleportation and motion. This is me last year.

I was presenting my very first version of my proof of concept at the W3C in Belgium, in Brussels. And I was showing an Italian Piazza that I found in the North of Italy. It was a square Piazza with a fountain in middle, and there were some nice pieces that were interesting, like a restaurant. So I was building this place in A-Frame.

So you have... It's obviously very Cubistic, and it doesn't need to be accurate, because it's for people with no vision. But for me, it was just to create some positional audio. So what you're doing here is you're having a compass that basically reads out which direction you're facing.

And then you have those red bars, which are beacons. And you can walk towards them. And you basically experience, as you walk through the Piazza, at 3 feet per meter -- and you can hear the footsteps. And it will tell you you're facing the beacon West of the Piazza and you're walking down, and you can hear how you're passing the restaurant and the fountain on the left or the right.

Obviously this was a proof of concept. It wasn't really strong. But we tested it with Antony up here. He works with Nobility in Texas.

And I was told that when he was wearing this VR headset with this phone that he actually started walking, which I thought was interesting. And was a small little victory for immersion. So I decided to continue. Also, another test was for field of vision.

The same with reduced motion. Reducing your peripheral vision, so you don't get motion sick if too much is going on. And here an experiment -- if you move too fast, let me just play this video real quick. The idea is that it gives you a blinking effect.

If your head moves too fast, then it closes your eyes. This is also an experiment to see if it actually helps against motion sickness. Yeah. And the last prototype that I was working on...

Was about augmenting some kind of control. For people that only have that one single click control. So you augment the controller into VR space, so you can navigate the space without having to use any kind of motion in VR. So it's very...

To remote control yourself, basically. That's the idea. Now, there are a lot of challenges... Obviously the technology that I'm using is cell phone.

And cell phones are very dedicated towards touch. And one of the key points about screen readers in the web is that the premise is to enhance or replace content, and that is the enemy of immersion. Because the product that you're trying to sell with VR is experience. And immersion.

It's not really only content. And it's like describing a melody with someone that can hear. It's basically aesthetics. The idea is in order to give people that are focused on hearing a space with immersion, you can't replace the sound that comes from the bottom left just by describing: This is a sound bit coming from the bottom left.

It's not the same experience. So screen readers use alt text for images. Describe the image. And that's basically what you get.

Whereas when you can immerse yourself in a VR space, there needs to be more done towards the idea of people diving and exploring things with positional tracking, with the idea of being in the space. Yeah. So we can talk about the next thing, actually. I'm going to invite Ian into the scene.

So he can talk a little bit about the next steps of what we built and why we built with the Oculus Go. And why don't you take over?

Hey, everyone. How is it going? Can you hear me all right? Cool.

Thanks to Roland for giving me the opportunity to do this. My background is not in accessibility at all. My background is in audio. I'm a developer and I do a lot of stuff with spatial audio, and about a year ago, I got into WebXR, and I've immediately been hooked.

And for me, I think that this was a really cool project. And so I'm just gonna talk you through my process of thinking, and kind of what we were trying to do here. So as Roland just described, this last scene was in the Piazza in Italy. And that scene was made from A-Frame.

It was pretty -- what we would call low spatialization in A-Frame. So when Roland approached me, he asked: What can you do to really bring this audio to the next level and make it so that it's sonically accurate? So when someone is in that space, they can actually hear sounds in the correct placement of the room? So we also want to keep accessibility at the top of our heads at all times.

So we approached the last two weeks' worth of development and prototypes with that. And we chose the Oculus Go and WebXR for the platform and prototype for development. There's a couple of reasons for that. WebVR is an exponentially smaller file size.

When you download a game on Steam, you're looking in the gigabytes. WebXR, you're looking in the megabytes. If you look at the past, in historical tech, everything has moved in that direction. You start with a large file, and now all of a sudden I can stream a song through Spotify in under a second.

So there are a lot of larger trends, but WebXR, because of that reason, makes it the most accessible way to send someone a virtual reality experience, without a doubt. Next I want to talk about the Oculus Go, and why we specifically chose the Oculus Go. In this field, there are tons of headsets, but the Oculus Go has a couple really great things going for it. The form factor is probably the biggest.

Unlike an HTC Vive or other headsets, the Oculus Go is very much a place on your head and jump into WebXR experience. So for someone missing a limb or for someone who is impaired, this is the easiest headset to put on. I'm gonna talk a little bit later how it may not be the most powerful headset, but as far as accessibility, it's $199. Very easy to put on.

And so that's why we chose the Oculus Go. And as you can see on the bottom, there's a couple examples of other headsets that are similar in form factor. The two that I'm pointing at right now require cell phones. The Oculus Go actually has the internals of a Samsung S6, as far as CPU and GPU, but it's built into the device.

Meaning you don't need another device to place into your head set to let you experience VR. So with that in mind, we tried to create an experience in WebXR that would work on the Oculus Go. And I'm gonna talk a little bit about that. This is just a demo of what was the product of my work in the past week with Roland.

And I just want to preface -- I'm gonna talk about spatial audio in the next 30 minutes. And spatial audio requires a very high quality playback system, or a controlled playback system like headphones. As you can see, we don't have that in this room. So a lot of the things that I'm gonna be talking about may not translate very well.

But I would implore all of you to try to check out these demos on your own with headphones, either afterwards or more likely at home, because these are WebXR. They're just a link. You can go on your computer and you'll be able to hear these things. So I'm just gonna play this demo really quick and see if it works.

You are now standing in a virtual town square. You are walking towards the East of the Piazza. You are looking at a farmer's market. You are looking at a person feeding pigeons.

You are walking towards the beacon South of the Piazza. You are looking at a person feeding pigeons. You are walking towards the beacon West of the Piazza.

So what's kind of going on behind the scenes in this video that may be hard to hear off these speakers is this audio is being spatialized. Each one of the beacons and each one of the places around this Piazza has a unique sound, and that sound is positioned in space, meaning the closer you get to it, the louder it is. But one of the cool things that I'm gonna talk about later is this uses first order ambisonics and high quality audio spatialization, meaning that when you turn your head around, you're also having filtering.

So imagine a car going by. The pitch of that car changes based on its speed, how fast it's moving. The frequency of its sirens, things like that. So this demo is done with something called Google Resonance, and that's how we were able to achieve first order Ambisonics.

Cool. So I'm just gonna outline what I'm gonna talk to you guys about today. The main topic is just spatial audio and the opportunities for spatial audio in accessibility. So I'm gonna give you history, basic definition, and just a summary of what spatial audio is.

How we got here. What are the different tools. I'm gonna talk about head tracking, and specifically what makes spatial audio possible on a technical level. I'm gonna talk about how you deliver spatial audio on the web.

And then I'm gonna go into one last demo that I have, which also uses Google Resonance. So how did we get here? First of all, can anyone give me an example of non-spatial audio? Monosound, but also stereo sound.

So I'm gonna define spatial audio in a sec, but multichannel audio really is the grandfather of spatial audio. And we couldn't have spatial audio without multichannel audio. So we obviously started out with one channel, with the phonograph. We had a lot of technical limitations with that.

Distortion is one of them. Packing a lot of audio data into one channel is a very hard thing to do. We move on in the 1930s. We have the introduction of stereo recordings, which gives artists the ability to pan to somebody on the left, pan to somebody on the right, and play something in the center.

Next, in the '70s, you have Pink Floyd, who is one of the first music groups to really explore Quadrophonic sound, meaning four channels of audio. And for the first time, Pink Floyd was able to position musical elements behind a listener. Which was a pretty cool thing at the time. This was the '70s.

After that, we have the creation of Surround Sound by Dolby, and something called Ambisonics, which I'm gonna talk about a little bit later in this talk. But one thing I want to point out is that we start out with technical limitations, and with every added channel, you have more creative expression. You're enabling people to put more of their data, their ideas, into practice. And then Ambisonics takes that creative expression and says: How can we make this as sonically accurate as possible?

How can we mimic real life sound? So with that knowledge of multichannel audio, what is spatial audio? Spatial audio is sound that has been recorded, mixed, or integrated in a way that simulates real world sound waves and human hearing, because of three things. The head related transfer function.

It's a mathematical equation that quantifies the characteristics of your ear, your shoulders, and I'm gonna go into that with some graphics in a bit. But that is the math behind how sound travels into your body and how you're interpreting it. And that can be quantified in certain ways. Another pillar of spatial audio is head or camera tracking.

Meaning at all times the spatial audio system needs to know where the user's head is, in order to position and spatialize things at a high accuracy. And the last thing is a controlled playback system. For spatial audio to work, you need headphones or an array that's been positioned correctly, so that you can correctly play back audio in a way that our body will interpret, as far as space and depth. So I just talked about the head-related transfer function.

I'm just gonna quickly touch on this, because it's pretty much integral for spatial audio working. These graphics were made by Google Resonance team, so I just wanted to give them credit on that. But HRTF takes a bunch of scientific things about your ear into play here. Right here you have how sound is bouncing off the different crevices inside your ear, and each one of our ears is actually different.

Each one of us has a different HRTF. What is being used in the world today is an averaged HRTF, that's best estimating how all of our ears work. But you can actually go ahead and measure your own HRTF and find the true mathematical calculation of your HRTF. And that's built upon a couple things.

So you have the crevices in your ear, but you also have the size of your head. So I'm speaking in front of you all, and if you turn your head, my voice will actually reach one of your ears faster than the other one. And in realtime, our brain is calculating this and saying: Oh, this person is farther away. This person is closer to us.

This person is on our left, our right. So this is actually happening right now. So what is spatial audio, what is not spatial audio is actually a trick question. Because everything is spatial audio to a human being, including my voice right now in this room.

Here's just an example of head tracking. Which, for an audio system to know where you're looking, it either needs to have a VR headset, a camera looking at you, or if this is a video game, the camera in the digital experience. So VR is not the only example of spatial audio. Spatial audio has been around for the past ten years or so, and being used a lot in video games.

Specifically first person shooters. You need to know where an enemy is. How far away they are from you. If they shot one of their weapons.

Spatial audio is actually really crucial for video games, and you'll see some of the highest skilled video games players always wearing headphones, and there's a reason for that. Because spatial audio can actually give you an advantage in some games. All right. So let's talk about how VR in my opinion is the perfect platform for spatial audio.

And how a lot of technologies and momentum in this space have kind of gotten us to what I would consider peak spatial audio. So the best thing about VR is that most of these hardware devices have very low latency head tracking. Meaning that a system can query a gyroscope or something like an accelerometer and find out your position very quickly and accurately. For a spatial audio system, this is really, really good.

Because nothing is worse than moving your head and having a latent sound not move with you. Also there's a big emphasis on headphones on VR devices. So the Oculus Go has two good speakers that have very good left and right channel separation. Some headphones don't have that, but with the Oculus Go's headphones, you can actually separate where things are on the left and the right channels.

And the last thing is that software and hardware integration has improved dramatically. And not just in the WebXR field, but in the Unity, Unreal, and other game engines. Spatial audio has been integrated, improved, and worked on now for a long time. So here's a demo that is not about accessibility, but kind of shows off Ambisonics, and all its capabilities.

So this is an app that I created called Room Simulator. This is a first order ambisonic experience that works across desktop, mobile, 3DOF and 6DOF devices. I believe it's one of the first to do this on the internet. And I'm just gonna give you a little example.

In this app, you have a ball that plays an mp3. And you're able to change the materials in a room, and hopefully you guys can hear a little bit what that sounds like. (dramatic music playing)

So I just changed it to marble. (sound gets more reverberant) So I'm changing the materials of the walls in this virtual cube. And when I changed things to marble, there's more echoes from the sound source. When I changed things to curtains, there's more absorption of the sound in the room.

(percussion comes in) So this example just proves first order ambisonics in a browser is 100% possible. And yeah. So I'd like to talk a little bit about Ambisonics before I go into my work with the Piazza. Ambisonics is what I like to refer to as the 4K of spatial audio.

What's available on the web now uses something called Panner Node and HTML5 audio to position sounds in a space. Ambisonics does that at a much higher level and offers greater accuracy in spatialization. So the last two demos that I showed were both using Ambisonics, and that gives us the best possible chance at spatializing things at a high accuracy. And Ambisonics is...

It's been around since the '70s. I showed in the first slide. But more recently, Ambisonics has really been integrated into VR and digital experiences. And I think that's awesome.

And I really hope that Ambisonics continues to grow. So how do we deliver spatial audio? I kind of just touched on it a little bit. There's two main ways that you can deliver spatial audio in WebXR.

There's the panner node, which is used in 3JS and A-Frame. That's what I would consider low quality spatialization. That's basic left-right, and it also uses distance approximation, and something called volume attention to give distance sounds. The next level up is Google Resonance, which is what I just mentioned, that uses Ambisonics.

And that can scale from first to third order Ambisonics. High quality spatialization is filtering, volume attenuation, and much higher accuracy of placement of sounds. My recommendations, when delivering spatial audio on the web, is: Keep the files to be mono. Meaning one channel.

Because a lot of times, you don't accurately get the difference between mono and stereo, and it's not worth the second channel if you're not really able to appreciate it. And that brings me to my next point, which is: Because we're on the web, we're always trying to optimize file size and get files as small as possible. So that's why you should use the mp3. And I know I'm gonna get some flak online for this.

But I would argue that the mp3, at 320 kilobits per second, is indistinguishable from the wav file. So I would argue that the mp3 is the right call for this. This is being live streamed, so I know the comments somewhere... People are going to actually be mad.

But let's get to accessibility in spatial audio. And the point of this talk. So sound localization is integral for navigation and auditory cues. And I went to the University of Michigan, and I was really fortunate to have a professor there who was blind, and the way that she described how she listened to rooms when she entered them was pretty insane.

She could tell me the size of the room, how many people were in the room, and then based on where she was, she could listen to echoes and tell you if she was close to certain walls or not. So sound localization is really, really important, and something that we take for granted all the time. It saves our lives. It's how we hear cars coming before they hit us.

So sound localization -- a very good thing. And we can't do that with basic audio on the web. Which is a real bummer. So one of the points I'd like to drive home the most is that spatial audio lets content creators offer a more realistic and understandable sonic experience for everyone.

Not just the visually impaired or someone with disabilities. This is for everyone. And I think that that's a completely untapped area right now. On the accessibility side, someone who is visually impaired or blind, spatial audio is gonna give you the opportunity to understand a space in a digital world more than you ever had before.

So imagine entering websites and being able to hear more expressive information coming from the audio about that website. Right now, we can't do that without reverb and things like that. But I think that spatial audio will allow people to really understand digital spaces in a new way. So the potential use cases for spatial audio and accessibility: Obviously sound localization training.

Having someone use an Ambisonic virtual reality experience to train -- I guess muscles is what it comes down to -- but train your ability to locate sounds using virtual reality. That's a big use case. And another use case, based on augmented reality and machine learning, something we're gonna show an example of in a second, is object recognition and audio feedback. If I hold my phone up to a plant, it says: This is a plant.

That uses audio. But imagine I'm holding my phone up to a plant and it's across the room. Understanding where that plant is with audio feedback. Also another untapped area.

So Roland showed this. So I'm gonna skip this for time purposes. But here's an example of that AR app that I was just talking about.

SeeingAI is a Microsoft research project for people with visual impairments. The app narrates the world around you by turning the visual world into an audible experience. Point your phone's camera, select a channel, and hear a description. The app recognizes saved friends.

Jenny near top right. Three feet away.

Describes the people around you, including their emotions.

28-year-old female, wearing glasses, looking happy.

It reads text out loud as it comes into view, like on an envelope or a room entrance.

Conference 205.

Or scans and reads documents like books and letters. The app will guide you and recognize...

So it's pretty cool. There's a lot of use cases for augmented reality and spatial audio. I think right now the biggest division is that augmented reality really has real world applications. Unfortunately, it's a little bit less developed.

And virtual reality has more training and higher quality experiences, so that's why I think it could be used for sound localization training, more than AR can. Cool. Do you want to talk about this?

This is just... I was participating in an MIT hackathon last year, and the winners of the hackathon was a spatial audio AR app that actually was planting AR snippets across the city, in order to guide the person... Give audio cues. The potential for that would be if a person that is blind walks the street at nighttime, he could hear the restaurant sounds from that restaurant across the street.

So he would know where he is at any time in the space. So it gives him... And you can leave messages or make it a social experience. Teaching people about spaces with audio too.

But I can play the video real quick. (bicycle bell ringing) (birds chirping) (mailbox opening)

That was a hackathon prototype. You know.

The sound of the door is very useful. Knowing where the entrance is -- even better. Knowing the sound of the air conditioning is not important. But what goes on inside the building is important.

Hello. Good day.

Okay. So I'm gonna talk now about the second project that we did with Google Resonance. And I may pull Thomas up here to talk a little bit more about this. But about two weeks ago, I met Thomas.

Thomas Logan works at Equal Entry. Part of the company that throws this. So shout out to them and Thoughtbot for hosting. But he's working on a really interesting project, and it's about visually impaired and blind people crossing very busy intersections, and that's kind of what we worked on, but he has an even larger project that I'm gonna invite him to come talk about.

Just very quickly, this was part of the Call for Innovation challenge at the beginning of the year with the New York City Department of Transportation. Remember, two meetups ago, we had a presentation about how difficult the subway can be to navigate in New York City? This project and this sort of call for innovation is about pedestrian safety and pedestrian... People who are blind or low vision, crossing the different intersections in New York.

So if you think about most intersections in New York City, there is a visual indicator. Of a walk or don't walk. But it's purely visual notification. So for people that are blind or low vision that need to be able to know if it's safe to cross the street, there's something called an accessible pedestrian signal, that sonifies that experience, and makes that available.

For some quick stats in New York City, there are about 13,000 intersections that have visual signals, and there's about 350 intersections that have audio signals. So it's about 2% of the intersections have that. Hence it's really a call for innovation, to think about lots of different ways to solve that. Chuck Graham, who's here with me, we worked on this project, just thinking about: How do you make something that costs currently $30,000, about, to deploy per intersection -- make that with maker hardware or other ideas.

That was our proposal, addressing this call for innovation. But when I saw the Piazza for this talk, I thought very much of the test intersection that was part of the call for innovation. 7th Avenue and West 23rd Street is a very busy intersection, and there's a very large population of people that are blind or low vision that live right beside that intersection. So it's a great thing to potentially be able to prototype and work with people in the community on different calls for innovation.

So I'm excited to see what you guys did to the intersection.

Yeah. Thanks, Thomas. So the main goal, when I spoke to Thomas, was: How do we take a scene that's very rich in audio and make it work on the Oculus Go or make it work in virtual reality? And the reality is: In real life, there are so many different sound objects that make up what we know of as just a basic audio scene, and in virtual reality, it's very hard to replicate every one of those objects.

So Roland and I had to go and kind of figure out in person what's really worth sonifying, what's worth having, and what can we package in a WebXR experience, but still have some sort of meaningful audio behind it. So this is just an overview of the intersection that Thomas just mentioned. And there's also Visions, which is a clinic for blind people, that's around the corner. And that's one of the reasons why it's highly trafficked.

So Roland took all these pictures, and we just went and listened and watched how people navigate the space. And what makes sounds. What do they hit to make sounds. So these are all just different pictures from 7th and 23rd.

And this is an early prototype of what a VR experience for this intersection might look like. This worked fine on an HTC Vive, but when we began to try to put this on the Oculus Go, which is around the Samsung S6, it didn't work at the frame rate that we wanted. So we had to downgrade a little bit and focus more on audio than visuals. And this is the final product, after about a week and a half of work.

This is a first order Ambisonic experience and it lets the user press an APS, listen to the sound experience, and cross the street. Where is it? (street sounds) (rhythmic beeping) (cross signal) (rhythmic beeping) (cross signal)

So again, it's kind of hard to translate the spatial stuff that's going on. Probably the only thing I can say that you just noticed is basic volume attenuation. So that APS has two sounds. It has the ticking walk sound.

It also has the location sound. And when the camera in this video moves past that sound, it was also on the right side. Which you can kind of hear. But you can hear that sound become more in the distance, and more present as you go near it.

So one of the things that we were able to accomplish was sound localization and accuracy. So when you put headphones on, when you look towards the street, you hear street noises. You hear car noises. When you look towards the APS, it's actually gonna sound like it's in the direct front of you or the side of you, wherever you are, when you press it.

So that's kind of the biggest win out of this prototype we made for the intersection. We're trying to go for sonic accuracy and trying to make it so that someone could benefit from using this in some way. So yeah. Just a conclusion of trying to build these different WebXR spatial audio experiences.

The Oculus Go, like I had been saying, is very low powered. So you can't really do a lot of spatial audio. But you can do some things. And this is both good and bad, but to be honest, in the state of the industry, hardware and software, to be able to pull off a first order ambisonic experience that works somewhat well on a Samsung S6-level device is pretty good.

So we have something that works, but I think with some improvements, which I'll speak on in a little bit, this can get a lot better. One of the big downsides of choosing the Oculus Go is that most untethered VR devices do not have something called the text-to-speech API and don't have speech recognition as well. So the beacon -- you're looking at the North beacon. And you're looking at the fountain.

That's all done with speech. Text-to-speech. And that's a generative text. So that unfortunately is not -- that API is not supported on the Oculus Go and other untethered devices.

So that was something that was a bit of a challenge. And so some of the next steps just from going through this process: It really seems like there's a division of AR, real world applications, and VR training experiences. And I think that as time goes on, both of those are gonna get stronger. And as time goes on, the untethered VR headset is going to get more powerful.

So you may have more options, as far as sound objects that you can use on, say, an Oculus Santa Cruz, when it comes out. If it ever comes out. And some of the other next steps -- we will definitely be getting text-to-speech on these devices. And just pushing in general for this industry to adopt sonic accuracy, instead of just audio playback.

I think there's a really big difference, and for someone who is visually impaired, that difference could mean a lot. And it could mean not understanding a digital space. And in virtual reality, you have this great medium where you can explore depth and immersion and all this stuff, and if you're not correctly using spatial audio, in my opinion, you're really missing a lot of that. So I can't tell you how many VR experiences I've gone into, and it's headlocked audio.

Meaning your audio travels with your head. In general I think that spatial audio is getting more and more popular, but it's not at the level that it needs to be, and it needs to be adopted more. And I think that accessibility is a great platform to kind of drive that home. Because we need spatial audio for accessibility to work.

And I think that's a good angle to have on this stuff. And with that, I think that we can take some questions now. And I appreciate you all listening. It looks like the world is ending outside.

(laughter) So thank you.

Before we dive into the questions, though, really quick, to explain the VR experience we built for the street crossing as a training experience, to teach visually impaired and blind users what to listen to. Because since New York City is so very crowded and a loud space, a lot of people, a lot of cars and fire trucks and construction, it's really important to kind of position the things they should be listening to in an accurate position and then over time you can add -- in this VR experience -- noises and things that are in the way, like bikes that are crowded the space, and with the remote control, you're able to actually feel out the area.

So you could actually have collision with the remote control extension, similar to a blind stick in a VR space, and navigating it. So you can actually feel if there's a manhole cover on the floor that is slightly ajar and would make you trip. So you would be able to figure out... Oh, there is something coming up in that space, at that time, because we accurately mapped that in the VR experience.

So you can play around in the space and do it in the comfort of your home. Right? So that's the whole prerogative. So thank you, Ian, and thank you all for listening.

And we're ready for questions.

Ian, how critical is the technology and method of recording the audio that you use in your work?

It depends. So for this type of playback, I kind of spoke to this before, but I recommend the use of mono files, and that kind of goes into your question about recording. There's different ways of recording multichannel audio, and in Ambisonics, you have things that have four microphone capsules that can capture audio from every direction. That's used a lot in VR film.

For the spatial audio stuff that I was talking about, you're looking to recreate a space with sound objects. And I would recommend that those objects be mono. So as far as recording mono objects, just a simple microphone. But the higher end Ambisonic microphones could work.

I just don't know if they would be the best pairing for something like WebXR, where file size is really crucial.

So in practice, in the Piazza, are you walking up to the fountain, holding a microphone to it, to get that object's audio output?

So the great thing about mono audio in 2018 is that 20 people have already done that and they've already Open Sourced those sounds and they've already mixed them. We did with microphones go to the intersection on 7th and 23rd and had terrible luck, because of how many people came up to Roland and I, just asking like... What are you doing? (laughter) It was just...

We didn't get good recordings, at the end of the day. So a lot of this stuff I look to synthesize the material and make it sound as best as possible. And I look to game sounds like XYZ, BBC Sound Library. A lot of the sound libraries that are public and free to use have amazing sounds.

All the VR developers in this room, or watching this live stream -- I highly recommend going that route.

I just wanted to ask you... This is related to one of the last things you said about mapping a space. And I feel like this is probably a really naive question, but is there the capacity to do that in realtime? So you're talking about how you could do it in your living room, or you could gauge that there's a manhole cover.

But would the person be doing that as they're navigating the space in realtime?

Well, so we saw at the hackathon, with the droplets of sounds -- that's geolocation. It's not super accurate. But I would imagine with higher end headsets that are inside-out tracking, which means you don't have it tethered, but actually the device maps the space, you can. And the HoloLens does also.

Place the objects in the space. And those could be sound objects. So the potential of the technology is there. And more and more growing.

So it's definitely... You definitely at some point could build a non-coding experience where you say... Okay. I'm mapping this space for you.

I'm dropping things and I'm bringing that home. So you can play it at home to figure out how the space looks and feels. Yeah.

Just to add one more thing on top of that. Most audio spatialization systems use a cube. And they create an artificial cube, and that is how reverb is calculated, how distance is calculated. What you're describing would probably be the next evolution of spatial audio, that wouldn't be limited to a cube.

So you could walk down the street and the scene that is spatializing the audio is not a cube, but actually may be getting data from the buildings around you. I think that's the future, for sure. But for now, we're really limited to spatializing within a cube.

So one of the problems I was working on last year -- I would be curious, because I didn't get it solved effectively -- was doing directional sound objects. For example, on the intersection you showed, I'm pretty sure that, as you walked past it, you get the sound changes as the head, the camera object, moves by. But the speaker perhaps on that is facing in a certain direction. So even if I'm looking at it and I maintain directionality, the sound would naturally distort as I walk past, because the speaker is sending it in a certain direction.

The same way, if I'm talking to you right now, if you walk around the room at me, my voice would still be going this way. What are the steps that can be implemented for WebXR to add directionality to the sound objects?

That's a great question. The only way to answer that is with a very technical answer, unfortunately. Google Resonance is the tool I use that lets me put Ambisonic audio onto the web. A source in Google Resonance is something that emits sound, and you have different parameters you can change within that software.

Things like the minimum distance to hear that sound, the maximum distance to hear that sound. Is the rolloff of the frequencies logarithmic, which is kind of like an audio term, but makes a little bit of a difference, and I believe you can set something called source width, which is what you're talking about, which is: How wide the sound appears to the user when they walk up to it, or appear. So I think there's also reflectivity in Google Resonance.

So using kind of a combination of those parameters. You probably have to get pretty creative. You might be able to get the effect that you're looking for. So yeah.

I would say it's most likely gonna be just some really hardcore Google Resonance coding.

Thanks!

Google Resonance has probably been around for a year now. I think it's probably the premiere audio spatialization library. The great thing about Google Resonance is that it's not just on WebXR. So one of my friends uses it on Unreal Engine a lot.

We can spatialize things on Unreal Engine using Google Resonance, using the same kinds of parameters I just spoke to you about in WebXR. I can translate that to Unreal Engine and do the same changes of parameters. And the great thing is that Unity and Unreal Engine are much more powerful in terms of their features. So when you start to get into the game engine stuff, Google Resonance starts to open up its feature set, and you have something called inclusion, which means: When I put my hand in front of the microphone, it knows where the sound is and what's blocking it.

That's not a feature that's available on the WebXR version of Google Resonance.

I'd like to ask a follow-up, which is: If you develop your Google Resonance layout on Unity or Unreal, because it's a little more flexible for quickly editing 3D space than trying to do it in the A- Frame environment, can you copy what you've designed there and shove it back into WebXR? It would be looser functionalities or what-not?

Yes. So what he's asking is: If you use a game engine to build digital worlds, and you use the higher end Google Resonance spatialization properties, can I now bring this into WebXR? Yes. But in a limited form.

And it will be a manual transmission. Other than a couple libraries that I know of in Unity and Unreal Engine that export to WebGL -- I wouldn't recommend doing that. I would recommend rebuilding the scene in WebXR and looking at the parameters I use in my Unity scene and copying that to the WebGL scene. I've done that before.

The really interesting thing about virtual reality is that WebXR is mainly web developers who are working on these things. Unity and Unreal Engine is a whole different side of this. So you have to be able to interface with them, and at the end of the day, there's high level spatial audio concepts like occlusion, like source width, like volume attenuation -- all these kind of high level fundamental things, and what Google Resonance lets you do is take most of those and transfer it to each platform.

I guess the whole Unity and Unreal... I understand. Getting back... You had a person who was visually impaired put on a headset.

How did the headset know where that person was standing? To the accuracy that you can safely cross the street? That's where you've lost me.

So it doesn't. This is part of the prototype. The Oculus Go does not have 6 degrees of freedom. So it doesn't know where you're standing.

Yeah. That's what I'm trying to understand. So...

So it only has orientation data, where your head is facing.

So... How do you see, then, taking it to a level where this is a practical device?

That's kind of what I was trying to hint at before with the AR and VR stuff. Which is that I think that practical walking down the street application is gonna be hard to deliver in the VR format. With what's currently around.

About six companies working on what they call AR Cloud, as you know... To find... I mean, I've been working on audio for Tango, where you can map a room. And I've seen interesting demos for visually impaired people with the HoloLens, where they use sound cues to know where there's a staircase, to know where walls are.

It's almost like a submarine sort of sonar tracking system. And certainly with Tango, I've deployed sounds in space as virtual speakers. It is extremely kludgy, working through it. Not using Resonance, but using Unity.

And I could see spending a lot of time trying to make something sound desirable in that world. But then taking it out of that controlled thing... I mean, I know all of the engineers on Tango were sent to Google Maps. And that's where they're trying to create worlds where you'll be able to tag things in the real world.

In realtime. And I suppose there's room for audio there too. To happen.

Yeah. I think that's a great idea. Almost like an Open Source library of sounds, where someone could walk down the street and someone's already placed that audio. The problem is kind of hinting back to one of the first questions I got, which is: Google Resonance is cube-based.

And so it's very hard to take a cube-based spatialization library and transfer it to real world applications. And it's a technical limitation that we are facing right now. So I think the next person who can figure that out is going to be a very rich person.

So my question is actually about mobility accessibility. I am someone who can't use both of my hands. I use primarily my left hand. And we briefly saw a couple of examples of hand tracking tools or muscle tracking tools, but what I'm curious about is things like eye tracking software.

Would that be something that would work in terms of implementing it and incorporating it into VR and AR accessibility?

Yeah. So eye tracking is actually something that a few VR headsets that you saw on the list of devices are actually working on. It's mostly for rendering accuracy. Because you want to create the most sharpest image on the slice you're looking at, at any time.

For performance. But definitely that would be definitely something that could be extended to input. Definitely. One of the early inputs I built as a component was the simple accelerometer of the phone, where you basically just tap the phone, click, instead of having to press a finger, so you do a much more simple movement towards the trigger motion.

So the idea -- as soon as we have devices that have input, the WebXR API is going to get us this information, and make it transparent through the browser, and then we can tap into this information and make creative stuff out of it. Because the accelerometer -- it's just a tap. A creative take on hardware that is there.

Cool, thank you.

Just from a spatial perspective, what would be the difference between spatialization of outdoors and indoors? Meaning exterior and interior? And what's your view on that?

Sure, yeah. That's also a very good question. The demo that I showed with that green and red ball -- that's an example of strictly indoor spatialization. And when we go indoors, we have the ability to place certain materials on the cube walls, and then with Google Resonance go in and say: This wall is a curtain.

And Google Resonance goes: If this wall is a curtain, it can absorb a certain amount of sound in a certain way. We're gonna mimic that for you. I was changing the indoor materials, and that's how the marble is very echoey, and the curtains were really kind of early reflections only. But to your question about outdoors...

Spatial audio and reverb in general works with both early reflections... So the direct sound source, without a microphone, directly to your ears, and then it's working with late reflection. So my sound bouncing off walls. My sound coming from external speakers.

And it's taking both of those, and it's packaging them into one stream of audio and saying: This is where Ian's voice is right now. So with the outdoors, you don't normally have a lot of early reflections. Very large spaces. So...

It's different to approach an outdoor scene for spatialization reasons, because you don't have a lot of things that are gonna reflect sound and give cues to people. So for outdoor scenes, if you're trying to give some sort of sense of depth or something, you're gonna have objects that need to come back and forth into your close proximity for you to really understand where things are. Without that, it's gonna be pretty hard.

We have time for a couple more questions. I'll bend my neck around the corner. Make sure nobody's hiding. Can you talk a little bit more about kind of the difficulty that you see in getting people to actually adopt using Ambisonics and actually spatialization of audio?

Because it sounds like that's not really happening at the moment?

Yeah. I should also just clarify that audio spatialization doesn't just include PannerNode and Ambisonics. There are a lot of other theories around how you spatialize audio. There's a Studio in New York City called Mock One Studios that actually doesn't use Ambisonics, but is still using a cube-based spatialization platform, and that sounds really good.

So as far as adoption, it really comes down to the content. With VR video, YouTube, and other video hosting sites, we're using a four-channel Ambisonic setup. This is how you upload audio for your videos that are 360 onto our system. There's no other way.

So things like that help adoption a little bit. But... Game engines, Unity, Unreal, they don't always use Ambisonics to spatialize sound, and as far as encouraging people to use Ambisonics, it really has to fit right for the project. So if someone is doing a VR 360 film, I would say: Try to use Ambisonics, because all the platforms that people will be viewing this content will be Ambisonic-ready.

So you should be using Ambisonics. Someone using a game engine or Unreal Engine, it really depends on what you're trying to do. So if this person is making an AR application with a game engine, using AR Kit or AR Core, Ambisonics and cube-based spatialization may not be the best route to go. I would just recommend -- and I know this is, like, really a lofty goal -- but audio comes into play at a much earlier level in the creation of something than it does right now.

Right now, what usually happens is the audio person gets a call last, and they say: We have $5, $10 for you. Can you come mix this for us? What should happen is the audio person should be there on the first day. The inception of these projects.

And that is not happening. So I think that if you bring audio into an earlier stage of development, you're gonna have better spatialization, whether it's Ambisonic or not.

That leads to my question about... It's not really so much technical. But cognitive. Like, our culture puts a lot of emphasis on visual information.

Not that we're that good at interpreting that anyway. But we don't... We sort of de-emphasize... For all our fascination with music and what-not, we sort of deemphasize audio.

So it's not just a technical conversation. It's a concept. This piece of entertainment is riven with audio. We're gonna bring the sound guy in to the kickoff meeting.

Right? It's kind of a mindset. Isn't it?

I totally agree with you. I'm biased, though. It's hard for me to answer that, because I've been really interested in sound my entire life. So I would love for everyone to do that.

But I think it really comes down to creative budgets. A lot of times, audio does not get any part of the creative budget. And so the more money put towards audio, the better. I think that audio is an incredibly subconscious medium and people take it for granted all the time.

And good audio is often something that people don't talk about, because it's so realistic and it's so present that there is nothing to talk about. There's nothing bad. Right. Exactly.

So it's a very subconscious medium. To bring it to the forefront, I don't think it's gonna be a stereo mp3 on Spotify or music app. I think it's gonna have to be something like this. That takes someone and says: You have to try something new.

And this is how you're going to experience audio in the future. So... Like I said, I hope all these things happen. I really do.

I think WebXR and the fact that I can send you an Ambisonic experience is definitely gonna help that.

All right. I think we have time for a couple more questions. Walei first and then over on the wall.

In terms of research, have you guys looked into what already exists in terms of audio spatializations? There's a whole genre of audio games, and a couple years ago, there were some really impressive games on iOS, such as Audio Defense and 6th Sense and Papa Sangre, which really utilized audio spatializations and the gyroscope. I would hear the zombie on my right and turn to the right and the sound would sound like it's coming in front of me.

Have you looked into that, in terms of how the interface works, and things like that?

Sure. There's a ton of research in this field. So I can't say I know all of it. But I've seen a couple different projects that are very traditional research projects.

So usually some sort of digital interface or a game. And what they're testing is: Can VR successfully help someone localize sound in a digital world? Or is that a myth? And so the research that I have seen is just kind of scratching the surface of...

Does audio spatialization actually give someone the ability to locate something? Or are there other things going on that help that experience? And I don't think that there's a lot of research in this field. And if you have any suggestions of things to check out, I would appreciate that.

Yeah. I mean, I would recommend definitely checking out audio games. Because they're specifically geared for blind folks. And I believe...

And I don't know if this website exists anymore... But it was like audiogames.net. Something like that. It has a whole library of games to choose from.

There used to be, like I said, some really impressive games on iOS. I don't know if they still exist. But I'm sure there are some out there that I haven't kept out with. Or some in the works.

That, you know, like I said, they're made for blind folks to just have an audio experience, whether it's like a zombie shooter, or blindfold racing or whatever, that I think you can learn a lot from, in terms of how they convey that information in an audio form, using a relatively mainstream device, like an iPhone 5 or 6 or 7 or 8 or whatever.

Thank you.

I wanted to add to this that since Google Resonance is really a fairly new development, being able to be played on Three.js, it's been a couple of weeks. Or not even?

I don't know. It depends on how many people are developing it. About a year.

It's very new. But it gives us the power to recreate those games that are for iOS specifically natively and build web-based apps with them, and make them device-independent, and shared via email. That's the power that I think... That makes this a more mainstream thing, anyway.

It'll make its way across the room.

So I just wanted to make a comment that, when you were talking about being able to appreciate the importance of sound, I was thinking of the experience of going to a ballet, and if you're sitting in the front, in the front row, you hear the sound of the ballet slippers on the floor. And that enhances the experience so much. And it would be really great if that could be shared with everyone in the audience.

Right.

Again, as someone who's produced all kinds of media for different markets, you always have this one fantastic sound mixing room. But if this was going on TV, at the end of the day, you turn it on the crappiest little speaker you could get, because you had to assume that whatever you were doing had to work in that format. And so when you're designing an audio experience for a museum and you know you're gonna control whether it's a $50 or a $150 headphone attached to whatever the device is...

And you can think through how the end user is going to experience that, and you have some control over the delivery of that situation, WebXR is the bliss of... Yes. Now anyone can watch it. But you don't know how the end user is going to ultimately see it.

Is it going to be on a mobile device? With ear buds? It's again a very different experience from sitting at a $1,000 game machine with a $350 pair of headphones. So again, it's always thrown back to some degree...

And that's without even the spatial component of when that comes out. I have no idea what the quality control of everyone's device is going to be, in terms of being able to localize sound and that kind of thing. So it's not really a question. Just a sort of statement.

I don't know if you... Anyway. That's my reaction to the situation.

Yeah. It's a good reaction. And something that I have to think about a lot. And...

Yeah. I was gonna say that's one of the great things about the Oculus Go. Is that it's creating a standard for VR hardware that we can build off of. And that's really helpful.

And it has good separation of the headphones. One thing I would say is that I come from an audio mixing background. So I've done a lot of tests of playing music back on flat monitors. Playing music back on devices.

That same practice needs to be taken into VR. All the sounds should be edited with the thought of... What is the max decibel that I have for all of these sounds? And what is the file format of these sounds?

So I would say it transfers over, and when people put a lot of thought into that, that's gonna be the best case scenario for whatever device you're listening to. Not around audio. And kind of Roland hinted about this a little bit, but WebVR can be progressively enhanced. So when you view something on a phone, versus an HTC Vive, you do get a better experience.

And audio is kind of like that. As... Because with Unreal Engine, you can have much higher spatialization. With WebXR, you're having only First Order Ambisonics.

So you kind of have to embrace the technical limitations and try to do the best.

Cool. Just a general A-Frame question. I was curious if you had seen any mixtures of A-Frame and physical computing or any... Just your general thoughts on that.

If you've seen anything that's maybe blowing your mind or something like that. Just to rephrase, have you guys experimented with physical computing, using A-Frame or other... I guess, more web-based coding? Or is that something you're thinking about?

Or just general thoughts?

For those who don't know what physical computing means?

So physical computing... Like Arduino, et cetera.

So there was a hackathon that ended at the end of the month last month. The judges are judging on it. One of the examples is an A-Frame demo that showcases Arduinos with temperature and moisture controls for farming. So check out that hackathon.

And there's a lot being done with the Internet of Things. Integrations with Amazon Echo. If you follow the Slack channel on A-Frame, there are some projects on there that are quite interesting. Yeah.

And also, if you come to my A-Frame meetup, if I find something that's in that space, definitely we can work and hack on things. I'm always excited to try out new stuff. Arduino or not.

Just to add on top of that, so the person who made that demo is named Mark Heller. If you want to Google him. And he actually used my WebVR boilerplate, which is publicly available, to make that demo. One of the reasons why Mark's demo is very cool is that it does not use a computer.

The Arduino is powered off of a battery, and if you've ever done work with an Arduino before, you need a USB plug into your computer. Which means that if you want to develop something with Arduino and WebVR, you're gonna have to do it on your local computer, which kind of defeats the purpose of WebVR. So whatever Mark Heller was doing as far as interfacing data traveling over a backend server is probably what you want to look into.

Plus he uses an AR component to it, so it augments the information on top of it with AR.js. So that's quite impressive.

All right. So with that, I think we are at time. I'm gonna piggyback on the second to last comment and say that I'm looking forward to checking out the slides so that I can see the demo videos with proper headphones. Because I think that will be pretty awesome.

So thank you to Ian and Roland for this presentation. That was awesome. Thank you to Level Access for sponsoring the Meetup, to Thoughtbot for hosting us, and White Coat Captioning for handling the captioning, and Joly with the Internet Society of New York for streaming everything with the audio and visual components. Thank you all for coming.

We have a little time to mill around. I know we have some devices here. We can't stay here all night, just to warn you, but we do have a little bit of time. So thank you so much.

(applause)