Haptics for XR – How designing great haptics can support use case accessibility | Accessibility VR Meetup Recap

So thanks a lot, everyone, to be here. So do not hesitate, if you have some questions, feel free to share it in the chat. Otherwise, at the end we'll have a Q and A session. A few words about me.

I'm Eric. I've got 8 years in the industry, I got a PhD in haptics, and then I moved forward to being technical director of a haptics company, and I created Interhaptics. So what's Interhaptics in a few words? Interhaptics works a little bit like Unity for haptics.

It allows the users to create seamlessly haptics feeling and haptics experiences for their applications. Without writing lines of code. And it allows them to share their creations on different platforms. So the talk today will be focused on how haptics can bring value to XR.

So what are the practical use cases that make sense to working with haptics? We'll spend a few minutes to talk about the sense of touch, how we do process the sense of touch as humans, and then we will have multiple practical use cases, where we show with the use of Interhaptics how you can create haptic experiences specifically for accessibility in a really easy way. And we'll be testing out how these experiences can be applied to VR and how these can expand on the user accessibility of the use cases.

So first things first. What is haptics? Haptics is every kind of technology that is related to the sense of touch. Everything which is around our capability to manipulate and receive a feeling from a sense of touch is haptics.

And also the group of technologies that allows you to feel a sense of touch are targeted as haptics. For example, the vibrations on your telephone is haptics, as well as if you have an Apple Watch, that's haptics also. But one thing to clarify at the beginning of this discussion is that we are -- from a haptics technology point of view -- we are far, far away from the capability to simulate. So it is not really possible to simulate realistic touch feeling through haptics technology.

Because we are really much far away from the capability of the haptic actuator to recreate realistic feeling. However, haptics actuators have a wonderful potential to enrich user experience in today's applications without the need to simulate. So it's extremely important to bear in mind when we approach -- everyone approaches the understanding of haptics and the haptics experiences. Because it's never good to sell the dream about being able to feel, for example, in VR that's not possible with today's technologies.

In the next few years, we'll have more technology. There are wonderful publications about that, that I can direct you to. So let's dive in a little bit. On the different use cases.

What haptics can bring in terms of value for VR. So there are more or less five different areas for haptics. We can increase engagement, we can give a sense of presence, in some cases, decrease user fatigue, in a lot of cases, if used right, haptics can decrease the dizziness effect and if applied to training, it can increase their effectiveness. One of the items that makes a lot of sense for accessibility is that haptics can increase the user engagement.

Yes, I will move back. Apologies. I cannot really see my own slides. While presenting.

It's too small. So the use cases where haptics makes sense... At the current stage... Are multiple.

So we do have haptics for... I need to turn. I'm not able to see my slides. We can use haptics to give the touch feedback and confirmation for any kind of interactions, we can use haptics for increasing user immersion, increasing remote presence, or user awareness, and use haptics for notifications.

So let's dive in a little bit to these five use cases. And let's see really what we're talking about. So the UI confirmation for haptics is really simple to understand. The touch feedback of a button or a slider or a knob.

Haptics can be used to enhance and give the user confirmation that the digital action is happening. It strikes me that still today there is a lot of VR applications where there is a keyboard which they don't have any haptic feedback. That generates frustration to the user and makes it extremely harder to type in the virtual reality, because simply our body doesn't know where to stop when we are clicking the keys. This is one of the really basic applications for UI confirmation.

You can imagine a knob with different tick marks that will allow you to have a better control of the interaction. The second case -- probably the most obvious -- is the user immersion. We use haptics to enhance the sense of immersion of the user. For example, if you are in a 360 video, you can use haptics to enhance certain kinds of characteristics of the video.

For example, if you have an FPS, and there is a war scene, you can use haptics to enhance the feeling of shooting guns. Or if you have a lever, you can use haptics to enhance the movement of this lever through specific feedback. We'll see how this is easy later. Then the third use case is remote presence.

And that is using haptics to convey emotions. Convey emotions remotely. For example, animation of a beating heart in VR also gives the feeling of a beating heart -- it's much more engaging for the user. Or, for example, the ability to have a remote handshake, where haptics conveys the action of interactions with the remote person.

And this use case has a wonderful potential in the future for everything correlating with telepresence and remote feeling. Then the fourth use case is use haptics for contextual awareness. Use haptics as a sixth sense, really. Use haptics as a capability to convey information to the user that are impossible to convey in other directions.

We are going through essentially overload. That means that we are supplying too many informations through the visual or audio. For example, the necessity to turn left or incoming danger of something that is happening. This is one of the really cool applications of haptics that is not at all explored, which can give the user -- especially in virtual reality -- a lot of different informations, because we have two different haptics devices in VR, which are the two controllers.

Which are not usually explored today. And the fifth one is probably the most obvious. Is to use haptics for notification. To tell the user that something is happening.

In this case, there is Apple, which is doing wonderful work with Apple... Apple Watch. Which was... We were discussing today -- one of the participants to the Meetup, who is deaf, and is really capable of distinguishing which kind of notification she received.

A call or an SMS, through the different haptics pattern. Which is a really, really great thing for avoiding sensory overload and allowing the user to use this private channel, which is the sense of touch. These are more or less the five use cases. We'll see a more actionable way to achieve that in the end, but I want to spend a few minutes on how touch perception works.

A little bit to understand -- which is the design process that an app creator should go through, when thinking about haptics in these applications. Yes, Meryl, I'm talking about you! Thanks for the comment. So...

Haptics -- the sense of touch goes through the perception of tiny sensors that we have in our fingertips. So I don't -- I didn't switch on the camera, but we have 1,000 of them, in, for example, a normal finger tip. And this receptor responds to how much our skin is pressed or stimulated in a different way. We have -- most of them are on the surface.

We have three or four kinds of them. We don't have to go through this. It's not important. The only thing that's important to remember is that when we poke our skin with anything, with our finger, we are touching anything, the local mechanoreceptor near the skin starts to communicate through nerves that go up to the brain, spiking -- which are tiny electrical signals that travels across our nerves, and up to our sensory cortex, and that are interpreted as touch feeling.

And how these mechanoreceptors encode different touch feelings is not at all understood today. It's not like for eyes or ears, where we have a really good understanding of different perceptions in the retina, transmitting to the brain, a specific signal of a specific image. Touch is really not well understood today. And also, there is also the items that touch perception -- is actually really complex.

For example, touch perception is divided into two. Tactile and kinesthetic. Contact location, pressure, shear, slip, vibration, and temperature are all perceived through your skin. And then you have the kinesthetic perception, which is the strength you're using to manipulate an object and the orientation of your body.

These two together are perceived through mechanoreceptors and are transmitted to your brain and then perceived as touch feeling. And this is quite important to keep in mind. Because they will help us in the design process later on. Haptics in general is decomposed in two.

We have haptics that is manipulation of objects. So I grab an object and manipulate it. You see how this makes sense for VR. We're interacting with objects all the time.

And the fact that I am exploring and feeling an object with my hands or my body, or someone is touching me, and this is like a reactive touch feeling -- so why this division makes sense in VR? Simply because in VR, the technological chain from hand position and touch feedback are separated. So there is a complete technological chain that manages how we interact with objects. With every interaction system that is around us.

We at Interhaptics have an interaction system. Which is like Unity's interactive system. And there's a completely separate channel for touch feeling and touch feedback. However, for manipulation, in reality, they are the same.

Our brain is hard wired to use them together. And this is the reason by which we have this strong growth of haptics devices for hand manipulation in VR. Which allows to have a more natural interaction with different objects. Because you have both the manipulation and the feedback activity, which are closer to reality.

So the user is able to interact more naturally. And this is the reason by which, for example, Oculus Quest invested so much to have hand tracking working, or even Magic Leap have hand tracking working. Because it's way more natural for the user just to use their own hands, if they knew exactly how to use them. You can comment on it later on, Tim.

What about touch and touch feedback? For example, for images, for vision, we know that we see in three colors. Some see in four colors. But more or less the amount of signals that you can perceive is pretty much understood.

For sound, it's the same. We know pretty much how many different channels, different spectral channels we can perceive with the audio. For haptics, and for touch, we really don't know. There is really...

We don't really know which are the basic components of the touch feeling. And there is a lot of different methods that are proposed. There is one of them that I think it makes sense to show here. Which really will help us to understand a way to design haptics in VR.

It is the theory of the 15 dimensions of touch. Which has been proposed by SimTouch, a California company which is digitizing the touch sensation of fundamentally -- a lot of materials, and is working with Unity for that experience. And they propose 15 dimensions that allows them to describe sensation. However, when you look into that, and we cross this with the incapability today of haptics to render well reality...

We see that we have texture and frictions which are sensations that exist when you are sliding over a surface. Planar sensations. And we have stiffnesses, which are sensations that exist when you are pushing against a surface. It is a force that is counterbalancing your movement.

Then you have temperature sensations, which is temperature feeling. Which allows you to feel the heat and feel the heat transfer. And in these three, we forgot about probably the most used haptics sensation in today's technology, which is vibrations. Because vibrations in nature don't exist.

They just exist in haptics. And vibrations feelings are completely manmade and completely technology made. So it is natural to say that in this categorization of touch feeling of reality, vibrations don't exist. Because they are an artificial sensation.

We see that while going into understanding what these 15 dimensions of feeling are, we can create something that allows an easier way to think about them. We see that they have four different groups. Fundamentally, two sensations which are defined in space. One is texture.

That happens when you're sliding over a surface. And one of them is stiffnesses, which are forces, which happen when you are pushing against a soft material, or there is a force feedback that happens. And two of them are based on time. One is vibrations pattern and the other one is heat transfer.

Which is temperature. We see here that right now we have four different sensations. And four different digital haptic sensations. We can start to apply and to use these four groups in a proactive manner to integrate within digital technologies.

These four approaches to haptics for different purposes. Based on this, we created a haptic design based on these four different sensations. Like you see here. If we go in the back of the room, we have a video over there.

That will help you to visualize a little bit better what the haptics design is about. There will be a little music right now. I invite you to mute the video. And I will be able to talk on top of it.

So effectively, we have four different perceptions. And within our haptic composer, we can graphically define the feeling that we want our virtual experience to have. And we have all the tools to prepare and create and visualize the texture while interacting with a specific object in virtual reality. We'll feel it through our controllers, or different kinds of devices such as exoskeleton.

You will be able to feel it. Okay. Let's go back to the presentation. So...

We have this haptic design, which is fundamentally like a form of touch which allows you to create digital haptic sensations, which is called Haptic Composer. And the idea right now is to go through six really easy actionable use cases for haptics that are very common in today's applications. And we see how in really well designed haptics, we can enhance these use cases in a really easy manner, in a really fast manner, without spending time to code, and without investing too much time to understand how the technology works.

So... The first one is probably the most obvious. We discussed it before. The enhancement of a virtual keyboard with haptics.

And why we should do that. Because... Why we implement haptics into a virtual keyboard typing... We have a greater interaction control.

Because we have feedback when we're interacting with the buttons. And we decrease the cognitive load that the user is forced to support, to know that he clicked a button. Because otherwise, he needs to rely only on the aural channel or the visual channel. And how can we announce it?

It's really easy. What is the response of the button when in reality while we are touching it? It increases the force, the active typing sensation that we have, when we're pushing the finger down. At a certain point, it drops, and we feel this discontinuity.

And we see here on the screen that simply by using a stiffness feeling that you can design with the Haptic Composer, we can define the haptic curve shape that we want this button to have, while we are pushing it. And of course, stiffness -- we saw before that it is defined in space, so it will be in this case... Defined in centimeters. So it's one centimeter of length, and while you're pressing this one centimeter, you will feel the increase and then decrease of haptic feeling, which will give us the information that the button has been pressed.

And fundamentally, doing this activity with the haptic composer takes a couple of minutes. The second one is... It makes sense also for accessibility... Is sliding menus.

So if you are rolling and sliding menus, it's sometimes implemented -- the haptic feedback every time you're changing a selected area. And when you have changed a selected area, you're giving an impulse to the user to tell him: You change it. And this allows to have an easier interaction control, and again, it's easier for the user to use the channel to perceive that something happened. For example, the Apple Watch has this over the scrolling wheel.

When you are sliding. While you turn a little bit. You feel that something is happening. There is still a sound on the video, they tell me.

I will pause it. So... We have... It's really kind of easy to implement again with Interhaptics, because it's enough to define the rhythm at which you want the haptic feeling to happen.

For example, every centimeter in this case. And assign to every centimeter a click that would happen when the user is scrolling. Because, again, this is a texture-based system. And texture is defined in space.

Like you would define the pattern of icons that you want to select. You can define the pattern of haptics and superimpose it. This is an actionable use case for navigation within UIs. Third use case -- that's pretty funny and really quite engaging.

It's about using haptics -- for example, it's a drawing interaction. In this case, in reality, when you are having a drawing interaction, you are feeling the sensation transmitted move through your pen, for example. All along the bodies, and you perceive it with your finger. And what you are feeling through your pen -- it's a texture.

It's the texture of the paper or the material that you're writing on. Why we don't do the same in virtual reality? Why can't we supply the person designing in 3D with a tactile feeling to tell him what he is drawing and complement the experience of drawing in 3D with a tactile channel, with a different feeling, with a different, for example, color or brushes that he is using in 3D, to design. And again, doing this -- it's extremely easy to define the texture, which is in space, because you are interacting with space.

You define the different patterns that you would like your experiences to have. And this pattern will help the user to know instinctively what he is doing, just by doing it and feeling it through their hands. And this can generate really, really good feedback in the user, because he comes back to strategies that he uses in reality, and he understands exactly what he is doing, because he is doing it with his own hands.

One of the last ones is guidance information. We have two haptics devices. One left, one right. We can use these haptics to tell them to turn left or turn right.

In this case, we can use vibration patterns where we can modulate the amplitude of vibrations as a function of how much the user should turn. And then we can use these haptics channels to guide the user left or right. While he is interacting with your application. And this is extremely important, for example, for people who cannot really see well or cannot really hear well.

And where you cannot really use these two channels and rely on them, for them to be able to guide you in your own application. You can decrease the sensory overload and leave the application as-is, to use this haptic channel to guide the user in VR. This is extremely interesting, because for the first time, the user has two haptic devices in two hands, as opposed to, for example, a phone, which has only one hand.

And for phones, these things don't work. We have it for turn by turn applications. But the problem is that there is just one actuator. And it doesn't give you any time information.

So for phone, it doesn't work, but in VR it can absolutely work, because you have two actuators. And the last one I wanted to talk about here is about again, using haptics for complementing and giving contextual information, for example distance from an object. Here we can use the vibration patterns and moderate vibration amplitude and repetition to give the user an information on how far away it is from an object. You experience this every day with cars when you have parking assistance and sound that supplies you this kind of information.

You can do the same in VR through haptics, not with sound. This is the idea. To give you textual information through the tactile channel, which allows you to decrease cognitive overload. On the visual and sound channels.

And the last one is... The interaction confirmation that is really, really useful for training. And it's interesting to enhance grab and release actions while performing a procedural training through two different haptic patterns that allows the user to have... Essentially confirmations that you executed an activity, and this is used as a learning reinforcement during the performing of the scenario.

And this again helps you to avoid to use too many colors. Red colors, green colors, to give the user the confirmation that something happened. And by relying on the touch channel, you can simplify the UI of your application. And focus for a clean experience for the user by leveraging the tactile channel.

To give the user relevant informations. If you want to know more, we have a full Udemy course for haptics design in XR, with a more expanded version of these slides, and five practical use cases in Unity using Interhaptics, which you can download for free. Interhaptics.com, if you're interested in our application. And five different tutorials, which explain to you how to use the product for realizing one of each...

One use case for each one of the five categories that I have described before. UI confirmation, notification, remote presence, immersion, and the last one was... I can't remember it! So in this case, if you want to go more and dig deeper, and start your exploring, you don't have to think and you don't have to learn.

Start from zero. We want to give you all the learning resources that you need to start to approach the haptic design for your application. Specifically for accessibility. And I close here with a couple of takeaways, that haptics can really reinforce and enhance the user experience in XR.

And the process to think about haptics in VR or in XR can take inspiration from reality, but they need to be reimagined to understand the constraints of haptic technology. Pretty much in the beginning, people were simulating a way to do good VR. Widespread through experimentation and user feedback. In this case, we want to give you a lot of best practices already done, and you can build upon them without the need to do too much exploration.

And the third one is that we in Interhaptics provide a haptic composer, which allows really fast interaction of haptics for any implementation and allows you quickly to use haptics in your VR application. Particularly in this case for accessibility. And I thank you all for your attention. Tim, I give you the stage.

TIM: Thank you, Eric. That was lovely. I think that's gonna be a really great introduction to my talk. I really appreciate how much you spent on this.

And I'm gonna delve now into a specific use case. The past four years, up until April of this year, I had the opportunity to work at Magic Leap. On a mixed reality wearable. And so I was on the UX team that designed everything from the operating system, the notifications, and so forth, to LED patterns on the hardware, to the haptic patterns.

We worked closely with the Sonic Guards team to create sound design for the OS too. So I got to touch on a lot of different efforts. So what you see on the left here is the Magic Leap 1 system, composed of a wearable light wear. There's also a handheld controller.

Which is optional to some of the experiences. Because you can interact with voice and gesture as well. And then on the belt pack is a battery, processor, audio/volume controls, et cetera. And then on the right is a close-up of the control.

Which we'll go into more, since that's what we'll be talking about specifically. A little bit about my background. So I mentioned that I had worked at Magic Leap. Currently I'm director of product design at Faceware.

We do facial motion capture. Basically we have hardware and software that allows performers to capture their facial motions and map them onto 3D avatars. Before Magic Leap, I worked at IBM Watson, dealing with data viz and AI. I wanted to go ahead and show you Faceware Studio real quick.

This is Faceware Studio hooked up to Unreal. Our app on the left -- you can see someone wearing a head cam, with a camera pointing back at their face. And then to the right, you see Unreal Game Engine, with a 3D model of the face, mimicking the live face. So that kind of explains conceptually how this all works.

In one slide. Recently, we had a virtual concert on Roblox, in which Lil Nas X performed a concert and there were 33 million attendees. You can see him on the right, wearing one of our head cams. I'll talk a little bit about my XR advocacy work.

So during my time at Magic Leap, particularly the last year and a half, I worked closely with our Leapable Employee Group, and we were involved with internal and external advocacy efforts around XR accessibility. There are some examples here on the slide. Volunteering with the Dan Merino foundation. Recently, I broadened my advocacy, but I'm happy to share some of the things I learned at Magic Leap with you all.

Talking about the Magic Leap use cases, here on the top left, you see a social use case, called Social. Also known as avatar chat, in which you as a user can wear this device, and see virtual avatars of your friends, and you have the ability to converse with voice, with emojis, by moving your bodies. So that's one app that I worked on peripherally on the UX team. I was working on some of the haptics issues and so forth.

Top right is a screen. One of the use cases most people think of with a virtual reality wearable is showing a huge display in which Netflix is playing, or HBO on your wall. We had an app that was doing that. Bottom left is a virtual keyboard, which I'll talk more about in a second here.

Worked really closely on that. And bottom right is our launcher, our landing screen for the OS, with all the applications around it. To talk about haptics and the work I did with haptics, I'm gonna go a little bit into the symbolic input aspect. When I joined Magic Leap, my primary task was to design and prototype a virtual keyboard on which people could type with mixed reality, within mixed reality, rather.

I was given prototypical control, and some other engineers on the team were able to help me towards realizing this first virtual keyboard prototype. What I end up doing with the control is leverage the different components on it. So we have a touch pad that has force sensitivity. So that was one mechanism.

There was a digital bumper on the front, which is just simply a button. And then on the bottom is an analog trigger. And then we have a home button on the top. What you don't see on the components, but is labeled more to the whole device, is 6DOF, the ability to point in space.

That's what I do to define the virtual keyboard. Here are some examples of other forms of text entry that I helped lead the design of on our team. The virtual keyboard being on the left. So you can actually see an animated GIF.

I'm gonna show you a larger video where you can see an example of the numeric virtual keyboard. You can see the edge of a controller and a 6DOF ray pointing at virtual buttons, and the user -- what you can't see is the user is pressing the touch pad or trigger to type those characters. And there's haptic feedback there. The mobile app keyboard -- a pretty straightforward iOS or Android keyboard.

Definitely some interesting roles tied into head pose with the device. So you'll be looking at a particular 3D object, and if it had text entry capability or cursor capability, the mobile app on a phone could be used to perform text entry as well. Bluetooth keyboard I worked on heavily. And then voice dictation on the right, which was wrapped into our virtual keyboard, and one of the things from an accessibility standpoint that was really important to me and my team was allowing users the ability to shift seamlessly between these different modalities.

So at any given time, you can pick up one device, drop another, cursor control transitions to the new device. I'll go ahead and show you... I'm gonna go ahead and mute that video for a second. Since I can kind of do some talking over it.

Basically what you're seeing here is a virtual keyboard. The user has gone into settings and they're changing the name of the control. To do that, they must point the control at a text field. And press trigger or touch pad.

And then they're able to type by moving the 6DOF cursor on those keys and pressing again. There's haptic feedback for hovers, button presses, key presses, and so on to enrich this experience. Right now you're not seeing the 6DOF raycast. That's because the thumb is moving along the touch pad and moving that cursor.

Go ahead and shift the... One second here. I'll go ahead and shift the conversation to sensory feedback. So now that you understand the use case of typing in mixed reality, we can go into, like, how we leverage sensory feedback to help reinforce these interactions and make them more accessible.

So for starters, you obviously have the visual display. And graphics on the display. So you have additionally LEDs on all the hardware. There are 12LEDs around the touch pad of the control, which are interesting.

Often the trick is getting the user to pay attention, to notice them, because the control might be out of the way sometimes. But that's another reason we reinforce these interactions with haptic feedback as well as sound feedback. And I was in a unique position to work on all three of these efforts and kind of coordinate the design across them. As you can imagine, some of the more important patterns featured both visual haptic and sound feedback all at once.

For instance, pressing a button, for instance, is something we very much want to accentuate with all three. Whereas some of the others are purely visual or purely sound. In some cases, purely haptic. Here's a video of the controller.

Mostly sharing this with you just so you can see the LED patterns on it. You'll see the labels on the bottom right. So this is pairing. And this one here is a sleep pattern.

You can kind of see this slow breath. Shutdown is a sunset. Kind of radial sunset. I'll go ahead and play that one more time, just in case.

The reason I'm sharing that is because I'm gonna show some haptic patterns that work in conjunction with these. There's pairing again. Two blue chasers around the edge of the touch pad, joining. Sleep is a slow kind of white oscillation.

And then the shutdown, which is the sunset colors. I'll talk a little bit about our process to evaluate and make decisions in terms of both haptics... In terms of the components and the types of use cases we outline. So I was involved at a pretty early stage.

I mentioned -- I joined when we were looking at the controller and we had control prototypes. But we did not have a permanent haptic solution in the control. We knew at some point moving further along, we would have to make some decisions. In regards to that.

Before we entered mass production. So the UX team, mechanical team, electrical team, and human factors team all worked together to evaluate some different haptic options. So we looked at LRA haptics, linear resonant actuator haptics, which are probably best described as springed pistons. And also ERM, or eccentric rotating mass haptics, which you can think of as a kind of lopsided haptic motor.

Moving around circularly. The nice thing about the ERM is that it's very, very powerful. And great for, like, rumbly sensations, but it doesn't fire up really quick. It doesn't have the snappiness, nor the impulse that an LRA provides.

We also -- problematic with the ERM -- was this repeat typing scenario. Or repeat fire scenario. So if you were on a keyboard and you were typing very fast, or typing a few characters in a row -- you would want to feel those as distinctly different haptic sensations. The ERM doesn't fire up quickly enough to accentuate each of those presses.

So that proved to be kind of a deal breaker. In the end, we went with an LRA haptic called ToughType. To program those haptics, we built a Unity prototype in which we could alternate two variables. Pulse count and pulse time.

And it sounds archaic. But between those two variables, that was enough for us to generate the different haptic patterns across the OS, of which there were about 20. And I'm gonna show you some of those next. But before I get there, I would like to mention one of the things that was a real challenge was dealing with things rumbling and shaking inside the control, and that impacted users' perception of premiumness with the control.

So we had to try different frequencies that were aligned or misaligned with the resonant frequency of the control itself. To get consonant and dissonant results. So yeah. Just kind of to bring up the control itself -- it's another instrument.

The body of the control is part of this haptic instrument. And that became very clear. I'm gonna move to cover some of the specific haptic patterns in the OS that I find most interesting. So I don't know if I'll be able to share my camera on screen today.

I didn't see that option. But I can do my best to vocalize and describe these haptics in case we're not able to go that route. So the first pattern is called starting up. If you hold down the home button on the control while it's off, that's the little round button below the touch screen, for any duration, it's gonna fire up in about two seconds and you'll see what we call a sunrise pattern.

While it's firing up, and by the way, it immediately glows bright blue to let you know it's on. Before that sunrise effect kicks in. While it's climbing up, you're hearing a bllbbrrbbllb... Ascending pitches spanning about 2/3 of an octave.

And I'll go ahead and vocalize that again. blllbbprrrbblll... It probably looks really interesting with my bird avatar making those sounds. Conversely, shutting down is also 2/3 of an octave, but descending pitches, and that sounds like this... pllbbbbllbbbbll...

The actual animation is a lot quicker on the device. I don't know why, but that particular GIF for shutting down is quite slow. And then charging, if you were to plug in the control to a charge source, you're going to hear... voom-voom, two small lurches, about 0.2 seconds apart.

And then you will see a graphic LED along the halo, or the edge of the touch pad, conveying the charge level. The color green, if you are able to see color, indicates a healthy level of charge. But if green is problematic for you, you can use the percentage filled of the halo as an indicator of charge. So in this case, we're about 80% charged.

And then lastly, you know it's charging, not just displaying charge, because you see it oscillating on that last level of charge, implying that it's increasing from 80% to 81%. There's also a UI tap on the bottom right. Whenever you interact with the touch screen, sorry, touch pad on the control, there's a nice glowing effect around the touch pad to let you know it's picking up. It's also nice just visually in terms of making the device frankly just look cool.

To see someone playing with the thing and seeing it glow as they use it. It doesn't consume too much electrical power. Here are some more exciting haptic patterns. I mentioned the UI hover and how important it was.

The GIF on the top left shows that. So as the user is pointing at a button with 6DOF or via the touch pad and moving across those patterns, you're hearing... Thump-thump-thump. Very soft.

It goes such a long way to accentuate the experience of typing on the device. Or using any kind of buttons. Just having that feedback is huge. We don't even use sound all the time with these kinds of hover sounds.

But we always accompany with hover. And you notice it very differently when you use, for instance, a touch pad on a physical keyboard or you don't have that haptic feedback. Going over a virtual button. You will notice it right away, that it's not there.

A variation on hover is this UI force press and hold. If you are to hold down the touch pad for 0.8 seconds on a character on a keyboard, you're going to see an alternate series of characters show up. This is a context menu, and a kind of vrroom... Accentuating that context menu popping out.

When you highlight an object or prism, you see a whale getting pulled out of another volume that contains the whale. This is a 3D model of a whale popping out of that volume. It's accompanied by popping haptic and sound. And then you have social fist bump.

If you're an avatar, and you're chatting in avatar chat, someone can put their real life hand out in front and the virtual avatar conversing with you can fist bump you. So there's a nice fist bumpy sound. A sound effect of party streamers and visuals, and from Eric's talk, this is an example of remote presence in haptics that I find very interesting. Lastly, I just want to close with a slide.

If you're interested in developing your own haptics, there are resources on the Magic Leap developer site. I didn't work on this particular effort. But if you're a Unity or Unreal developer, you can play around with haptics in your own experiences. Thank you very much.

And I appreciate it. Thanks, Eric, for having me again.

ERIC: I think we have a Q and A. Correct?

TIM: Can I ask my question first?

ERIC: Of course.

TIM: I'm really curious about the Haptics Composer application. Something like that would have been really useful when we were working at Magic Leap, creating these patterns for our control. I wanted to know if you could describe some challenges in developing that, in a way that was user friendly. It seems to be quite a challenge to create something that visualizes something that people are used to feeling.

ERIC: Thanks for the question, Tim. It's a challenge that is not yet involved. It's an ongoing challenge. The biggest problem that we faced is how we were able to represent haptics in a visual way.

And what was the way to do that? And this is where we spent most of our activity. The design process is something that's pretty much on how the haptic rendering in the background works. But how we were able to give them a preview of haptics without giving them any haptics feedback.

For this reason, we found three methods. One for each one of the different mechanical feedbacks. Textures, vibrations, and stiffness. For vibrations -- for textures, we have the representation of the surface.

So we see here in the background with the video, on the bottom right, we have a 3D preview of the virtual surfaces that the user would feel. For the vibration, we have a visual preview of a square that was smaller or bigger as a function of how strong the haptic was. And you were able to visualize the pattern. And the force stiffness -- that is still the hardest today.

How we are doing it... We have a cursor that as a function -- it shows how a virtual finger would press the material. And what would be the force profile. That we will see.

So these are three ways we have found how to represent them visually. Haptics over there. Does it answer your question?

TIM: Yeah, thank you. I'm interested in exploring the software more. Did you have a question for me?

ERIC: Yes. And it's about how long was the... If you can share, of course... How much time did you take to nail your haptics experiences?

And what was the activity that took most of the time?

TIM: That is a great question. Yeah. I didn't do such a... I didn't clearly break down the timeline.

So when I started at Magic Leap, I would say we had a haptic in our control that was just kind of there for the time being. As like a temporary solution. And that would have been late 2016. By early 2017, we were working together as a company to figure out a more final solution.

We spent... That said, we spent months and months, probably three or four, just evaluating haptics. So we would be -- not myself, but people who worked in this area -- would take a haptic down to the basement, into the lab, and conduct tests on it. And provide feedback in terms of how quick the impulse was.

And once we proved that these haptics were responsive enough, they would get put into the control itself. And we would get... We would feel like, for instance, how it felt to have a haptic in the body, versus closer to the touch pad. We got to try out all these things.

But that was also temporary. At some point, we had to figure out how to permanently mount the haptic into the control. And that wasn't my job, per se, but I was someone who was basically -- would evaluate whether it still felt premium after it was mounted in the proper place. We probably spent more time doing that than even coming up with the haptic patterns themselves.

The whole time we were working on this, we were obviously thinking about where this was going. In the end, with actual creation of patterns, I would recall creating the list of patterns, of instances -- mostly interactions, but some notifications -- and wayfinding things that I thought would be crucial for us. And then some more Easter eggs. Some fun things.

And so this list -- with this list in mind, we basically went with a prototype and just started pressing the control and watching the onscreen graphics. And feeling those vibrations. You know what's interesting? As the sound design team created their contributions, we refined it even more.

The sound designers said -- we need to bring the ramping path up a whole step so it's consonant with the sound design. We would make adjustments with that. We would also be adjusting duration. I remember one time -- the first time I ever had a control in my home, I was used to having it in a loud office, but I brought it into my home to evaluate.

I fired the control up, and the vibration was so strong and audibly loud that I knew right away it was... It wasn't going to be premium to someone taking this out of the box. So at that point, we dumped... We dropped the amplitude by half.

But yeah. Eric, sorry, longwinded answer to your question. There were a lot of refinements along the way, with every addition. Hearing the sound and just seeing the graphics would sometimes result in a change.

But we didn't have a lot of flexibility. Kind of like what you had mentioned. Being able to create the feeling of silk. We were far from that.

We were primarily using haptics as a way to accentuate interaction.

ERIC: Cool. Thanks for the explanation. It's insightful for us to know how -- what was the designing process for that. But I think it's really great.

Thanks for sharing.