VR Interactions – A Comparison Between Gaze, Controller and Hand | Accessibility VR Meetup Recap

SAM QORBANI: Hello, everyone. It is a pleasure to be here. Thank you all who are here today. This is my first time presenting in virtual space, so, from up here, it seems like an out of body experience and I think it is very funny feeling.

I didn't want to turn today's presentation into a full academic one. It is going to be a bit boring so I am trying to kind of find another angle to it but I still show you what I have been learning in the last couple of years about VR. I am a third-year PhD student at Carltan University in Canada. And VR in the S.T.E.M. education as a broad category is my research topic focusing on safety procedure in a chemistry lab as an example.

I am hoping to pursue accessibility and inclusiveness to explore by my research making VR educational content and training available for communities in Canada. Today I will be talking about my recent experiment that came out of a school course on VR. I took this course last term and it involved exploring interactions in VR. By VR I am specifically talking about real-time Immersive Virtual Reality or IVR IVR and not 360-degree videos that we normally see.

I am curious, how many here are involved in teaching VR? Any person? Any audience member? What about developers?

Or designers? Oh, I can see one hand.

I am a developer. Any content designer in VR? OK. So I am hoping that I can bring something or have something for everyone.

Thank you. Yeah, I can see. Nice, designer UX. Dev design.

Design student. Great. Thank you. I structured today's presentation around these couple of questions.

What was done for this particular topic -- disclaimer I couldn't do a full user study due to the COVID restrictions but we are planning to do it remotely and send it APK for people who have it. I will talk about why I am doing this particular project and then how it is relevant. As a part of the courses recommended reading, I came across this paper called the Ayes -- eyes don't have it.

The eyes don't have it: An empirical comparison of head based and eye based selection in virtual reality. Eye-only selection offered the worst in terms of error rate, selection times and throughput. Head-only selection offered significantly better performance but they used ISO to do the experiment and I quote the author said we found that head-only selection offered the fastest selection times and the best accuracy. I will explain what I refer to as selection time and whatnot.

OK. Based on that paper, it kind of triggered by interest. So my fellow PhD student and myself, we took an interest to explore the next step of the aforementioned study to see if there was a practical base for the head selection. We added a docking task to it.

In general, the task include three sections. Selecting, dragging and dropping. Over the course of building the study, it became more of a comparison between three types of interaction in immersive VR. The first on the slide is head based.

And the controller is only for triggering locking and unlocking the object. Second one is control-based. One control is just for trigger. The other one is just for shooting the and dragging and dropping the object.

The third one is hand-based. That is still a work in progress because it later changed and we wanted to still have the controller on one hand and just use the other hand for using the cast to drag and drop objects. Just have to mention the limitations of using this is that it lacks a long stick that, a long pole that you stick an object at the end of it, so as you can imagine it is not as flexible.

It wouldn't bend. You can rotate it but it wouldn't bend. That's the limitation of using that. The next topic I am going to talk about is why I am doing it specifically exploring this area.

Motivation is a big part of what we do, isn't it? For me, the journey began in 2005 when my wife and I moved to Canada. Starting over as an immigrant in a new country is not always easy. Canada embraced and accepted us as skilled workers but still we had to start from scratch.

Our humble beginning was like many other immigrants. My wife educated in applied mathematics found a job in a store as a cashier and my IT background landed me a job in a computer repair shop. Here is where my stery took a turn. A small misunderstanding about the repair of a computer with a customer created an issue between me and the customer we know from now on as John.

I always say we had many firsts. John, and I and my wife pun intended. The first time I road an ATV around north of Toronto, or riding a large inner tube down a snow-covered hill in minus 20 degree temperature, or traveling to beautiful PI to his cottage, to the deep sea fishing where my wife and I thought we were going to die due to the motion sickness and not to mention the first time eating fudge during our trip to niagra falls.

He also taught my wife kayaking or canoeing. I really don't know the difference and I don't care as I am not an outdoorsy person. This is a picture of my wife that took from him during one of their adventures. I even had my first ER visit in Canada with John and trust me there is a funny story about it.

Ask me at the end if you are interested. I should give a shout-out to our health care professionals and frontline workers dealing with the pandemic situation now. Before I forget, John is also an excellent handyman. With limited help, he built this ramp at his house.

The only thing I have ever built was a bathroom stool that fell apart when I sat on it. John doesn't make me look good in front of my wife when it comes to building stuff. I can go further about what John can do. But why I am telling you all this?

Well, because John has a rare progressive disease that impact his health. It changes the shape of his bones especially fingers and toes. Every few years we has to go through a surgery to reshape them. I don't want to be more graphic about it but it is not easy as you can imagine.

Good thing is, John still can walk with the help of a cane but that possibility is decreasing day by day so he mostly uses an electric scooter to get around. Beside everyday challenges he is facing around the physical world such as stairs with no ramps, no accessibility door buttons that are out of reach, he has to deal with digital challenges as well. John has always been a good computer gamer but until now he couldn't use a VR because of the interaction device, the input devices.

As you know, the forms of input devices in VR is just one example of the challenges he and probably many other users face. So, I just explained this to say that I don't consider John as a person with disability. I believe he is a person with different abilities. Do you remember this big red button I showed you a few slides back?

It is actually a mouse click he uses along with a joystick to use his computer. I didn't do background research about it to know who first came up with this design but I can bet it must have been long-time after the regular mouse used every day was designed and produced. Like always, the norm seems to be let's design for normal, and able-bodied people, and people with special needs will adapt. Here are a few things that inspire me.

It is the year 2020 and as predicted by researchers in 2017, the state-of-the-art of technology VR is already out of date today. It was the same year when CEO of the unity game engine said that in order for AR/VR and XR become an industry three components should come together. It should be mobile. The headset should be under $1,000.

The third part is VR content. Where we are today with VR is just the beginning. As we are exploring more and as technology get better, faster and smaller, the size of headset will reduce and it will be less costly. What we can achieve in terms of application and content is limitless.

The VR technology we see today is the result of decades of research and advancement. Hopefully, we can include a wider range -- of users in the next phase. Some of the benefits of AR/VR include the immersion, multi sensory, and sense of presence amongst the characteristics are sited as the strongest teachers. They are showing positive affects on learning, providing the possibility for active learning and increasing user engagement and improving spatial skills.

It is also suggested immersion in immersive VR increases learning effectness, engagement and learning attitude. Motion sickness is highlighted in the literature. Many factors need to be considered in both design of hardware and software application in VR. Why some of the devices currently designed for VR may not be adequate Leo effective for certain groups who am I to say it is a bad design.

No designers intentionally design a bad app, product ever service. Internal and external forces can impact the design and hey, there is always a deadline to be met. I am trying to say the design decisions need to go beyond what they are now. We can be more considerate to include people with different abilities to be able to use the product and application we make.

It might not always be possible but we can and we should try harder. I want to bring a bit of awareness that we as business owners, designers, developers, everyone who is involved, should come and could do better in terms of building and designing for accessibility and inclusiveness. Here are some of the timeless design principles applicable to any design from my favorite author. Maybe I can talk about how I use them in another session.

And the last, and again, the last word from experience that is through the video I will play. I will show you, sorry. The last word from experience, if you are developing for mobile immersive VR, be mindful that will first VR experience is extremely important especially if they are first-time users as a good or bad impression can form their opinion about IVR. If that, they may never try it again.

Here are a few technical tips. Specification and limitations of the platform you have designing for or developing for. Be aware cyber sickness is real. Use game engine to achieve the best and optimal framework and maintain it.

Oculus Quest, for example is 72 frames per second means each frame has to be rendered in 13 -- 1-3 milliseconds. I got away with 17-24 but be very careful. Use compression. Use mesh combine and every trick possible to avoid latency.

Avoid using large textures. Immersion in IVR can occur despite of unrealistic environment and that is based on the studies. You might get away with having a simple solid color instead of a very detailed texture on your 3D objects. Avoid shadows and reflective material and textures.

And fast-moving objects since fast head movement causes nausea for some users use alternative methods. Walking is always preferred but may not be possible. Don't make speed changes violently. There are lists of best practices around VR but you have to search and learn what works best in your scenario.

At the end, I have a video of the experiment we did design in unity just to show you how that head selection and dragging works. I am hoping we can play that one, Thomas?

Thomas: Do you want to do that one now?

SAM QORBANI: Yes.

Let me move it slightly forward. This is our first time trying a video in here so this is exciting. I might have to hide this. I will bring it back.

SAM QORBANI: OK. Great. Once the user is in this VR space, what they see is this three buttons and this menu explaining what they to do. I will play the video and explain over.

What happens now is they are actually interacting. That is the head tracking happening. Where they look -- once they are in the scene, they read a short description of what they are supposed to do, and then with this cross hair they can point to that object and then trigger on the left controller, move their head and drop it. That's an academic test.

It doesn't really -- it is not an application of sorts. Then we record the time from the time they picked an object and the time that they dropped it. And then pressing on the button takes them back to this menu and this time this is all done by controller and both the trigger and recasting are on one device but DWI separate them. Again, we record the time they take for them to select and drop the object.

And the last one which is the work in progress is actually casting by the hand. This version I have on this video isn't showing it in action but basically it is the same task. They have to select an object and drag and drop it to the target area. It is purely done for academic purpose to be able to measure the error rate and performance and all those fun stuff.

That was my presentation. Thank you very much. If you have any questions, I am ready to answer.

Thank you very much, Sam. I did want to let everyone know because this is our first time doing a video in the room that you basically, for future reference, at the top right of Mozilla Hubs section there is a number of objects. Right now on my screen it shows there is five objects and you can hover over the YouTube video itself to get that more full screen in your view. You can also do that for presentations.

I thought the video worked really well so I really appreciate that.

SAM QORBANI: Thank you.

I will start off with the first question. What were the findings of the three different interaction methods?

SAM QORBANI: Good question. Again, our user study is underway. We haven't completed it but I have done some testing with what's called simulation testing. I did -- let me come down.

From my interest, I sent it to John and tested it. On the first interaction tab, the gaze one, or the head one, for the first attempt he did it in about 12 seconds and then the second attempt he did it in 4 seconds but we are expecting to increase because we have to have a special application for this group of users, right? So it is still a work in progress. But for the other one, we expect that head tracking interaction would be faster and the more accurate one compared to others.

Did I answer your question?

Yeah, that's great. And is the timing of that number of seconds, what was like the optimal amount of time? Or is that good? 4-5 seconds sounds like a good amount?

SAM QORBANI: Yeah, that is good for that number of -- again, there isn't much to compare with. These are all new. So my goal is to find this and, you know, make it kind of a practical way of interacting. Imagine, because there wasn't any -- one thing I didn't explain is that the controller trigger that I talked about, John, cannot hold it in his hand, so I created a document station and put this Oculus Quest controller on it and all he did was press it to make the trigger action so it just opened up a way for him to interact in VR and he quickly learned it.

I don't have much to compare it with. We are hoping to test it in other ways and see what is.

Great. Thank you. If anyone has a question, you can go off mute or type it in to the chat.

So when you are testing, is all your testing done with mobile device only of Quest?

SAM QORBANI: Yes. Yes.

Excuse my ignorance but is there a reason for that?

SAM QORBANI: We wanted to make it mobile-friendly, you know? I think the future of VR headset is mobile. For this one, I just wanted to kind of make it more available on the cheaper, smaller devices. Oculus Go is the cheapest one.

I would like to test on a platform for accessibility purposes.

Hi, I wanted to ask. I tried to use some kind of head-based gesture as a trigger action? For instance, nodding?

SAM QORBANI: I didn't catch it completely.

A nodding action to confirm something.

SAM QORBANI: We talked about it but defining that kind of gesture is a bit of a challenge. We use unity but, you know, we use the Oculus Integration Package SDK so it has to be custom made gestures and we haven't explored it yet.

OK. Thank you.

Could you talk or go back over what you have to do to the Oculus Quest controller itself to make it easier to use for John? Or what the change to the hand controller?

SAM QORBANI: Sure. I don't have a picture of it here but, as you know, the Oculus Quest controller has a small stem and you have to hold it around your hand. Hold it in your hand and there are the joystick under your thumb and a few buttons, right? So the way John's fingers kind of are bent it can't hold it.

What I had to do, I printed a piece that works as a rest -- you can rest your hand on that and then this controller sits into a hole that I printed out in that piece and then it works like the big red button I just showed you that he uses as a mouse. The controller sits on this docking station and he puts his hand on top and press it on the top so the action, the trigger works, to lock and unlock, you know, and my idea came from game design awhile ago that one button, so I was thinking, you know, what if I create interaction pack that only needs one click pressing one button so that we can use VR.

That was the 3D printed piece. It is basically a box with a hole on it so the controller sits inside and you can place it on top of the table.

Great. Thank you.

ELGIN: This was a really good presentation. Thank you for sharing it. I was wondering what are your next steps in your research that you are looking at doing?

SAM QORBANI: Yes, great question. My Professor expects me to complete this with a full user study and then we are also working with the ethics application going on to me, I don't like these brands but in the ethics community, we call it a vulnerability group, so there has to be a special ethic application which takes 3-4 months to complete but to me, as I showed you, John is more than capable so I need the permission to build similar physical difficulties to test this on maybe like a game or a more meaningful application.

This task you have seen with the selection of dragging and dropping is purely for academic purpose but I am envisioning designing a game of sorts to, you know, test or the VR game for John if I make it.

ELGIN: Thank you.

Thomas: For the hand tracking use case, and I guess, I imagine like often with the hand tracking cases, applications might record or try to standardize a certain gesture recognition to say when we see the hands or fingers doing this particular movement it translates to this action. So did you have ideas around for example, for John's hand interactions for doing certain types of gestures?

SAM QORBANI: Yes. Again, in case of John, it would be very difficult because the limited movement of his hand/fingers but I don't see any problem of exploring that area. Definitely there is application for it. With now mentioning it, I can see possibilities, definitely.

There is a way to design special gestures and certain, you know, pre-sets that, you know, yeah. There is a recent update of the Oculus Quest I hear there is more possibility in this regard. I haven't explored it yet.

Great.

This is a great presentation. I appreciate it.

SAM QORBANI: Thank you.

Could you talk more about the head gaze function and the coding of it and how you did that?

SAM QORBANI: Sure. It wasn't really difficult. All you do is define raycast and once you do that, you can design the start or source area, and then the second object would be your dropping target area. You write a few lines of code and connect them together.

I can share. The idea is to put them on the GitHub and share public once we publish it.

Yeah, I would love to see that.

SAM QORBANI: Definitely. Hit me with a message or email if you are interested I can definitely share with you.

Are you going to post your email somewhere? Or did I miss it?

SAM QORBANI: Let me share with everyone here.

You can put it in the chat if that's all right with you or your Twitter.

SAM QORBANI: Again, I mentioned I didn't want to get into technical details because it would be boring, you know, but I can answer your questions in direct email.

Thank you. That would be awesome.

SAM QORBANI: No problem.

Thomas: I think let's all give a hand to Sam. Sam, thank you so much for your presentation today.

SAM QORBANI: Thank you for the opportunity.