Note on designing for compelling gaze interactions

[Ross_Beef]

With the introduction of gaze targets and eye behaviors into the avatar SDK, I thought it would be good to outline how gaze works, and some thoughts on gaze targets

We've built gaze targeting to be very simple in the first version of the expressive update, enabling both UE and Unity developers to tag / add a component to objects in the scene and have it register as a target.

How does gaze work?

• Our gaze system takes into account both moving and stationary objects in the field of view of the player's avatar and predicts a likelihood that gaze will switch to one of these targets. Factors such as proximity, movement (eventually, more granular targeting tagging) are used to score each target.
• In user research we observed that the focus of a user is rarely more than 10 degrees off the direction indicated by their nose bridge / headset orientation, mostly as a function of the lens construction of VR headsets, which ensures that the clearest point in the users' field of view is quite narrowly defined. This means that, in general, you can get a good sense of what someone is likely to be looking by assessing objects within a smaller field of view than their actual FOV, users normally move their head more to focus on objects in their periphery than they would do in the real world.
• This simplifies the problem of gaze targeting considerably. It also makes it much easier to rule out some of the uncomfortable edge cases of social experiences, where another player's eyes seem to follow you right to the painful periphery of how their eyeball would rotate.
• In terms of simulation, our gaze system models different types of eye movement: fixations, pursuits and saccades:
• Fixated – Gaze is focused on a gaze target. This state periodically
  triggers micro-saccades, small jerk-like movements that occur while looking at
  an object as the eye adjusts focus and moves its target onto the retina.

• Smooth Pursuit - A movement where the gaze smoothly tracks a gaze
  target across the Avatar’s field of view.

• Saccade - Fast, sweeping eye movements where the gaze quickly moves to
  refocus on a new gaze target.

• In the absence of any gaze targets, the eyes will move periodically about a number of set points in a distribution in the users' field of view, modeled on the way that eyes generally move to take in a scene. Other avatars heads are already tagged as targets.

Optimizing for gaze

In social contexts, gaze should primarily be about other people and the objects in the scene. While gaze targets are already set for the avatar, it's worth thinking about how to most effectively design for interactive objects and points of interest.

What we've found most compelling about the gaze system is those moments when an avatar turns to face you, and their eyes very clearly re-focus onto you (the first person), moving ever so slightly separately from the head motion and picking up the focus in a way that feels organic. It's also been great to use gaze as an indicator of attention shifting, particularly when the user is interacting with UI that someone else might not be able to see (e.g. a control UI that's in front of them).

Considerations being:

• What I'm holding. If a user is interacting with an object (e.g. a bow and arrow), having this object be tagged should mean that as it is picked up, it might count as a gaze target for both the user whom is holding it and also others that they are interacting with.
• What I'm pointing at. Extending targets to what is 'in use', you might think about making any UI element or control that is being pointed to into a gaze target, in circumstances where you're using raycasting from the finger to interact with components in the scene. Thus, if I point 'over there' and highlight an object in a scene, it can then be considered as an object that I'm looking at.
• What I'm interacting with. Similarly, if we're playing monopoly and I'm clearly interacting with (and therefore likely looking at) the cards in my pile, articulating these shifts in attention through gaze can be particularly effective.

How are you thinking about using gaze targets in your app?
What additional control would you like to have over gaze?
We'd love to hear more!

[owenwp]

It would be useful to have support for one shot attention grabbing events. For instance, an explosion that starts with a very high gaze score, nearly guaranteeing even distant avatars will look immediately, then decays over time. It could be used for a lot of time sensitive events, like objects playing audio cues, sudden visual state changes, etc.