Add a new Voice first focused UI mode that is entirely voice controlled

[Anyverse]

Using the latest language model from Meta create a new focused mode which is voice controlled. This mode can be invoked manually by prompt or via user interface shortcut on the main Menu.

VR is the ideal use case for an entirely Voice navigated UI since our hands are holding controllers and it can be impractical to switch between physical keyboards and motion controllers for productivity without switching to hand tracking with a virtual keyboard for a natural experience.

How it could work

This mode would automate actions when prompted to perform a task.

• Open an application
• Search for something (online)
• Change a setting for me like pass through
• Automate creation of play area
• Enable developer mode
• Download an app that handles 'task'
• Help me to perform a task
• How do I use this feature?
• Open my photos or videos
• Create me a slideshow of my best photos
• Recommend me 'type' of game/applications

Detailed Description

1. User Interface

It would switch to focused mode via an overlay or modal dialog on top of the standard UI when invoked by voice prompt or clicked manually.

2. Voice text entry

Voice input for any text fields in applications and web browsers regardless of implementation e.g. WebView, WebXR, OpenVR, OpenXR, game engine.

3. Language model assisted navigation

Ability to search for content by voice prompt or automate the actions based on responses given from the new mode.

• For example; "I would like to find something" and the language model creates a detailed search criteria based on our brief input.
• Another example; User prompt = "My headset is performing slow today" the Language model understands the question and performs a search using a relevant search query; "Why is Meta Quest 3 performing slow in v64". Then it applies the suggested actions it finds from online sources to solve the issue with the hardware.

These would not require any further input except requesting the user for confirmation for performing suggested actions on the device.

4. Refined Querying

If the query is not understood because of ambiguity, it will ask the user to refine the query e.g. in the instance when an application is installed locally but something by the same name exists online, it may ask "Should I perform commands on A or B item?"

5. Automated Scripted Actions

• While deciding possible solutions from the websites it would suggest a list of possible supported actions and with the users confirmation it could refine those down to a list of possible solutions and apply those on confirmation from the user.

• When the action is not pre-programmed, it can define its own script using a new prompt to UI interaction scripting language much like automated testing scripts like selenium. This would have its own validation and guided troubleshooting mechanism to verify scripts are correct and suitable.

6. Offline basic voice functionality

When there is no Internet it could fallback to an offline mode which can still handle voice input, voice navigation and automate setting changes. With the full experience available when connectivity is restored.

7. Fast Updates

This can be implemented as a service or separate application that is not dependent on any version of Android. This makes it easier to upgrade to newest versions of Android for Meta Quest.

Since the mode would not be tied to the operating system, it would be possible to update this component via the Meta Quest app store so the update cycle can be iterative and frequent.

8. Developer API's and Integration

Integration of this feature would be facilitated by requiring accessibility Metadata to be added to user controls for apps to be accepted in the app store or by creating a new API that enables this. This would allow the feature to automate text entry in text box controls and comprehension of the use case and context for correct action.