Asymmetric Remote Collaboration using Tangible Interaction in AR
Over the course of a semester, I developed an asymmetric remote collaboration tool using object tracking on the Apple Vision Pro.
Object tracking - the ability to track physical objects in your environment and attach digital content to them - enables tangible interaction methods that allow users to control the digital environment in a very intuitive and natural way. However, problems arise when using tangible interaction for remote collaboration, as two remote users cannot control the same physical object.
In this project, I explored the potential for tangible asymmetric collaboration - scenarios where there is a “local user” that controls the tangible object, and a “remote user” that cannot move the object but can see a visual representation and provide directions.
The project is coded entirely in Swift, and uses Apple’s SharePlay system to set up networking between remote users. Local users broadcast the information necessary to construct digital visualizations of tracked objects, while remote users can create targets to guide the local user. The system supports multiple objects at once.
Asymmetric tangible collaboration has potential for physical rehabilitation, design visualization, and board games. In a game of chess, each player could move physical versions of their pieces, while seeing digital visualizations of the other player’s pieces.
Cornell Tech
Independent Research
Fall 2024
A video demonstration of object tracking & asymmetric remote collaboration on the Vision Pro.
During remote collaboration, users see each other’s digital avatar, alongside a digital visualization of the box that the other user is holding.
The physical object is tracked and brought into the digital environment. A red hologram is created as a target for the user to move the digital object towards; the target turns green and disappears upon being reached.
Enhancing Gaze+Pinch Depth Control using Microgestures
Over the course of a semester, I developed a microgesture-based technique that enhances depth control for the gaze + pinch interaction method.
“Gaze+Pinch" - using eye tracking to select the object you look at when you pinch - is a popular selection method in XR, due to the minimal hand movement required. Not having to move your hands to select things makes it an ideal technique for crowded spaces and public transportation. However, this only applies on the initial selection, and moving grabbed objects still requires significant hand movement.
To resolve this, I created two control mechanisms using microgestures - secondary finger actions separate from the primary hand action - to extend and enhance depth control. These enable objects to be placed anywhere in the environment while barely moving the hands at all.
Cornell Tech
Independent Research
Fall 2024
A video demonstration of microgesture depth control for gaze+pinch.
In the demonstration scene, the user uses microgestures to grab and move cubes into far-away target spheres with minimal hand movement.
A visualization of the two control mechanisms: “pinch movement” (moving the location of the pinch closer or further from the palm) and “index scrolling” (scrolling the middle finger along the top of the index finger).
Multi-Object Image Plane Lasso Selection
Over the course of a semester, I developed a multi-object image plane lasso selection technique as a Unity package for use in VR applications.
Users of this technique can draw a three-dimensional “lasso” in the virtual environment, and any objects that appear to be within that lasso from the user’s perspective will be selected. Basing selection on the user’s perspective minimizes selection errors while allowing selection at any distance, and drawing is a simple and intuitive method to quickly define a selection area. Users can also draw a line without completing a full loop, and anything that lies along the line from the user’s perspective will be selected.
I coded scripts that enable the user to draw a lasso or line through the scene, and will then project selection rays from the user that pass through the lasso and into the scene. It was designed to be compatible with other packages and easily integrated into any XRI scene with minimal effort.
Full documentation was also created for the package, complete with a manual and scripting API.
Cornell Tech
Independent Research
Spring 2024
A video demonstration of the lasso selection technique.
This debug tool visualizes the raycasting technique used in selection. Rays originate at the user’s eyes and pass through the lasso (yellow). Rays that hit an object (red) select that object; other rays pass without hitting anything (orange).
Other settings include controlling the input method & visuals.
A demonstration of the lasso selection technique, where the user has drawn a complex lasso that selects specific objects (highlighted in yellow) while avoiding others.
The user can customize the detection thresholds and raycast settings, and can also define which events should be triggered upon objects being hovered or selected.
Radial Marking Menu for VR
Over the course of a two-week full-time research position, I developed a radial marking menu as a Unity package for use in VR applications. The menu is fully customizable by the user.
Navigation and selection is simple and intuitive; holding down the menu button opens the menu, and you can move your controller to select an option. Hovering over a submenu (marked with an outwards arrow) will take you to that submenu. Releasing the menu button while hovering over an element activates that element, triggering its corresponding Unity event. The activation of functions through a single connected movement allows for the development of muscle memory, allowing for users to access tools and options without thinking.
Users can create menu items with icons or labels that trigger public functions in the Unity scene, as well as create nested submenus to organize them. This allows the marking menu package to be used for practically any purpose.
In order to allow the UI to seamlessly accomodate user settings and change on-the-fly, I programmed a custom shader that renders the back of the UI entirely through code. It modifies the number of sections, which sections are submenus, and the size of the menu all by changing the parameters attached to the shader.
Full documentation was also created for the package, complete with a manual and scripting API.
Cornell Tech
XR Collaboratory Research
Winter 2023
A video demonstration of the package. Primitive shapes (both in the background & attached to the tool) are used to demonstrate the menu’s functionality; selecting a shape in the menu triggers functions in the scene that will make the corresponding shape in the background teleport, and will also attach that shape to the controller.
The package has an optional “expert mode” where the menu will start out hidden (left) before expanding to show all icons after a short amount of time (right). This encourages development of muscle memory by discouraging reliance on reading icons, and allows for expert users to navigate & activate functions without ever opening the full menu, reducing distractions.
A large amount of settings are made visible in the inspector for users to customize.
A demonstration of the submenu system; hovering over “More Options” (right) will lead you to the left menu, while hovering over “Back” (left) will take you back to the right menu.
The menu customization system; the user can define an arbitrary number of menu elements, each with names, icons, and events, as well as an arbitrary number of submenus.
Creative Tools for VR
For our final project in INFO5340: Virtual and Augmented Reality, I worked in a team of 4 to create a 3D modeling app in Unity for the Quest 2. We were tasked with creating a modeling app that replicated features and interaction techniques from Gravity Sketch, a popular VR app. Starting from Unity’s basic XR Interaction Toolkit, we implemented a swath of features enabling complex navigation, selection and manipulation of objects.
As the only member of our team with previous Unity experience, I took a leadership role in the team, delegating tasks and helping my team learn how to use the program more effectively. I also assisted with debugging & troubleshooting throughout our project.
My other responsibilities included:
Implementing a 3D color visualizer to change object colors
Implementing undo/redo using a command-based system
Implementing an input action controller & state manager
Using Blender to design and model custom assets when needed
Making numerous design/UX improvements throughout all of our app’s features
Our work was chosen as one of the top 2 projects from the class to be showcased in Cornell Tech’s open studio event.
This information can also be found on our project page.
Our feature walkthrough, where a team member describes and demonstrates all of the features we implemented.
Cornell Tech
INFO5340: Virtual and Augmented Reality
Fall 2023
Our app demonstration, where a simple car is built from scratch in 5 minutes.
Highlighted Features
Users can rotate the thumbstick to unwind time, traversing the scene’s edit history quickly and intuitively.
In mesh editing mode, users can grab vertices to edit primitive objects and create more complex meshes.
Users can easily change the color of grabbed objects by moving their controller within a 3D color visualizer.
In scaling mode, users can grab constrained axes to quickly adjust the size of objects.