CHI 2025 Interactivity - M.S. Thesis

Desk Console: Augmenting 3D Virtual Controls on Physical Desks for Immersive Authoring

Yeeun Shin , Seung Hyeon Han , Woohun Lee

Abstract

Immersive authoring provides a powerful 3D content creation experience in virtual reality (VR) by freeing users from the tedious loop of desktop editing and VR validation. However, complex control panels required for creative tasks often disrupt immersion with awkward or unstable spatial interactions. To address this, we present Desk Console, an authoring interface that transforms 2D control panels into virtual 3D controls on a physical desk, enabling tangible spatial interaction similar to real-world interactions. Desk Console transforms traditional control panels into 3D representations based on input types and provides passive haptic feedback through the desk’s physical surface. We demonstrate Desk Console’s capabilities through an interactive 3D scene design application.

Desk Console Interface

Desk Console is an immersive authoring interface that transforms 2D control panels into 3D virtual controls on a physical desk, enabling spatial interaction like a mixing console or cockpit. It integrates the desk as an interaction surface to merge physical and virtual elements, turning complex GUI operations into tangible manipulation. Passive haptic feedback provides stability, while intuitive 3D interactions foster creativity through exploratory and embodied workflows.

3D Represented Controls Mapped from Categorized UI Input Types Image

Virtual 3D Control Design Mapped from 2D Panels

Interactive Application Implementation

Evaluation

We used raycasting, a widely adopted method for interacting with virtual screens, as the baseline. The study employed a within-subjects design to account for order effects.

Desk Console User Study Interfaces Image

Results

Broader Control Exploration

Users explored more controls within the same time, and adjusted the vertical slider and color picker significantly faster with Desk Console. They shifted from focusing on a single task to experimenting with multiple options, showing broader exploratory behavior.

“I felt more comfortable with this(Desk Console), so I tried different controls earlier and experimented with more various options. “(P9)

Motor-Cognitive Offloading

Desk Console significantly reduced mental and physical demand, effort, and frustration compared to raycasting. Familiar desk-based actions and immediate tactile feedback helped offload cognitive load into embodied routines, shortening decisions and enabling smoother engagement.

Embodied Interaction Supporting Creative Flow

Desk Console significantly enhanced exploration, expressiveness, and immersion compared to raycasting. Tangible manipulation with passive haptic feedback enabled users to maintain spatial awareness, supporting motor-cognitive strategies for more engaged creative exploration.

Qualitative Observations

Bimanual Manipulation

Tactile feedback reinforces bimanual coordination, sustaining engagement.

“Buttons or something similar in a cognitive sense make it much easier… I feel like I could use both hands next time. Using both hands really lightens the load. “(P11)

Focused Visual Engagement

Users’ eye shifts were reduced between the control panel and the creation, resulting in shorter and more stable gaze patterns.

“I get the sense that moving up and down is way less distracting for the eyes. With Desk Console, it feels secure in my hand, so I approached it with a similar mindset, much like how I’d work in Photoshop with a mouse, where I don’t really look at the controls and just focus on the task at hand.” (P6)

Color Exploration Patterns

Interviews revealed distinct experiences with the color picker across the two interfaces. To examine this further, log data that captured traces on the color picker was visualized. The results show more frequent adjustments in raycasting, while Desk Console supported finer, adjacent color exploration.

“I could easily find other shades of red and make decisions like, ‘I should use this red color.’” (P11)

Demo @CHI '25

BibTeX

@inproceedings{shin2025desk,
author    = {Yeeun Shin and Seung Hyeon Han and Woohun Lee},
title     = {Desk Console: Augmenting 3D Virtual Controls on Physical Desks for Immersive Authoring},
booktitle = {Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA '25)},
year      = {2025},
doi       = {10.1145/3706599.3721165},
url       = {https://doi.org/10.1145/3706599.3721165}
}