Immersive speculative enactments dataset: investigating human-robot teaming in virtual reality

Project Description: Virtual Reality (VR) technology offers new opportunities for immersive communication and collaboration in human-robot teams, particularly across long distances. In human-robot teams, autonomous agents can have both physical and virtual bodies, moving between real and virtual worlds to meet shared goals. For example, an agent inside a physical robot may move to a virtual body in a VR environment to work with human teammates. In time-critical situations, effective communication is important. Body language (e.g. gestures, gaze and posture) is a key factor in communication, in addition to speech. Having a visible body is necessary to display body language. This research investigated the design considerations for visually representing robots in VR, to support human-robot teaming in time-critical scenarios. For this research, we conducted in-person experiments with pairs of human participants in a room on the ANU campus. Participants were invited to role-play in fictional immersive simulations using VR headsets to explore the design of future human-robot teaming technology. Participants acted in the role of a human teammate in a team made up of humans and robots for 3 scenarios in a simulated VR environment. The scenarios were set in a simulated bushfire emergency control room and featured different time-critical emergency situations (a fire emergency, a search and rescue, and an emergency evacuation). Participants completed surveys after each scenario, to assess the robot design. At the end of the session, participants were interviewed together on their immersive experiences to gain insights on the design of virtual robots in VR. This research was supported by a funded PhD scholarship from the Defence Science and Technology Group (DSTG), from the Australian Department of Defence. Research Questions and Project Objectives This research was an exploratory user study using a research-through-design methodology. Research question investigated: RQ: What are the main design considerations for visually representing autonomous robots in Virtual Reality to support distributed human-robot teaming (HRT) in time-critical scenarios? Sub-questions investigated: SQ1: In what contexts, tasks, and scenarios is the visual representation of autonomous robots in VR suitable for distributed human-robot teaming in time-critical scenarios? SQ2: What areas are of interest for future research in interacting with robots in VR for human-robot teaming in time-critical scenarios? Research aims: 1. To determine the kinds of tasks, scenarios, and contexts the visual representation of autonomous robots in VR is suitable for, when supporting distributed HMT in time-critical scenarios. 2. To inform the areas and hypotheses of interest for future work. 3. To inform the creation of design recommendations or guidelines for the visual representations of autonomous teammates VR for supporting HMT. Experiment Method This study examines the design considerations for the visual representation of robots in virtual reality for supporting distributed human-robot teaming in time-critical scenarios. Participants undertook this study in pairs. Participants were asked to team with virtual robots in 3 speculative scenarios in an immersive virtual reality (VR) simulation. The scenarios were set in a bushfire emergency control room and participants role-played team tasks and goals in time-critical scenarios (e.g. enacting search and rescue, fire response operations). Participants role-played as human teammates within the team, and the robot teammates were pre-programmed into the VR simulation. Participant consent was sought before participating in the experiment. This study had three main parts. 1. In the first part, participants were asked to complete a set of surveys relating to their demographics (age and gender they identify with) and individual characteristics (familiarity with VR, personality profiles, and trust in robots) in Qualtrics. 2. In the second part, participants were asked to interact with their teammates in a set of fictional immersive scenarios in virtual reality in Immersive Speculative Enactments (a human-computer interaction research-through-design method). a. Participants were asked to enact tasks in speculative bushfire emergency contexts and scenarios in VR, wearing a VR headset. b. In each scenario, participants were asked to role-play and enact some tasks as part of a team within the simulated environment. (For example, in the VR simulation, participants led a team of remotely deployed drones to monitor an area, detect fires, report incidents, and respond to fire events.) c. The VR enactments were recorded (within VR) for behavioural analysis, and eye-tracking data was also captured to track participant attention. Participants were also videoed in the real-world lab environment whilst using the VR headset to capture behavioural data. d. After each scenario, participants were asked to complete post-task surveys in VR, regarding their individual experience of the design of the virtual robot. 3. In the final part, both participants were interviewed together, using a semi-structured interview approach, to capture their thoughts and experiences on interacting with the robots in the VR scenarios.
Type
collection
Title
Immersive speculative enactments dataset: investigating human-robot teaming in virtual reality
Brief Title
VR for HRT dataset
Collection Type
Dataset
Access Privileges
ANU College of Systems and Society
DOI - Digital Object Identifier
10.25911/57k8-r241
Significance Statement
The first user study to explore multi-embodiment in virtual reality for enhancing human-robot teaming.
Brief Description
This data was collected from a HRI/HCI user study examining multi-embodied robot teammates in VR.
Full Description
Project Description: Virtual Reality (VR) technology offers new opportunities for immersive communication and collaboration in human-robot teams, particularly across long distances. In human-robot teams, autonomous agents can have both physical and virtual bodies, moving between real and virtual worlds to meet shared goals. For example, an agent inside a physical robot may move to a virtual body in a VR environment to work with human teammates. In time-critical situations, effective communication is important. Body language (e.g. gestures, gaze and posture) is a key factor in communication, in addition to speech. Having a visible body is necessary to display body language. This research investigated the design considerations for visually representing robots in VR, to support human-robot teaming in time-critical scenarios. For this research, we conducted in-person experiments with pairs of human participants in a room on the ANU campus. Participants were invited to role-play in fictional immersive simulations using VR headsets to explore the design of future human-robot teaming technology. Participants acted in the role of a human teammate in a team made up of humans and robots for 3 scenarios in a simulated VR environment. The scenarios were set in a simulated bushfire emergency control room and featured different time-critical emergency situations (a fire emergency, a search and rescue, and an emergency evacuation). Participants completed surveys after each scenario, to assess the robot design. At the end of the session, participants were interviewed together on their immersive experiences to gain insights on the design of virtual robots in VR. This research was supported by a funded PhD scholarship from the Defence Science and Technology Group (DSTG), from the Australian Department of Defence. Research Questions and Project Objectives This research was an exploratory user study using a research-through-design methodology. Research question investigated: RQ: What are the main design considerations for visually representing autonomous robots in Virtual Reality to support distributed human-robot teaming (HRT) in time-critical scenarios? Sub-questions investigated: SQ1: In what contexts, tasks, and scenarios is the visual representation of autonomous robots in VR suitable for distributed human-robot teaming in time-critical scenarios? SQ2: What areas are of interest for future research in interacting with robots in VR for human-robot teaming in time-critical scenarios? Research aims: 1. To determine the kinds of tasks, scenarios, and contexts the visual representation of autonomous robots in VR is suitable for, when supporting distributed HMT in time-critical scenarios. 2. To inform the areas and hypotheses of interest for future work. 3. To inform the creation of design recommendations or guidelines for the visual representations of autonomous teammates VR for supporting HMT. Experiment Method This study examines the design considerations for the visual representation of robots in virtual reality for supporting distributed human-robot teaming in time-critical scenarios. Participants undertook this study in pairs. Participants were asked to team with virtual robots in 3 speculative scenarios in an immersive virtual reality (VR) simulation. The scenarios were set in a bushfire emergency control room and participants role-played team tasks and goals in time-critical scenarios (e.g. enacting search and rescue, fire response operations). Participants role-played as human teammates within the team, and the robot teammates were pre-programmed into the VR simulation. Participant consent was sought before participating in the experiment. This study had three main parts. 1. In the first part, participants were asked to complete a set of surveys relating to their demographics (age and gender they identify with) and individual characteristics (familiarity with VR, personality profiles, and trust in robots) in Qualtrics. 2. In the second part, participants were asked to interact with their teammates in a set of fictional immersive scenarios in virtual reality in Immersive Speculative Enactments (a human-computer interaction research-through-design method). a. Participants were asked to enact tasks in speculative bushfire emergency contexts and scenarios in VR, wearing a VR headset. b. In each scenario, participants were asked to role-play and enact some tasks as part of a team within the simulated environment. (For example, in the VR simulation, participants led a team of remotely deployed drones to monitor an area, detect fires, report incidents, and respond to fire events.) c. The VR enactments were recorded (within VR) for behavioural analysis, and eye-tracking data was also captured to track participant attention. Participants were also videoed in the real-world lab environment whilst using the VR headset to capture behavioural data. d. After each scenario, participants were asked to complete post-task surveys in VR, regarding their individual experience of the design of the virtual robot. 3. In the final part, both participants were interviewed together, using a semi-structured interview approach, to capture their thoughts and experiences on interacting with the robots in the VR scenarios.
Contact Email
bransky.k@gmail.com
Contact Address
4 Wighton Tce, Casey ACT 2913
Contact Phone Number
0407627028
Principal Investigator
Karla Bransky Kelly
Supervisors
Penny Kyburz
Collaborators
Sabrina Caldwell; Tom Gedeon; Kingsley Fletcher; Matthew J. Knight
Fields of Research
460806 - Human-computer interaction; 460810 - Social robotics
Socio-Economic Objective
220407 - Human-computer interaction
Keywords
Multi-Embodiment; Human-Robot Teaming; Virtual Reality
Type of Research Activity
Pure basic research
Date Coverage
02/08/2023
15/15/2025
Date of data creation
2024
Year of data publication
2025
Creator(s) for Citation
Karla
Kelly
Publisher for Citation
The Australian National University Data Commons
Publications
doi
10.1109/RO-MAN63969.2025.11217829
Exploring the Effects of (Re)Embodiment on Perceptions of Robot Teammates in Virtual Reality Environments
Bransky, K., & Sweetser, P. (2025). Exploring the Effects of (Re)Embodiment on Perceptions of Robot Teammates in Virtual Reality Environments. 34th IEEE International Conference on Robot and Human Interactive Communication (ROMAN), 500--507. https://doi.org/10.1109/RO-MAN63969.2025.11217829
doi
10.1145/3806398
Identity, Form, and Function: Exploring Re-Embodiment for Human-Robot Teaming in Virtual Reality Environments
Bransky, K., Sweetser, Penny, Caldwell, S., & Gedeon, T. (2026). Identity, Form, and Function: Exploring Re-Embodiment for Human-Robot Teaming in Virtual Reality Environments. ACM Transactions on Human-Robot Interaction. https://doi.org/10.1145/3806398
doi
10.1145/3747327.3763037
You Like This Robot? I Don't! How Individual Differences Influence Perceptions of Robot Teammates in Virtual Reality
Bransky, K., & Sweetser, P. (2025). You Like This Robot? I Don’t! How Individual Differences Influence Perceptions of Robot Teammates in Virtual Reality. Companion Proceedings of the 27th International Conference on Multimodal Interaction (ICMI ’25 Companion), 27–31. https://doi.org/10.1145/3747327.3763037
Access Rights
Contact Chief Investigator to negotiate access to the data
Access Rights Type
Restricted
Data Management Plan
Yes

Request data files

Number of files: 498

Size: 72 GB

Identifier: anudc:6486

Status: Published
Published to:
  • Australian National University
  • Australian National Data Service
Related items