Urban Sight
Application of Digital Twin for Collaborative Design in Mixed Reality Environment
2022 – 2023
Master Thesis
Research Advisor: Jeng, Tay-Sheng

Our study enhances Building Information Modeling (BIM) by focusing on real-time collaboration. We integrate Digital Twin and Mixed Reality (MR) technologies to create a 3D collaborative environment for more effective building and city management. Using Unity Engine and Autodesk API, we display BIM models on Microsoft HoloLens 2, enriched with real-time IoT sensor data. This collaborative approach ensures real-time updates and visual feedback, significantly improving the efficiency and intuitiveness of management tasks.

#Collaborative Design  #Mixed Reality  #Digital Twin  #BIM  #Real-Time

Background & Motivation

Driven by the shortcomings of traditional collaborative design and the promise of Digital Twin technology, our research explores its application in a Mixed Reality environment to enhance the efficiency of collaborative design processes. By merging the immersive features of Mixed Reality with the real-time, data-driven aspects of Digital Twin, we aim to offer a more integrated and collaborative design experience.

Research Objective

Related Work

The interconnection among the pertinent literature in this study is illustrated in below, segmented into three sections, including Collaborative Design, Digital Twins, and Mixed Reality. The diagram right is organized by this research, and a three-dimensional diagram has been drawn to analyze the methods of collaborative design.

 

They are defined as: synchronous-asynchronous, in-situ-remote, co-design with the same model, and distributed design with different models. This research mainly focuses on synchronous and co-located collaborative design.

Integrating Digital Twin with Mixed Reality tackles two main challenges in Collaborative Design:

1) communication and visualization

2) spatial understanding

 

We propose three scenarios:

 

Scenario A

focusing on Several designers communicating their initial design concepts in parallel

 

Scenario B

focusing on Facility Management and Environmental Control

 

Scenario C

exploring the potential of Smart City Management and Simulation.

 

We have considered the different situations of collaboration in both first-person and third-person perspectives.

The Scenario

This study validates digital twin applications through a system with three main components: Digital Twin (blue blocks), Mixed Reality (red blocks), and Collaboration (yellow blocks).

 

The blue blocks focus on real-time data collection and updating the digital model.

The red blocks use Unity to overlay virtual data onto the real world, enhancing urban planning.

The yellow blocks enable real-time, multi-user collaboration, improving communication and decision-making.

System Design

Digital Twin

This part of research introduced in SmartHygiene+ project. The Digital Twin Web Dashboard setup of the system was divided into two parts: 1.Digital Twin Modeling & 2. IoT Sensor Connection.

Mixed Reality

1. Digital Twin Import

 

The Autodesk Model Derivative API lets users convert various file types, like Revit files, into web-optimized SVF formats. We further convert SVF to glTF, making the 3D models compatible with web and augmented reality apps. It combines JSON for metadata and binary data for efficient storage and transfer. glTF files can be saved in a binary format called glTF binary (glb). The glb format was chosen because of its smaller file size and faster loading time.

2. Mixed Reality Deployment

 

The system is integrated into the Microsoft HoloLens 2 headset, which project build using Universal Windows Platform (UWP). This research developed mixed reality applications with the Microsoft Mixed Reality Toolkit (MRTK).

Collaboration

1. Collaborative Environment Setup

 

The Collaborative Environment Setup aims for a multi-user shared experience, focusing on network connectivity, data sync, and object location sharing. It has two main parts:


1. Synchronize Users

This allows users to sync actions and exchange data.

2. Share Objects Location

This helps accurately place model objects in the mixed reality space.

The system uses Photon Cloud and Azure Spatial Anchor to facilitate this collaboration. This system not only maintains a unified perception of the shared space but also provides a scalable and secure infrastructure for mixed reality applications.

2. Collaborative Interface Design

 

The following step is to establish a user-friendly and intuitive collaboration interface design. The interface is important to the overall user experience. We aim to seamlessly integrate multi-channel interaction input methods and provide easy access to application functionality. In Collaborative Interface Design, we delve into the innovative inclusion of voice commands, gesture recognition, and eye tracking as interaction modalities, which all enhance user interaction and engagement within the mixed reality environment.

1. Azure Anchor Panel
The Azure Anchor Panel leverages the capabilities of Azure Spatial Anchors, further enhancing the experience of interacting with the mixed reality environment. It provides users with the means to create, identify, and manage spatial anchors in the virtual landscape.

2. Perspective Panel
The Perspective Panel is strategically designed to control the viewpoint by gesture recognition and eye tracking. It offers an interactive and immersive way for users to navigate through the mixed reality environment and manage their perspective. This interactive experience enhances user involvement and comprehension of the shared space. Key functionalities of the Perspective Panel to move position with the slider bars.

3. Hand Draw Panel
The Hand Draw Panel accentuates the user’s immersive experience by leveraging the power of gesture recognition, enabling them to draw some ideas and modify them directly within the 3D environment. This interactive panel not only facilitates real-time collaborative design, but also fosters creativity, innovation, and a shared understanding within the team.

As the system is main designed to facilitate collaborative experiences in the mixed reality environment, user experience on collaboration is important. This includes users’ experiences in sharing and manipulating objects in the shared space. This study analyzes our perspectives on the collaboration capabilities when we are using the system, including users’ views on synchronize users and share object’s location, based on the collaborative environment.

User Experience

Demo Video

資產 65
播放影音