Distributed Collaboration in Infrastructure Assessment through Mixed and Virtual Reality - a


  • Zaid Abbas Al-Sabbag University of Waterloo
  • Chul Min Yeum University of Waterloo
  • Sriram Narasimhan University of California, Los Angeles




Visual inspection, Mixed reality, Virtual reality, Remote collaboration, Damage detection


In this study, we propose a real-time distributed collaborative system that inspectors equipped with Mixed Reality (MR) and Virtual Reality (VR) headsets can perform synchronous structural inspections on- and off-site. MR headsets are equipped with a holographic display that allows on-site inspectors to visualize the locations of structural defects overlaid on the structure, while VR headsets allow off-site inspectors to visualize and interact with a reconstructed 3D virtual environment. A proposed workflow begins with generating a colorized 3D point cloud map of the site using visual sensors and lidars prior to the inspection. The map is then stored in a centralized server to be accessible to multiple MR/VR users. MR inspectors performing the structural assessment on-site are spatially aligned in the reconstructed map using image-based localization. MR inspectors can then add annotations and capture images of structural defects that are stored on the map and shared with remote VR users. Simultaneously, VR inspectors can perform the inspection using the localized images stored on the map and shared by the MR inspector. VR inspectors can also add annotations that are both visible to other VR and MR inspectors, achieving collaborative inspection. An experimental study is conducted in a lab environment to demonstrate the proposed workflow using Microsoft HoloLens 2 (HL2) as the MR headset and Oculus Quest 2 as the VR headset. A panoramic camera (Ricoh Theta Z1) was attached to the HL2 headset to give the VR user a 360 degrees view of the site from images captured by the MR inspector for inspection and map reconstruction.