Towards Real-Time Recognition of Users’ Mental Workload Using Integrated Physiological Sensors Into a VR HMD

Abstract

This paper describes an ``all-in-one’’ solution for the real-time recognition of users’ mental workloads in virtual reality through the customization of a commercial HMD with physiological sensors. First, we describe the hardware and software solution employed to build the system. Second, we detail the machine learning methods used for the automatic recognition of the users’ mental workload, which are based on the well-known Random Forest algorithm. In order to gather data to train the system, we conducted an extensive user study with 75 participants using a VR flight simulator to induce different levels of mental workload. In contrast to previous works which label the data based on a standardized task (e.g., n-back task) or on a pre-defined task-difficulty, participants were asked about their perceived mental workload level along the experiment. With the data collected, we were able to train the system in order to classify four different levels of mental workload with an accuracy up to 65%. In addition, we discuss the role of the signal normalization procedures, the contribution of the different physiological signals on the recognition accuracy and compare the results obtained with the sensors embedded in the HMD with commercial grade systems. Preliminary results show our pipeline is able to recognize mental workload in real-time. Taken together, our results suggest that such all-in-one approach, with physiological sensors directly embedded in the HMD, is a promising path for VR applications in which the real-time or off-line estimation of Mental Workload assessment is beneficial.