Multisensory Feedback Systems in XR Controlled by AI Emotion Recognition

Abstract

Extended Reality (XR), encapsulating Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), has transformed how users interact with digital content, offering immersive environments that blend physical and virtual elements. However, conventional XR systems typically provide static, one-size-fits-all sensory feedback—visual, auditory, or haptic—without regard to the user’s current affective state. This limitation can reduce immersion, hinder learning outcomes, and exacerbate negative emotional experiences in high-stakes applications such as exposure therapy or safety training. In this manuscript, we present a comprehensive design, implementation, and evaluation of an AI-driven multisensory feedback system for XR that dynamically adapts haptic, auditory, and visual feedback in real time based on users’ recognized emotional states. Our system employs a multi-modal emotion recognition engine combining convolutional neural networks (CNNs) for facial expression analysis with recurrent neural networks (RNNs) for physiological signal interpretation (heart rate variability and galvanic skin response). By fusing these inputs, the engine categorizes user emotions into five core states—joy, anxiety, frustration, sadness, and neutral—with an overall recognition accuracy exceeding 90% on benchmark data. A dedicated feedback controller then applies rule-based mappings from detected emotional states to feedback parameters: for instance, increasing haptic pulse frequency and shifting to cooler lighting when anxiety is detected, or introducing soft vibration and warm color palettes for joy.

We implemented the prototype within the Unity 3D engine, targeting the Oculus Quest 2 platform. Haptic feedback was delivered via a wearable haptic glove (HaptX SDK), spatialized audio through the Oculus Audio SDK, and dynamic lighting via custom Unity shaders. In a within-subjects user study (N = 40), participants performed a sequence of memory recall tasks and maze navigation challenges under two conditions: Adaptive (emotion-aware feedback enabled) and Control (static, non-adaptive feedback). Objective performance metrics—recall accuracy and token collection rates—improved by 12% and 6.6%, respectively, in the Adaptive condition (p < .01), while subjective presence measured via the Igroup Presence Questionnaire (IPQ) increased by 18% (p < .001). Qualitative interviews further revealed that participants felt more “in tune” with the environment and better able to regulate stress when feedback adapted to their emotional fluctuations. Our contributions are threefold: (1) a robust, multi-modal emotion recognition pipeline optimized for real-time XR applications; (2) a flexible, rule-based feedback controller unifying haptic, auditory, and visual modalities; and (3) empirical evidence demonstrating significant gains in performance and presence. We discuss implications for XR-based training, therapeutic interventions, and entertainment, highlighting how emotion-adaptive feedback can support personalized learning trajectories, mitigate cognitive overload, and foster deeper emotional engagement.