BOOKAR

Abstract,

As a result of stimulating the basic human senses, emotional states occur in humans. Of these senses, the visual sense is the most basic human sense. This sense perceives visual stimuli and elicits emotional states. Augmented Reality (AR) applications also work with these visual stimuli. Brain wave is one of the most used methods in emotion analysis. These waves are difficult to analyze because of the complexity of human emotions. The ability to distinguish emotions well depends on the stimuli that stimulate these emotions. These stimuli can be enriched with AR or Virtual Reality (VR), which is one of the components of Metaverse. AR has the potential to enrich traditional books with virtual objects. But little is known about how the unique possibilities of this immersive technology can affect reading habits and the emotional states evoked in the reader. This study investigates how AR systems, which are among immersive environments, are effective in distinguishing the emotional states of students in book reading activities, with the support of Electroencephalography (EEG). The BOOKAR dataset obtained within the scope of this study is among the first AR-supported datasets in emotion recognition using physiological signals with immersive methods. In order to reveal the emotional states of the readers, texts that stimulate emotional states such as disgusting, happy, neutral, and 2-dimensional pictures are presented within these texts. In the AR-based reading section, the 3-dimensional models of these 2-dimensional pictures and the rig-processed conditions of these models are presented as a stimulus to the reader. Twenty-two university students with a balanced distribution (12 males, 10 females) read 10 different texts from a physical book and a screen with virtual objects placed in an immersive environment. We use a combination of different feature extraction (for example, statistics, wavelet transform, and nonlinear dynamics), multi-instance learning, and classifier design methods (e.g., naive Bayesian (NB) and random forest (RF)), according to the standard pipeline for emotion recognition. The results show that AR-based reading has a significant discriminatory effect, especially on the valence-arousal emotional states of readers, and achieves higher classification performance than real reading. The results also show that time-frequency domain features are better at classifying emotional states from EEG signals with accuracy scores close to 100\%. It has been observed that the designed AR application also meets the usability characteristics. The proposed emotion recognition method in AR applications has significant potential for integration into various Metaverse-based applications.