ECL-tEEG: A HYBRID DEEP LEARNING MODEL FOR ESTIMATING LEARNERS’ EMOTIONS FROM EEG SIGNALS

Phan Thi Quy Thinh1, Phung Ngoc Nhan1, Mai Yen Khoa1, Tran Thanh Nha1, Vo Le Phuc Hau1, , Nguyen Viet Hung1
1 Ho Chi Minh City University of Education, Vietnam

Main Article Content

Abstract

Recognizing emotions and assessing student engagement are important for improving instructional effectiveness and personalizing teaching. Previous studies have primarily used video or image data with deep learning models to analyze learners’ behavior and emotions. However, these approaches are sensitive to camera angles, lighting conditions, and ambiguous facial expressions, which may reduce the accuracy of emotion assessment. To address these limitations, this study proposes ECL-tEEG, a model that combines a convolutional neural network (CNN), an LSTM–GRU architecture, and a Transformer to analyze electroencephalogram (EEG) signals for emotion recognition and student engagement assessment. The model uses the CNN to extract spatial features, the LSTM–GRU architecture to capture temporal dependencies, and the Transformer to model long-range relationships through parallel processing. Its performance was evaluated against two comparison models: CNN-LSTM-GRU and CNN-Transformer. ECL-tEEG achieved an accuracy of 71%, outperforming both comparison models in classifying positive, neutral, and negative emotions. The findings demonstrate the potential of EEG-based emotion recognition and provide a basis for intelligent instructional support systems that can personalize learning and help teachers adjust their instruction using neural signals alongside behavioral observations.

Article Details

References

Andayani, F., Theng, L. B., Tsun, M. T., & Chua, C. (2022). Hybrid LSTM-Transformer Model for Emotion Recognition From Speech Audio Files. IEEE Access, 10, 36018–36027. https://doi.org/10.1109/access.2022.3163856
Apicella, A., Arpaia, P., Frosolone, M., Improta, G., Moccaldi, N., & Pollastro, A. (2022). EEG-based measurement system for monitoring student engagement in learning 4.0. Scientific Reports, 12(1). https://doi.org/10.1038/s41598-022-09578-y
Florestiyanto, M. Y. (2024). Emotion Recognition for Improving Online Learning Environments: A Systematic Review of the Literature. Deleted Journal, 20(4s), 1860–1873. https://doi.org/10.52783/jes.2255
Hari Krishna, B., Sharon Rose Victor, J., Srinivasa Rao, G., Raja Kishore Babu, Ch., Srujan Raju, K., Ghouse Basha, T. S., & Bharath Simha Reddy, V. (2024). Emotion-net: Automatic emotion recognition system using optimal feature selection-based hidden markov CNN model. Ain Shams Engineering Journal, 103038. https://doi.org/10.1016/j.asej.2024.103038
Hsu, C. F., Chao, H.-H., Yang, A. C., Yeh, C.-W., Hsu, L., & Chi, S. (2020). Discrimination of Severity of Alzheimer's Disease with Multiscale Entropy Analysis of EEG Dynamics. Applied Sciences, 10(4), 1244–1244. https://doi.org/10.3390/app10041244
Jaworska, N., de la Salle, S., Ibrahim, M.-H., Blier, P., & Knott, V. (2019). Leveraging Machine Learning Approaches for Predicting Antidepressant Treatment Response Using Electroencephalography (EEG) and Clinical Data. Frontiers in Psychiatry, 9. https://doi.org/10.3389/fpsyt.2018.00768
Joshi, V. M., & Ghongade, R. B. (2022). IDEA: Intellect database for emotion analysis using EEG signal. Journal of King Saud University - Computer and Information Sciences, 34(7), 4433–4447. https://doi.org/10.1016/j.jksuci.2020.10.007
Lee, M.-H., Shomanov, A., Begim, B., Kabidenova, Z., Nyssanbay, A., Yazici, A., & Lee, S.-W. (2024). EAV: EEG-Audio-Video Dataset for Emotion Recognition in Conversational Contexts. Scientific Data, 11(1). https://doi.org/10.1038/s41597-024-03838-4
Mahmoud, A., Amin, K., Al Rahhal, M. M., Elkilani, W. S., Mekhalfi, M. L., & Ibrahim, M. (2023). A CNN Approach for Emotion Recognition via EEG. Symmetry, 15(10), 1822. https://doi.org/10.3390/sym15101822
Song, T., Zheng, W., Lu, C., Zong, Y., Zhang, X., & Cui, Z. (2019). MPED: A Multi-Modal Physiological Emotion Database for Discrete Emotion Recognition. IEEE Access, 7, 12177–12191. https://doi.org/10.1109/ACCESS.2019.2891579
Song, Y., Zheng, Q., Liu, B., & Gao, X. (2023). EEG Conformer: Convolutional Transformer for EEG Decoding and Visualization. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31, 710–719. https://doi.org/10.1109/tnsre.2022.3230250
Taware, S., & Thakare, A. (2023). Critical Analysis on Multimodal Emotion Recognition in Meeting the Requirements for Next Generation Human Computer Interactions. International Journal on Recent and Innovation Trends in Computing and Communication, 11(9s), 523–534. https://doi.org/10.17762/ijritcc.v11i9s.7464
Yu, C., & Wang, M. (2022). Survey of emotion recognition methods using EEG information. Cognitive Robotics, 2, 132–146. https://doi.org/10.1016/j.cogr.2022.06.001
Zhu, X., Song, Y., & Li, D. (2024). EEG Emotion Recognition Based on CNN+LSTM. 2024 IEEE 25th China Conference on System Simulation Technology and Its Application (CCSSTA), 144–148. https://doi.org/10.1109/ccssta62096.2024.10691696