ECL-tEEG: A Hybrid Deep Learning Model for Emotion Estimation of Learners Based on EEG Signals

Hau Vo1, , Quy Thinh Phan, Ngoc Nhan Phung, Yen Khoa Mai, Thanh Nha Tran2, Viet Hung Nguyen3
1 Le Phuc
2 Trường Đại Học Sư Phạm Thành Phố Hồ Chí Minh
3 Đại Học Sư Phạm Thành Phố Hồ Chí Minh

Main Article Content

Abstract

Emotion recognition and student engagement assessment play an important role in improving teaching efficiency and optimizing personalized teaching methods. Previous studies mainly use videos or images combined with deep learning models to analyze behavior and emotions. However, this method is easily affected by camera angles, lighting conditions, and unclear facial expressions, leading to errors in emotion assessment. To overcome these limitations, this study proposes ECL-tEEG, a model that combines CNN, LSTM-GRU, and Transformer to analyze electroencephalogram (EEG) signals to recognize emotions and assess student engagement. The proposed model utilizes CNN to extract spatial features, LSTM-GRU to learn temporal sequential features, and Transformer to exploit long-term relationships using a parallel learning mechanism. To evaluate the performance, the study conducted experiments on three models: CNN-LSTM-GRU, CNN-Transformer and ECL-tEEG. The experimental results showed that ECL-tEEG achieved 71% accuracy, outperforming the other two models in classifying positive, neutral and negative emotions. This study not only affirms the potential application of EEG in recognizing learners' emotions, but also lays the foundation for an intelligent teaching support system, helping to personalize the learning experience and support teachers in adjusting teaching methods based on brain signals instead of just relying on behavioral observations.

Article Details

References

Andayani, F., Theng, L. B., Tsun, M. T., & Chua, C. (2022). Hybrid LSTM-Transformer Model for Emotion Recognition From Speech Audio Files. IEEE Access, 10, 36018–36027. https://doi.org/10.1109/access.2022.3163856
Apicella, A., Arpaia, P., Frosolone, M., Improta, G., Moccaldi, N., & Pollastro, A. (2022). EEG-based measurement system for monitoring student engagement in learning 4.0. Scientific Reports, 12(1). https://doi.org/10.1038/s41598-022-09578-y
B. Hari Krishna, Rose, S., Goda Srinivasa Rao, Ch., K. Srujan Raju, T.S. Ghouse Basha, & Bharath, V. (2024). Emotion-net: Automatic emotion recognition system using optimal feature selection-based hidden markov CNN model. Ain Shams Engineering Journal, 103038–103038. https://doi.org/10.1016/j.asej.2024.103038
Hussein Ali Hamzah, & Abdalla, K. K. (2024). EEG-based emotion recognition systems; Comprehensive study. Heliyon, 10(10), e31485–e31485. https://doi.org/10.1016/j.heliyon.2024.e31485
Joshi, V. M., & Ghongade, R. B. (2022). IDEA: Intellect database for emotion analysis using EEG signal. Journal of King Saud University - Computer and Information Sciences, 34(7), 4433–4447. https://doi.org/10.1016/j.jksuci.2020.10.007
Lee, M.-H., Shomanov, A., Begim, B., Kabidenova, Z., Nyssanbay, A., Yazici, A., & Lee, S.-W. (2024). EAV: EEG-Audio-Video Dataset for Emotion Recognition in Conversational Contexts. Scientific Data, 11(1). https://doi.org/10.1038/s41597-024-03838-4
Lun, X., Yu, Z., Chen, T., Wang, F., & Hou, Y. (2020). A Simplified CNN Classification Method for MI-EEG via the Electrode Pairs Signals. Frontiers in Human Neuroscience, 14. https://doi.org/10.3389/fnhum.2020.00338
Mahmoud, A., Amin, K., Al Rahhal, M. M., Elkilani, W. S., Mekhalfi, M. L., & Ibrahim, M. (2023). A CNN Approach for Emotion Recognition via EEG. Symmetry, 15(10), 1822. https://doi.org/10.3390/sym15101822
Maximilian Achim Pfeffer, Sai, S., & Kwok, J. (2024). Exploring the frontier: Transformer-based models in EEG signal analysis for brain-computer interfaces. Computers in Biology and Medicine, 178, 108705–108705. https://doi.org/10.1016/j.compbiomed.2024.108705
None Mangaras Yanu Florestiyanto. (2024). Emotion Recognition for Improving Online Learning Environments: A Systematic Review of the Literature. Deleted Journal, 20(4s), 1860–1873. https://doi.org/10.52783/jes.2255
Shwetkranti Taware, & Anuradha Thakare. (2023). Critical Analysis on Multimodal Emotion Recognition in Meeting the Requirements for Next Generation Human Computer Interactions. International Journal on Recent and Innovation Trends in Computing and Communication, 11(9s), 523–534. https://doi.org/10.17762/ijritcc.v11i9s.7464
Song, T., Zheng, W., Lu, C., Zong, Y., Zhang, X., & Cui, Z. (2019). MPED: A Multi-Modal Physiological Emotion Database for Discrete Emotion Recognition. IEEE Access, 7, 12177–12191. https://doi.org/10.1109/ACCESS.2019.2891579
Song, Y., Zheng, Q., Liu, B., & Gao, X. (2023). EEG Conformer: Convolutional Transformer for EEG Decoding and Visualization. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31, 710–719. https://doi.org/10.1109/tnsre.2022.3230250
Wan, Z., Li, M., Liu, S., Huang, J., Tan, H., & Duan, W. (2023). EEGformer: A transformer–based brain activity classification method using EEG signal. 17. https://doi.org/10.3389/fnins.2023.1148855
Yu, C., & Wang, M. (2022). Survey of emotion recognition methods using EEG information. Cognitive Robotics, 2, 132–146. https://doi.org/10.1016/j.cogr.2022.06.001
Zhu, X., Song, Y., & Li, D. (2024). EEG Emotion Recognition Based on CNN+LSTM. 2024 IEEE 25th China Conference on System Simulation Technology and Its Application (CCSSTA), 144–148. https://doi.org/10.1109/ccssta62096.2024.10691696