Convolutional Neural Networks for Measuring Service Satisfaction of Hajj Pilgrims through Facial Expression Analysis

Undang Syaripudin; Jumadi Jumadi; Diena Rauda Ramdania; Indah Sri Lestari; Indri Nurfiani; Alfin Yogi Setyawan; Maisevli Harika; Mimin Mintarsih

doi:10.15575/join.v11i1.1677

Authors

Undang Syaripudin Department of Informatics, UIN Sunan Gunung Djati Bandung, Indonesia
Jumadi Jumadi Department of Informatics, UIN Sunan Gunung Djati Bandung, Indonesia
Diena Rauda Ramdania Human Centred Computing, Exertion Games Lab, Monash University Australia, Australia
Indah Sri Lestari Department of Informatics, UIN Sunan Gunung Djati Bandung, Indonesia
Indri Nurfiani Department of Informatics, UIN Sunan Gunung Djati Bandung, Indonesia
Alfin Yogi Setyawan Department of Informatics, UIN Sunan Gunung Djati Bandung, Indonesia
Maisevli Harika Informatics Engineering, Politeknik Negeri Bandung, Indonesia
Mimin Mintarsih Sharia Bussiness Management, Universitas Ma’soem, Indonesia, Indonesia

DOI:

https://doi.org/10.15575/join.v11i1.1677

Keywords:

CNN, Customer Satisfaction, Facial Expression Recognition, Hajj Services, Machine Learning, Real-time Monitoring

Abstract

Facial expressions serve as important non-verbal indicators of human emotions and can be leveraged to assess satisfaction levels in service environments. In the context of Hajj and Umrah, where verbal feedback may be limited due to language barriers or cultural factors, facial expression recognition offers a non-intrusive method to evaluate service quality. This study proposes a Convolutional Neural Network (CNN)-based model to detect emotional states such as happiness and dissatisfaction through facial expressions of pilgrims. A quantitative approach was adopted, employing preprocessing techniques including normalization, augmentation, and image resizing. The CNN architecture comprised multiple convolutional, pooling, and fully connected layers. The model was evaluated using accuracy, precision, recall, and F1-score metrics. Experiments with varying batch sizes (32, 64, 128, 256) across 50 epochs revealed that the optimal performance was achieved with a batch size of 64, resulting in an accuracy of 63%, precision of 66%, recall of 60%, and F1-score of 62%. During deployment, the model correctly classified 12 out of 16 real-world images, achieving a real-time accuracy of 78%. Therefore, the deployment results should be considered preliminary. Future studies will involve larger deployment samples, n-fold stratified cross-validation to obtain statistically reliable model performance, and subgroup analyses (e.g., lighting, facial pose, age, and gender) to better understand model behavior under diverse real-world conditions. All deployment images were collected with participant consent and processed without storing biometric data. These findings suggest that CNN-based emotion recognition can support real-time service evaluation and enhance the quality of pilgrim services during the Hajj and Umrah.

References

[1] Direktorat Jenderal Penyelenggaraan Haji dan Umrah, “Laporan Kinerja Tahun 2023,” 2023, Kementerian Agama Republik Indonesia

[2] Afrizal, “Ada 8 Bentuk Layanan Yang Diberikan Pemerintah Kepada Jamaah Haji.” [Online]. 2024. Available: https://karimun.kemenag.go.id/main/berita/detail/drs-h-afrizal-mmpd–ada-8-bentuk-layanan-yang-diberikan-pemerintah-kepada-jamaah-haji

[3] TribunNews, “BPS: Sangat Memuaskan, Indeks Kepuasan Jemaah Haji 2024 Capai 88,20” [Online]. 2024. Available: https://makassar.tribunnews.com/2024/09/20/bps-sangat-memuaskan-indeks-kepuasan-jemaah-haji-2024-capai-8820

[4] Zahdi, “Manajemen Haji dan Umrah (Analisis Pencapaian Kepuasan Layanan Haji dan Umrah Di Kota Bandar Lampung,” Disertasi, [Online]. 2021. Available: http://repository.radenintan.ac.id/id/eprint/15283

[5] Esposito, A., Esposito, A. M., & Vogel, C. (2015). Needs and challenges in human computer interaction for processing social emotional information. Pattern Recognition Letters, 66, 41-51.

[6] Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., & Pollak, S. D. (2019). Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological science in the public interest, 20(1), 1-68.

[7] Carroll, J. M., & Russell, J. A. (1996). Do facial expressions signal specific emotions? Judging emotion from the face in context. Journal of personality and social psychology, 70(2), 205.

[8] Ekman, P., Friesen, W. V., & Ellsworth, P. (2013). Emotion in the human face: Guidelines for research and an integration of findings (Vol. 11). Elsevier.

[9] Etcoff, N. L. (1986). The neuropsychology of emotional expression. In Advances in clinical neuropsychology (pp. 127-179). Boston, MA: Springer US.

[10] Stets, J. E. (2003). Emotions and sentiments. In Handbook of social psychology (pp. 309-335). Boston, MA: Springer Us.

[11] Izard, C. E. (1990). Facial expressions and the regulation of emotions. Journal of personality and social psychology, 58(3), 487.

[12] Morris, M. W., & Keltner, D. (2000). How emotions work: The social functions of emotional expression in negotiations. Research in organizational behavior, 22, 1-50.

[13] Shariff, A. F., & Tracy, J. L. (2011). What are emotion expressions for? Current Directions in Psychological Science, 20(6), 395-399.

[14] González-Rodríguez, M. R., Díaz-Fernández, M. C., & Gómez, C. P. (2020). Facial-expression recognition: An emergent approach to the measurement of tourist satisfaction through emotions. Telematics and Informatics, 51, 101404.

[15] Cahyaningtyas, C., Gudiato, C., & Sari, M. (2025). Deteksi Ekspresi Wajah Manusia Menggunakan Metode Convolutional Neural Network. Jurnal Fasilkom, 15(1), 138-145.

[16] Wulansari, T. (2024). Deteksi Ekspresi Wajah Untuk Mengukur Tingkat Kepuasan Pelanggan Dengan Menerapkan Metode Convolutional Neural Network (Doctoral dissertation, Politeknik Negeri Jember).

[17] Tran, T. K., Vo, Q. N., Hong, X., Li, X., & Zhao, G. (2021). Micro-expression spotting: A new benchmark. Neurocomputing, 443, 356-368.

[18] Muna, N. (2017). Klasifikasi Micro-Expressions Wajah Berbasis Subpixel Subtle Motion Estimation Menggunakan Multilayer Perceptron (Doctoral dissertation, Institut Teknologi Sepuluh Nopember).

[19] N., & Aithal, P. S. (2022). Real-time customer satisfaction analysis using facial expressions and head pose estimation. International Journal of Applied Engineering and Management Letters (IJAEML), 6(1), 301-312.

[20] Bouzakraoui, M. S., Sadiq, A., & Alaoui, A. Y. (2020). Customer satisfaction recognition based on facial expression and machine learning techniques. Advances in Science, Technology and Engineering Systems Journal, 5(4), 594-594.

[21] Saxena, A., Khanna, A., & Gupta, D. (2020). Emotion recognition and detection methods: A comprehensive survey. Journal of Artificial Intelligence and Systems, 2(1), 53-79.

[22] Bouzakraoui, M. S., Sadiq, A., & Alaoui, A. Y. (2020). Customer satisfaction recognition based on facial expression and machine learning techniques. Advances in Science, Technology and Engineering Systems Journal, 5(4), 594-594.

[23] Harianto, D., Filbert, S., Cahyakusuma, A. B., & Zakiyyah, A. Y. (2024, July). Analyzing customer satisfaction through face emotion recognition: A comparative study of convolutional neural networks (cnn) and long short term memory (lstm). In 2024 10th International Conference on Smart Computing and Communication (ICSCC) (pp. 50-54). IEEE.

[24] Mehendale, N. (2020). Facial emotion recognition using convolutional neural networks (FERC). SN Applied Sciences, 2(3), 446.

[25] Deng, D., Chen, Z., Zhou, Y., & Shi, B. (2020, April). Mimamo net: Integrating micro-and macro-motion for video emotion recognition. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 03, pp. 2621-2628).

[26] Goodfellow, I., Bengio, Y., Courville, A., & Bengio, Y. (2016). Deep learning (Vol. 1, No. 2). Cambridge: MIT press.

[27] Yamashita, R., Nishio, M., Do, R. K. G., & Togashi, K. (2018). Convolutional neural networks: an overview and application in radiology. Insights into imaging, 9(4), 611-629.

[28] Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25.

[29] O'shea, K., & Nash, R. (2015). An introduction to convolutional neural networks. arXiv preprint arXiv:1511.08458.

[30] Sharma, N., & Jain, C. (2019, March). Characterization of facial expression using deep neural networks. In 2019 5th International Conference on Advanced Computing & Communication Systems (ICACCS) (pp. 492-495). IEEE.

[31] Bibbo’, L., Cotroneo, F., & Vellasco, M. (2023). Emotional health detection in HAR: new approach using ensemble SNN. Applied Sciences, 13(5), 3259.

[32] Irham, M. H., Mubarak, A., & Salmin, M. (2023). Detecting Face Expressions in Real-Time Using Convolutional Neural Network (CNN) Algorithm. Technium, 17.

[33] Seandrio, A. L., Pratomo, A. H., & Florestiyanto, M. Y. (2021). Implementation of convolutional neural network (CNN) in facial expression recognition. Telematika: Jurnal Informatika dan Teknologi Informasi, 18(2), 211-221.

[34] Raza, A. R. A., Ishaq, M. I. M., Mehmood, B. M. B., Ehsan, R. E. R., & Iram, A. I. A. (2021). EFFECTS OF DOMESTIC VIOLENCE ON REPRODUCTIVE HEALTH: AN EVIDENCE FROM PAKISTAN DEMOGRAPHIC HEALTH SURVEY. Journal Of Akhtar Saeed Medical & Dental College, 3(04).

[35] Boer, Y., Valencia, L., & Prasetyo, S. Y. (2023). Exploring the potential of pre-trained dcnn models for facial emotion detection: A comparative analysis. In E3S Web of Conferences (Vol. 426, p. 01049). EDP Sciences.

[36] Muthamilselvan, T., Brindha, K., Senthilkumar, S., Saransh, Chatterjee, J. M., & Hu, Y. C. (2023). Optimized face-emotion learning using convolutional neural network and binary whale optimization. Multimedia Tools and Applications, 82(13), 19945-19968.

[37] Jain, D. K., Shamsolmoali, P., & Sehdev, P. (2019). Extended deep neural network for facial emotion recognition. Pattern Recognition Letters, 120, 69-74.

[38] Geetha, A. V., Mala, T., Priyanka, D., & Uma, E. (2024). Multimodal emotion recognition with deep learning: advancements, challenges, and future directions. Information Fusion, 105, 102218.