Facial Expression Recognition Using Fused Features: A Comparison of Deep and Machine Learning

Abbas Issa  Jabbooree; Hussein Alkaabi; Ali Nadhim Kamber

doi:10.47709/cnahpc.v7i3.6128

Authors

Abbas Issa Jabbooree 1Ministry of Communication, ITPC, High Institute of Telecommunication & Post, Baghdad, Iraq
Hussein Alkaabi Ministry of Education Iraq, General Direction of Vocational Education, Al-Najaf, 54001, Iraq
Ali Nadhim Kamber Ministry of Education, General Directorate of Education in Najaf, Najaf, Iraq

DOI:

https://doi.org/10.47709/cnahpc.v7i3.6128

Keywords:

Comparative stady, Deep Learning, Machine Learning, Facial Expression, data fusion, L2-SVM

Abstract

Facial expression recognition (FER) is a highly active field with applications in computer vision, human-computer interaction, security, and computer graphics animation. Recent advancements in deep learning and machine learning have increased interest in utilizing these techniques for accurate facial expression classification. This paper presents a comparative study that evaluates the performance of deep learning and machine learning as classifiers in FER systems, specifically after data fusion. Data fusion techniques combine and integrate multiple sources of information, aiming to enhance the overall classification accuracy by extracting two types of features using geometrical and appearance features trained using two types of convolutional neural networks. The feature outputs of these networks are fused to create a final feature vector for the classification process. The study evaluates the performance of deep learning on two benchmark datasets, the extended Cohn-Kanade (CK+) and Oulu-CASIA datasets, to assess the performance of deep learning. As a point of comparison, the traditional machine learning approach based on the support vector machine (SVM) is also evaluated on the same datasets. Performance metrics such as classification accuracy, precision, recall, and F1-score are utilized. The results obtained from the study highlight the strengths and limitations of both deep learning and machine learning techniques when employed as classifiers in FER systems. Notably, the experimental results demonstrate that the deep learning approach significantly outperforms the baseline methods, achieving an increase in recognition accuracy of 5.22% for the CK+ and 3.07% for the Oulu-CASIA dataset.

Downloads

Download data is not yet available.

References

Allamy, S., & Koerich, A. L. (2021). 1D CNN Architectures for Music Genre Classification. 2021 IEEE Symposium Series on Computational Intelligence, SSCI 2021 - Proceedings.

Barman, A., & Dutta, P. (2021). Facial expression recognition using distance and shape signature features. Pattern Recognition Letters, 145, 254–261.

Boughida, A., Kouahla, M. N., & Lafifi, Y. (2022). A novel approach for facial expression recognition based on Gabor filters and genetic algorithm. Evolving Systems, 13(2), 331–345.

Ding, H., Zhou, S. K., & Chellappa, R. (2017). FaceNet2ExpNet?: Regularizing a Deep Face Recognition Net for Expression Recognition. IEEE, 118–126.

Do, L. N., Yang, H. J., Nguyen, H. D., Kim, S. H., Lee, G. S., & Na, I. S. (2021). Deep neural network-based fusion model for emotion recognition using visual data. Journal of Supercomputing, 77(10), 10773–10790.

Gera, D., & Balasubramanian, S. (2021). Landmark guidance independent spatio-channel attention and complementary context information based facial expression recognition. Pattern Recognition Letters, 145, 58–66.

González-Lozoya, S. M., de la Calleja, J., Pellegrin, L., Escalante, H. J., Medina, M. A., & Benitez-Ruiz, A. (2020). Recognition of facial expressions based on CNN features. Multimedia Tools and Applications, 79(19–20), 13987–14007.

Han, Y., Wang, X., & Lu, Z. (2021). Research on facial expression recognition based on Multimodal data fusion and neural network.

Jabbooree, A. I., Khanli, L. M., Salehpour, P., & Pourbahrami, S. (2023). A novel facial expression recognition algorithm using geometry ? –skeleton in fusion based on deep CNN. Image and Vision Computing, 134, 104677.

Jeong, D., Kim, B. G., & Dong, S. Y. (2020). Deep joint spatiotemporal network (DJSTN) for efficient facial expression recognition. Sensors (Switzerland), 20(7).

Kola, D. G. R., & Samayamantula, S. K. (2021). A novel approach for facial expression recognition using local binary pattern with adaptive window. Multimedia Tools and Applications, 80(2), 2243–2262.

Lagias, A. E., Lagkas, T. D., & Zhang, J. (2018). New RSSI-Based Tracking for Following Mobile Targets Using the Law of Cosines. IEEE Wireless Communications Letters, 7(3), 392–395.

Liu, C., Hirota, K., Ma, J., Jia, Z., & Dai, Y. (2021). Facial Expression Recognition Using Hybrid Features of Pixel and Geometry. IEEE Access, 9, 18876–18889.

Liu, T., Wang, J., Yang, B., & Wang, X. (2021). Facial expression recognition method with multi-label distribution learning for non-verbal behavior understanding in the classroom. Infrared Physics and Technology, 112(December 2020), 103594.

Liu, X., Cheng, X., & Lee, K. (2021). GA-SVM-Based Facial Emotion Recognition Using Facial Geometric Features. IEEE Sensors Journal, 21(10), 11532–11542.

Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I., & Ave, F. (2010). The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 4(July), 94–101.

Mitiche, I., Nesbitt, A., Conner, S., Boreham, P., & Morison, G. (2020). 1D-CNN based real-time fault detection system for power asset diagnostics. IET Generation, Transmission and Distribution, 14(24), 5816–5822.

Nan, Y., Ju, J., Hua, Q., Zhang, H., & Wang, B. (2022). A-MobileNet: An approach of facial expression recognition. Alexandria Engineering Journal, 61(6), 4435–4444.

Ngoc, Q. T., Lee, S., & Song, B. C. (2020). Facial landmark-based emotion recognition via directed graph neural network. Electronics (Switzerland), 9(5).

Niu, B., Gao, Z., & Guo, B. (2021). Facial Expression Recognition with LBP and ORB Features. Computational Intelligence and Neuroscience- Hindawi, 2021, 1–10.

Ozcan, T., & Basturk, A. (2020). Static facial expression recognition using convolutional neural networks based on transfer learning and hyperparameter optimization. Multimedia Tools and Applications.

Park, S. J., Kim, B. G., & Chilamkurti, N. (2021). A robust facial expression recognition algorithm based on multi-rate feature fusion scheme. Sensors, 21(21), 1–26.

Porcu, S., Floris, A., & Atzori, L. (2020). Evaluation of data augmentation techniques for facial expression recognition systems. Electronics (Switzerland), 9(11), 1–12.

Pourbahrami, S., Khanli, L. M., & Azimpour, S. (2020). An automatic clustering of data points with alpha and beta angles on apollonius and subtended arc circle based on computational geometry. 2020 28th Iranian Conference on Electrical Engineering, ICEE 2020.

Sajjad, M., Zahir, S., Ullah, A., Akhtar, Z., & Muhammad, K. (2020). Human Behavior Understanding in Big Multimedia Data Using CNN based Facial Expression Recognition. Mobile Networks and Applications, 25(4), 1611–1621.

Sikka, K., Sharma, G., & Bartlett, M. (2016). LOMo: Latent ordinal model for facial analysis in videos. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-Decem, 5580–5589.

Sukhavasi, S. B., Sukhavasi, S. B., Elleithy, K., El-sayed, A., & Elleithy, A. (2022). A Hybrid Model for Driver Emotion Detection Using Feature Fusion Approach. MDPI.

Suraiya , Refat Khan , Munmun , Mayeen, M. R. I. (2020). Development of a Robust Multi-Scale Featured Local Binary Pattern for Improved Facial Expression Recognition. Sensoe MDPI, 20(5391).

Taini, M., Zhao, G., Li, S. Z., & Pietikäinen, M. (2008). Facial expression recognition from near-infrared video sequences. Proceedings - International Conference on Pattern Recognition, 1–4.

Tang, Y., Zhang, X., Hu, X., Wang, S., & Wang, H. (2021). Facial Expression Recognition Using Frequency Neural Network. 30, 444–457.

Wang, H., Wei, S., & Fang, B. (2020). Facial expression recognition using iterative fusion of MO-HOG and deep features. Journal of Supercomputing, 76(5), 3211–3221.

Wang, M., Tan, P., Zhang, X., Kang, Y., Jin, C., & Cao, J. (2020). Facial expression recognition based on CNN. Journal of Physics: Conference Series, 1601(5).

Yan, K., & Zhou, X. (2022). Chiller faults detection and diagnosis with sensor network and adaptive 1D CNN. Digital Communications and Networks, June 2021.

Yu, Z., Liu, Q., & Liu, G. (2018). Deeper cascaded peak-piloted network for weak expression recognition. Visual Computer, 34(12), 1691–1699.

Zhou, J., Li, J., Yan, Y., Wu, L., & Xu, H. (2023). Mixing Global and Local Features for Long-Tailed Expression Recognition. Information (Switzerland), 14(2), 1–19.

Zhu, X., Ye, S., Zhao, L., & Dai, Z. (2021). Hybrid attention cascade network for facial expression recognition. Sensors, 21(6), 1–16.

Zou, W., Zhang, D., & Lee, D. J. (2022). A new multi-feature fusion based convolutional neural network for facial expression recognition. Applied Intelligence, 52(3), 2918–2929.