Deep Learning Based Data Fusion Methods for Multimodal Emotion Recognition 


Vol. 47,  No. 1, pp. 79-87, Jan.  2022
10.7840/kics.2022.47.1.79


PDF
  Abstract

Multimodal emotion recognition is a robust and reliable method as it utilizes multimodal data for more comprehensive representation of emotions. Data fusion is a key step in multimodal emotion recognition, because the accuracy of the recognition model mostly depends on how the different modalities are combined. The goal of this paper is to compare the performances of deep learning (DL) based models for the task of data fusion and multimodal emotion recognition. The contributions of this paper are two folds: 1) We introduce three DL models for multimodal fusion and classification: early fusion, hybrid fusion, and multi-task learning. 2) We systematically compare the performance of these models on three multimodal datasets. Our experimental results demonstrate that multi-task learning achieves the best results across all modalities; 75.41%, 68.33%, and 78.75% for classification of three emotional states using the combinations of audio-visual, EEG-audio, and EEG-visual data, respectively.

  Statistics
Cumulative Counts from November, 2022
Multiple requests among the same browser session are counted as one view. If you mouse over a chart, the values of data points will be shown.


  Cite this article

[IEEE Style]

J. N. Njoku, A. C. Caliwag, W. Lim, S. Kim, H. Hwang, J. Jeong, "Deep Learning Based Data Fusion Methods for Multimodal Emotion Recognition," The Journal of Korean Institute of Communications and Information Sciences, vol. 47, no. 1, pp. 79-87, 2022. DOI: 10.7840/kics.2022.47.1.79.

[ACM Style]

Judith Nkechinyere Njoku, Angela C. Caliwag, Wansu Lim, Sangho Kim, Han-Jeong Hwang, and Jin-Woo Jeong. 2022. Deep Learning Based Data Fusion Methods for Multimodal Emotion Recognition. The Journal of Korean Institute of Communications and Information Sciences, 47, 1, (2022), 79-87. DOI: 10.7840/kics.2022.47.1.79.

[KICS Style]

Judith Nkechinyere Njoku, Angela C. Caliwag, Wansu Lim, Sangho Kim, Han-Jeong Hwang, Jin-Woo Jeong, "Deep Learning Based Data Fusion Methods for Multimodal Emotion Recognition," The Journal of Korean Institute of Communications and Information Sciences, vol. 47, no. 1, pp. 79-87, 1. 2022. (https://doi.org/10.7840/kics.2022.47.1.79)

Search
(IN TITLE, AUTHOR, ABSTRACT,KEYWORDS)

Advanced Search ▼

POPULAR KEYWORDS
(TOP 10 KEYWORDS)

Recent Publications
(LAST 3 YEARS)






pISSN : 1226 - 4717
eISSN : 2287 - 3880