A Bimodal Emotion Recognition Algorithm for Audio and Video Based on Emotion Modeling

Yang Liu; Shudan Feng

doi:10.34028/iajit

Article Details

A Bimodal Emotion Recognition Algorithm for Audio and Video Based on Emotion Modeling

Authors:

Yang Liu | Shudan Feng

Publisher:

Zarqa University, Jordan

Journal:

The International Arab Journal of Information Technology (IAJIT) Vol.23 Issue 2, pp.208-219

Document Type:

Article

Publication Date:

1/2026

Document Language:

English

Document ID (Org No.):

2001389855

pISSN:

2309-4524

DOI:

10.34028/iajit

Keywords:

Abstract:

 In audio-video bimodal emotion recognition, audio features and video features come from different modalities and have different representations and semantic information. Traditional methods rely only on the information of a single modality, which makes the fused features unable to comprehensively represent the emotional state, resulting in poor recognition results and small correlation coefficients. For this reason, a bimodal emotion recognition algorithm based on emotion modeling is proposed for audio and video. Firstly, the emotional audio is sub-framed by Fourier transform to obtain the Meier Frequency Cepstrum Coefficient (MFCC) features of emotional audio, extract the frame-level speech time-domain signal input features, and mine the audio SoundNET coding features of emotion; Secondly, the above three features are spliced together to complete the mining of total emotional audio features of emotion; then, the Recurrent Neural Network (RNN) and the long and Short-Term Memory Network (LSTM) are used to capture the emotional video features in depth; Finally, cross-modal learning and attention mechanism are used to integrate the extracted emotion features, and the emotion type is analyzed by the decision-level fusion network to complete the audio and video bimodal emotion recognition, which effectively avoids the problem of poor single-modal recognition results and improves the recognition accuracy and reliability. The results show that the proposed algorithm is effective in recognizing bimodal emotions in audio and video, and the correlation coefficient of the recognition results is large.

Article Details

A Bimodal Emotion Recognition Algorithm for Audio and Video Based on Emotion Modeling

Authors:

Yang Liu | Shudan Feng

Publisher:

Zarqa University, Jordan

Journal:

The International Arab Journal of Information Technology (IAJIT) Vol.23 Issue 2, pp.208-219

Document Type:

Article

Publication Date:

1/2026

Document Language:

English

Document ID (Org No.):

2001389855

pISSN:

2309-4524

DOI:

10.34028/iajit

Audio

video

fourier transform

soundnet neural network

recurrent neural network

long and short-term memory network

Issues

Article Details

A Bimodal Emotion Recognition Algorithm for Audio and Video Based on Emotion Modeling

Authors: Yang Liu | Shudan Feng

Publisher: Zarqa University, Jordan

Journal: The International Arab Journal of Information Technology (IAJIT) Vol.23 Issue 2, pp.208-219

Document Type: Article

Publication Date: 1/2026

Document Language: English

Document ID (Org No.): 2001389855

pISSN: 2309-4524

DOI: 10.34028/iajit

Audio

video

fourier transform

soundnet neural network

recurrent neural network

long and short-term memory network

Issues

Authors:

Yang Liu | Shudan Feng

Publisher:

Zarqa University, Jordan

Journal:

The International Arab Journal of Information Technology (IAJIT) Vol.23 Issue 2, pp.208-219

Document Type:

Article

Publication Date:

1/2026

Document Language:

English

Document ID (Org No.):

2001389855

pISSN:

2309-4524

DOI:

10.34028/iajit