Contrastive Learning-Driven Multimodal Emotion Analysis with Cross-Modal Complementary Balance

• Zibo Xu,
• Bin Song,
• Zhiyong Zhang,

Vol. 20, No. 1, January 31, 2026
10.3837/tiis.2026.01.018, Download Paper (Free):

• Multimodal fusion
• sentiment analysis
• cross modal contrastive learning
• Transformer
• GAT

Abstract

Multimodal sentiment analysis, integrating text, images, and audio, has gained increasing attention in recent years. However, effectively fusing heterogeneous modalities remains challenging due to information redundancy, modality imbalance, and noisy interactions. To address these issues, this paper introduces a novel multimodal emotion prediction method, CMCBEP-CL, which leverages contrastive learning and cross-modal complementary balance to enhance sentiment analysis accuracy. The method comprises three core modules: a lead mode supplementation module, a cross-modal comparative analysis module, and a balanced prediction result module. The lead mode supplementation module identifies and utilizes the most representative modality to supplement feature representations of other modalities. The cross-modal comparative analysis module extracts specific and differential features through contrastive learning, improving discriminative semantic information extraction. Finally, the balanced prediction result module integrates prediction results from each stage to ensure robust sentiment analysis. Experimental results on the CMU-MOSEI and CMU-MOSI datasets demonstrate that our proposed approach outperforms state-of-the-art methods, achieving improvements of up to 0.31% in accuracy and 1.4% in correlation coefficient, thus confirming its efficacy and superiority. This work's source code can be found at https://github.com/zbxshangqiu/CMCBEP-CL.git.

Statistics

Cite this article