Image Captioning with Synergy-Gated Attention and Recurrent Fusion LSTM

• You Yang,
• Lizhi Chen,
• Longyue Pan,
• Juntao Hu,

Vol. 16, No. 10, October 31, 2022
10.3837/tiis.2022.10.010, Download Paper (Free):

• Image captioning
• Synergy-Gated Attention
• Recurrent Fusion LSTM
• Deep Learning

Abstract

Long Short-Term Memory (LSTM) combined with attention mechanism is extensively used to generate semantic sentences of images in image captioning models. However, features of salient regions and spatial information are not utilized sufficiently in most related works. Meanwhile, the LSTM also suffers from the problem of underutilized information in a single time step. In the paper, two innovative approaches are proposed to solve these problems. First, the Synergy-Gated Attention (SGA) method is proposed, which can process the spatial features and the salient region features of given images simultaneously. SGA establishes a gated mechanism through the global features to guide the interaction of information between these two features. Then, the Recurrent Fusion LSTM (RF-LSTM) mechanism is proposed, which can predict the next hidden vectors in one time step and improve linguistic coherence by fusing future information. Experimental results on the benchmark dataset of MSCOCO show that compared with the state-of-the-art methods, the proposed method can improve the performance of image captioning model, and achieve competitive performance on multiple evaluation indicators.

Statistics

Cite this article