• KSII Transactions on Internet and Information Systems
    Monthly Online Journal (eISSN: 1976-7277)

PC-SAN: Pretraining-Based Contextual Self-Attention Model for Topic Essay Generation


Abstract

Automatic topic essay generation (TEG) is a controllable text generation task that aims to generate informative, diverse, and topic-consistent essays based on multiple topics. To make the generated essays of high quality, a reasonable method should consider both diversity and topic-consistency. Another essential issue is the intrinsic link of the topics, which contributes to making the essays closely surround the semantics of provided topics. However, it remains challenging for TEG to fill the semantic gap between source topic words and target output, and a more powerful model is needed to capture the semantics of given topics. To this end, we propose a pretraining-based contextual self-attention (PC-SAN) model that is built upon the seq2seq framework. For the encoder of our model, we employ a dynamic weight sum of layers from BERT to fully utilize the semantics of topics, which is of great help to fill the gap and improve the quality of the generated essays. In the decoding phase, we also transform the target-side contextual history information into the query layers to alleviate the lack of context in typical self-attention networks (SANs). Experimental results on large-scale paragraph-level Chinese corpora verify that our model is capable of generating diverse, topic-consistent text and essentially makes improvements as compare to strong baselines. Furthermore, extensive analysis validates the effectiveness of contextual embeddings from BERT and contextual history information in SANs.


Statistics

Show / Hide Statistics

Statistics (Cumulative Counts from December 1st, 2015)
Multiple requests among the same browser session are counted as one view.
If you mouse over a chart, the values of data points will be shown.


Cite this article

[IEEE Style]
F. Lin, X. Ma, Y. Chen, J. Zhou and B. Liu, "PC-SAN: Pretraining-Based Contextual Self-Attention Model for Topic Essay Generation," KSII Transactions on Internet and Information Systems, vol. 14, no. 8, pp. 3168-3186, 2020. DOI: 10.3837/tiis.2020.08.001.

[ACM Style]
Fuqiang Lin, Xingkong Ma, Yaofeng Chen, Jiajun Zhou, and Bo Liu. 2020. PC-SAN: Pretraining-Based Contextual Self-Attention Model for Topic Essay Generation. KSII Transactions on Internet and Information Systems, 14, 8, (2020), 3168-3186. DOI: 10.3837/tiis.2020.08.001.