• KSII Transactions on Internet and Information Systems
    Monthly Online Journal (eISSN: 1976-7277)

Robustness of Differentiable Neural Computer Using Limited Retention Vector-based Memory Deallocation in Language Model


Abstract

Recurrent neural network (RNN) architectures have been used for language modeling (LM) tasks that require learning long-range word or character sequences. However, the RNN architecture is still suffered from unstable gradients on long-range sequences. To address the issue of long-range sequences, an attention mechanism has been used, showing state-of-the-art (SOTA) performance in all LM tasks. A differentiable neural computer (DNC) is a deep learning architecture using an attention mechanism. The DNC architecture is a neural network augmented with a content-addressable external memory. However, in the write operation, some information unrelated to the input word remains in memory. Moreover, DNCs have been found to perform poorly with low numbers of weight parameters. Therefore, we propose a robust memory deallocation method using a limited retention vector. The limited retention vector determines whether the network increases or decreases its usage of information in external memory according to a threshold. We experimentally evaluate the robustness of a DNC implementing the proposed approach according to the size of the controller and external memory on the enwik8 LM task. When we decreased the number of weight parameters by 32.47%, the proposed DNC showed a low bits-per-character (BPC) degradation of 4.30%, demonstrating the effectiveness of our approach in language modeling tasks.


Statistics

Show / Hide Statistics

Statistics (Cumulative Counts from December 1st, 2015)
Multiple requests among the same browser session are counted as one view.
If you mouse over a chart, the values of data points will be shown.


Cite this article

[IEEE Style]
D. Lee, H. Park, S. Seo, H. Son, G. Kim and J. Kim, "Robustness of Differentiable Neural Computer Using Limited Retention Vector-based Memory Deallocation in Language Model," KSII Transactions on Internet and Information Systems, vol. 15, no. 3, pp. 837-852, 2021. DOI: 10.3837/tiis.2021.03.002.

[ACM Style]
Donghyun Lee, Hosung Park, Soonshin Seo, Hyunsoo Son, Gyujin Kim, and Ji-Hwan Kim. 2021. Robustness of Differentiable Neural Computer Using Limited Retention Vector-based Memory Deallocation in Language Model. KSII Transactions on Internet and Information Systems, 15, 3, (2021), 837-852. DOI: 10.3837/tiis.2021.03.002.