<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/recurrent-models-and-examples-with-mxnetr/"></div></div><div><a data-url="https://www.r-bloggers.com/recurrent-models-and-examples-with-mxnetr/" data-counturl="https://www.r-bloggers.com/recurrent-models-and-examples-with-mxnetr/" data-count="horizontal" data-text="Recurrent Models and Examples with MXNetR" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://dmlc.ml/rstats/2016/08/19/recurrent-models-and-examples-with-mxnet-r.html"> DMLC(distributed machine learning common)</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>As a new lightweight and flexible deep learning platform, <a target="_blank" href="https://github.com/dmlc/mxnet">MXNet</a> provides a portable backend, which can be called from R side. <a target="_blank" href="https://github.com/dmlc/mxnet/tree/master/R-package">MXNetR</a> is an R package that provide R users with fast GPU computation and state-of-art deep learning models. </p>
<p>In this post, We have provided several high-level APIs for recurrent models with <a target="_blank" href="https://github.com/dmlc/mxnet">MXNetR</a>. Recurrent neural network (RNN) is a class of artificial neural networks, which is very popular in the sequence labelling tasks, such as handwriting recognition, speech recognition. </p>
<p>We will introduce our implementation of the recurrent models including RNN, LSTM and GRU. In addition, the examples such as char-rnn will be showed to explain how to use the RNN models. By the way, several optimizers are added in MXNetR too.</p>
<p>This post demonstrates the implementation of the recurrent modules, including the structure of recurrent cells, unrolling of RNN models and the specific functions for training and testing.</p>
<ol>
<li>
<p>Three kinds of recurrent cells, including custom RNN, LSTM and GRU cells.</p>
</li>
<li>
<p>How to unroll RNN models to common feedward network.</p>
</li>
<li>
<p>How to train RNN models, including setting up and training RNN models by our specific solver using low-level simple-bind interface.</p>
</li>
<li>
<p>How to utilize the trained RNN models using RNN inference interfaces, including <code>inference</code> and <code>forward</code> functions.</p>
</li>
</ol>
<p>The link to the commits is <a target="_blank" href="https://github.com/dmlc/mxnet/commits?author=ziyeqinghan">https://github.com/dmlc/mxnet/commits?author=ziyeqinghan</a>.</p>
<h2>Recurrent Models</h2>
<p>Since the RNN model can be treated as a deep feedforward neural network, which unfolds in time, it suffers from the problem of vanishing and exploding gradients. Thus, there are several variants of RNN to learn the long term dependency, including Long Short-Term Memory (LSTM) [1] and Gated Recurrent Unit (GRU) [2]. </p>
<p>We will introduce three RNN models including the custom RNN, LSTM and GRU which has been implemented in MXNetR. To see the complete code, please refer to the relevant files <a target="_blank" href="https://github.com/dmlc/mxnet/blob/master/R-package/R/rnn.R">rnn.R</a>, <a target="_blank" href="https://github.com/dmlc/mxnet/blob/master/R-package/R/lstm.R">lstm.R</a>, <a target="_blank" href="https://github.com/dmlc/mxnet/blob/master/R-package/R/gru.R">gru.R</a> and <a target="_blank" href="https://github.com/dmlc/mxnet/blob/master/R-package/R/rnn_model.R">rnn_model.R</a> in the <a target="_blank" href="https://github.com/dmlc/mxnet/tree/master/R-package/R">R-package/R</a> directory respectively. </p>
<h3>RNN Cells</h3>
<p>The main difference between three RNN models is that they have corresponding cells with different structures to mitigate the problem of vanishing and exploding gradients.</p>
<h4>Custom RNN Cells</h4>
<p>The Common RNN can be considered as feedforward networks with self-connected hidden layers. As Figure 1 shows, the key of the RNN is that it allows the previous inputs has influence on the current output by using the recurrent connection.</p>
<p><img align="center" src="https://i2.wp.com/dmlc.ml/rstats/2016/08/img/mxnet/rnn/unfolded_recurrent_network.png?w=456" data-recalc-dims="1"></p>
<p>Given an input sequence <code>x</code> and the previous state <code>h</code>, a custom RNN cell produces the next states successively. Thus, there are two types of connections, the input to the hidden <code>i2h</code>, and the (previous) hidden to the hidden <code>h2h</code>. Then an optional batch normalization layer and nonlinear activation layer (e.g. tanh) are followed to generate the output states.</p>
<div>
<pre><code data-lang="r"><span>rnn</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>num.hidden</span><span>,</span><span> </span><span>indata</span><span>,</span><span> </span><span>prev.state</span><span>,</span><span> </span><span>param</span><span>,</span><span> </span><span>seqidx</span><span>,</span><span> </span><span>layeridx</span><span>,</span><span> </span><span>dropout</span><span>=</span><span>0</span><span>.</span><span>,</span><span> </span><span>batch.norm</span><span>=</span><span>FALSE</span><span>)</span><span> </span><span>{</span><span>
</span><span>if</span><span> </span><span>(</span><span>dropout</span><span> </span><span>></span><span> </span><span>0</span><span>.</span><span> </span><span>)</span><span>
</span><span>indata</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.Dropout</span><span>(</span><span>data</span><span>=</span><span>indata</span><span>,</span><span> </span><span>p</span><span>=</span><span>dropout</span><span>)</span><span>
</span><span>i</span><span>2</span><span>h</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.FullyConnected</span><span>(</span><span>data</span><span>=</span><span>indata</span><span>,</span><span>
</span><span>weight</span><span>=</span><span>param</span><span>$</span><span>i</span><span>2</span><span>h.weight</span><span>,</span><span>
</span><span>bias</span><span>=</span><span>param</span><span>$</span><span>i</span><span>2</span><span>h.bias</span><span>,</span><span>
</span><span>num.hidden</span><span>=</span><span>num.hidden</span><span>,</span><span>
</span><span>name</span><span>=</span><span>paste0</span><span>(</span><span>&quot;t&quot;</span><span>,</span><span> </span><span>seqidx</span><span>,</span><span> </span><span>&quot;.l&quot;</span><span>,</span><span> </span><span>layeridx</span><span>,</span><span> </span><span>&quot;.i2h&quot;</span><span>))</span><span>
</span><span>h</span><span>2</span><span>h</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.FullyConnected</span><span>(</span><span>data</span><span>=</span><span>prev.state</span><span>$</span><span>h</span><span>,</span><span>
</span><span>weight</span><span>=</span><span>param</span><span>$</span><span>h</span><span>2</span><span>h.weight</span><span>,</span><span>
</span><span>bias</span><span>=</span><span>param</span><span>$</span><span>h</span><span>2</span><span>h.bias</span><span>,</span><span>
</span><span>num.hidden</span><span>=</span><span>num.hidden</span><span>,</span><span>
</span><span>name</span><span>=</span><span>paste0</span><span>(</span><span>&quot;t&quot;</span><span>,</span><span> </span><span>seqidx</span><span>,</span><span> </span><span>&quot;.l&quot;</span><span>,</span><span> </span><span>layeridx</span><span>,</span><span> </span><span>&quot;.h2h&quot;</span><span>))</span><span>
</span><span>hidden</span><span> </span><span>&lt;-</span><span> </span><span>i</span><span>2</span><span>h</span><span> </span><span>+</span><span> </span><span>h</span><span>2</span><span>h</span><span>
</span><span>hidden</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.Activation</span><span>(</span><span>data</span><span>=</span><span>hidden</span><span>,</span><span> </span><span>act.type</span><span>=</span><span>&quot;tanh&quot;</span><span>)</span><span>
</span><span>if</span><span> </span><span>(</span><span>batch.norm</span><span>)</span><span>
</span><span>hidden</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.BatchNorm</span><span>(</span><span>data</span><span>=</span><span>hidden</span><span>)</span><span>
</span><span>return</span><span> </span><span>(</span><span>list</span><span>(</span><span>h</span><span>=</span><span>hidden</span><span>))</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<h4>LSTM Cells</h4>
<p>LSTM replace the cells in custom RNN with LSTM memory block. Figure 2 illustrates the architecture of an LSTM unit. </p>
<p><img align="center" src="https://i2.wp.com/dmlc.ml/rstats/2016/08/img/mxnet/rnn/lstm.png?w=456" data-recalc-dims="1"></p>
<p>It contains one memory cell and three multiplicative units, i.e., the input gate, the forget gate and the output gate. With the help of the memory cell and the gates, LSTM can store and learn long term dependencies across the whole sequence.</p>
<p>A LSTM cell produces the next states by based on the input <code>x</code> as well as previous states (including <code>c</code> and <code>h</code>). For gates, there are three types of connections, including the input to the gate, the (previous) hidden to the gate and the cell to the gate. The activation functions of the gates should use the sigmoid function to make sure the outputs of gates in range [0, 1]. For the memory cell, there are two connections, the input to the cell and the (previous) hidden to the cell.</p>
<div>
<pre><code data-lang="r"><span>lstm</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>num.hidden</span><span>,</span><span> </span><span>indata</span><span>,</span><span> </span><span>prev.state</span><span>,</span><span> </span><span>param</span><span>,</span><span> </span><span>seqidx</span><span>,</span><span> </span><span>layeridx</span><span>,</span><span> </span><span>dropout</span><span>=</span><span>0</span><span>)</span><span> </span><span>{</span><span>
</span><span>if</span><span> </span><span>(</span><span>dropout</span><span> </span><span>></span><span> </span><span>0</span><span>)</span><span>
</span><span>indata</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.Dropout</span><span>(</span><span>data</span><span>=</span><span>indata</span><span>,</span><span> </span><span>p</span><span>=</span><span>dropout</span><span>)</span><span>
</span><span>i</span><span>2</span><span>h</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.FullyConnected</span><span>(</span><span>data</span><span>=</span><span>indata</span><span>,</span><span>
</span><span>weight</span><span>=</span><span>param</span><span>$</span><span>i</span><span>2</span><span>h.weight</span><span>,</span><span>
</span><span>bias</span><span>=</span><span>param</span><span>$</span><span>i</span><span>2</span><span>h.bias</span><span>,</span><span>
</span><span>num.hidden</span><span>=</span><span>num.hidden</span><span> </span><span>*</span><span> </span><span>4</span><span>,</span><span>
</span><span>name</span><span>=</span><span>paste0</span><span>(</span><span>&quot;t&quot;</span><span>,</span><span> </span><span>seqidx</span><span>,</span><span> </span><span>&quot;.l&quot;</span><span>,</span><span> </span><span>layeridx</span><span>,</span><span> </span><span>&quot;.i2h&quot;</span><span>))</span><span>
</span><span>h</span><span>2</span><span>h</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.FullyConnected</span><span>(</span><span>data</span><span>=</span><span>prev.state</span><span>$</span><span>h</span><span>,</span><span>
</span><span>weight</span><span>=</span><span>param</span><span>$</span><span>h</span><span>2</span><span>h.weight</span><span>,</span><span>
</span><span>bias</span><span>=</span><span>param</span><span>$</span><span>h</span><span>2</span><span>h.bias</span><span>,</span><span>
</span><span>num.hidden</span><span>=</span><span>num.hidden</span><span> </span><span>*</span><span> </span><span>4</span><span>,</span><span>
</span><span>name</span><span>=</span><span>paste0</span><span>(</span><span>&quot;t&quot;</span><span>,</span><span> </span><span>seqidx</span><span>,</span><span> </span><span>&quot;.l&quot;</span><span>,</span><span> </span><span>layeridx</span><span>,</span><span> </span><span>&quot;.h2h&quot;</span><span>))</span><span>
</span><span>gates</span><span> </span><span>&lt;-</span><span> </span><span>i</span><span>2</span><span>h</span><span> </span><span>+</span><span> </span><span>h</span><span>2</span><span>h</span><span>
</span><span>slice.gates</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.SliceChannel</span><span>(</span><span>gates</span><span>,</span><span> </span><span>num.outputs</span><span>=</span><span>4</span><span>,</span><span>
</span><span>name</span><span>=</span><span>paste0</span><span>(</span><span>&quot;t&quot;</span><span>,</span><span> </span><span>seqidx</span><span>,</span><span> </span><span>&quot;.l&quot;</span><span>,</span><span> </span><span>layeridx</span><span>,</span><span> </span><span>&quot;.slice&quot;</span><span>))</span><span>
</span><span>in.gate</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.Activation</span><span>(</span><span>slice.gates</span><span>[[</span><span>1</span><span>]],</span><span> </span><span>act.type</span><span>=</span><span>&quot;sigmoid&quot;</span><span>)</span><span>
</span><span>in.transform</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.Activation</span><span>(</span><span>slice.gates</span><span>[[</span><span>2</span><span>]],</span><span> </span><span>act.type</span><span>=</span><span>&quot;tanh&quot;</span><span>)</span><span>
</span><span>forget.gate</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.Activation</span><span>(</span><span>slice.gates</span><span>[[</span><span>3</span><span>]],</span><span> </span><span>act.type</span><span>=</span><span>&quot;sigmoid&quot;</span><span>)</span><span>
</span><span>out.gate</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.Activation</span><span>(</span><span>slice.gates</span><span>[[</span><span>4</span><span>]],</span><span> </span><span>act.type</span><span>=</span><span>&quot;sigmoid&quot;</span><span>)</span><span>
</span><span>next.c</span><span> </span><span>&lt;-</span><span> </span><span>(</span><span>forget.gate</span><span> </span><span>*</span><span> </span><span>prev.state</span><span>$</span><span>c</span><span>)</span><span> </span><span>+</span><span> </span><span>(</span><span>in.gate</span><span> </span><span>*</span><span> </span><span>in.transform</span><span>)</span><span>
</span><span>next.h</span><span> </span><span>&lt;-</span><span> </span><span>out.gate</span><span> </span><span>*</span><span> </span><span>mx.symbol.Activation</span><span>(</span><span>next.c</span><span>,</span><span> </span><span>act.type</span><span>=</span><span>&quot;tanh&quot;</span><span>)</span><span>
</span><span>return</span><span> </span><span>(</span><span>list</span><span>(</span><span>c</span><span>=</span><span>next.c</span><span>,</span><span> </span><span>h</span><span>=</span><span>next.h</span><span>))</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<p>Instead of defining the gates and the memory cell independently, we compute them together and then use <code>mx.symbol.SliceChannel</code> to split them into four outputs. </p>
<h4>GRU Cells</h4>
<p>GRU is another variant model of RNN, which was proposed in 2014 [2]. Similar to LSTM unit, the GRU unit also aims to adaptively capture dependencies of different time scales by updating and reseting gate as shown in Figure 3.</p>
<p><img align="center" src="https://i2.wp.com/dmlc.ml/rstats/2016/08/img/mxnet/rnn/gru.png?w=456" data-recalc-dims="1"></p>
<p>The calculation of GRU is similar with the custom RNN and LSTM models. First, there are two types of connections for gates to be defined: <code>update.gate</code> decides how much the unit updates its activations and <code>reset.gate</code> which decides whether to forget the previously computed state. Then, the candidate activation <code>htrans</code> is computed using input data <code>x</code> and previous state <code>h</code> (<code>reset.gate</code> controls whether to forget the previous state). After getting <code>htrans</code>, use <code>update.gate</code> to decide the proportion of <code>htrans</code> and previous state <code>h</code> to calculate next state <code>h</code>.</p>
<div>
<pre><code data-lang="r"><span>gru</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>num.hidden</span><span>,</span><span> </span><span>indata</span><span>,</span><span> </span><span>prev.state</span><span>,</span><span> </span><span>param</span><span>,</span><span> </span><span>seqidx</span><span>,</span><span> </span><span>layeridx</span><span>,</span><span> </span><span>dropout</span><span>=</span><span>0</span><span>)</span><span> </span><span>{</span><span>
</span><span>if</span><span> </span><span>(</span><span>dropout</span><span> </span><span>></span><span> </span><span>0</span><span>)</span><span>
</span><span>indata</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.Dropout</span><span>(</span><span>data</span><span>=</span><span>indata</span><span>,</span><span> </span><span>p</span><span>=</span><span>dropout</span><span>)</span><span>
</span><span>i</span><span>2</span><span>h</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.FullyConnected</span><span>(</span><span>data</span><span>=</span><span>indata</span><span>,</span><span>
</span><span>weight</span><span>=</span><span>param</span><span>$</span><span>gates.i2h.weight</span><span>,</span><span>
</span><span>bias</span><span>=</span><span>param</span><span>$</span><span>gates.i2h.bias</span><span>,</span><span>
</span><span>num.hidden</span><span>=</span><span>num.hidden</span><span> </span><span>*</span><span> </span><span>2</span><span>,</span><span>
</span><span>name</span><span>=</span><span>paste0</span><span>(</span><span>&quot;t&quot;</span><span>,</span><span> </span><span>seqidx</span><span>,</span><span> </span><span>&quot;.l&quot;</span><span>,</span><span> </span><span>layeridx</span><span>,</span><span> </span><span>&quot;.gates.i2h&quot;</span><span>))</span><span>
</span><span>h</span><span>2</span><span>h</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.FullyConnected</span><span>(</span><span>data</span><span>=</span><span>prev.state</span><span>$</span><span>h</span><span>,</span><span>
</span><span>weight</span><span>=</span><span>param</span><span>$</span><span>gates.h2h.weight</span><span>,</span><span>
</span><span>bias</span><span>=</span><span>param</span><span>$</span><span>gates.h2h.bias</span><span>,</span><span>
</span><span>num.hidden</span><span>=</span><span>num.hidden</span><span> </span><span>*</span><span> </span><span>2</span><span>,</span><span>
</span><span>name</span><span>=</span><span>paste0</span><span>(</span><span>&quot;t&quot;</span><span>,</span><span> </span><span>seqidx</span><span>,</span><span> </span><span>&quot;.l&quot;</span><span>,</span><span> </span><span>layeridx</span><span>,</span><span> </span><span>&quot;.gates.h2h&quot;</span><span>))</span><span>
</span><span>gates</span><span> </span><span>&lt;-</span><span> </span><span>i</span><span>2</span><span>h</span><span> </span><span>+</span><span> </span><span>h</span><span>2</span><span>h</span><span>
</span><span>slice.gates</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.SliceChannel</span><span>(</span><span>gates</span><span>,</span><span> </span><span>num.outputs</span><span>=</span><span>2</span><span>,</span><span>
</span><span>name</span><span>=</span><span>paste0</span><span>(</span><span>&quot;t&quot;</span><span>,</span><span> </span><span>seqidx</span><span>,</span><span> </span><span>&quot;.l&quot;</span><span>,</span><span> </span><span>layeridx</span><span>,</span><span> </span><span>&quot;.slice&quot;</span><span>))</span><span>
</span><span>update.gate</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.Activation</span><span>(</span><span>slice.gates</span><span>[[</span><span>1</span><span>]],</span><span> </span><span>act.type</span><span>=</span><span>&quot;sigmoid&quot;</span><span>)</span><span>
</span><span>reset.gate</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.Activation</span><span>(</span><span>slice.gates</span><span>[[</span><span>2</span><span>]],</span><span> </span><span>act.type</span><span>=</span><span>&quot;sigmoid&quot;</span><span>)</span><span>
</span><span>htrans.i2h</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.FullyConnected</span><span>(</span><span>data</span><span>=</span><span>indata</span><span>,</span><span>
</span><span>weight</span><span>=</span><span>param</span><span>$</span><span>trans.i2h.weight</span><span>,</span><span>
</span><span>bias</span><span>=</span><span>param</span><span>$</span><span>trans.i2h.bias</span><span>,</span><span>
</span><span>num.hidden</span><span>=</span><span>num.hidden</span><span>,</span><span>
</span><span>name</span><span>=</span><span>paste0</span><span>(</span><span>&quot;t&quot;</span><span>,</span><span> </span><span>seqidx</span><span>,</span><span> </span><span>&quot;.l&quot;</span><span>,</span><span> </span><span>layeridx</span><span>,</span><span> </span><span>&quot;.trans.i2h&quot;</span><span>))</span><span>
</span><span>h.after.reset</span><span> </span><span>&lt;-</span><span> </span><span>prev.state</span><span>$</span><span>h</span><span> </span><span>*</span><span> </span><span>reset.gate</span><span>
</span><span>htrans.h2h</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.FullyConnected</span><span>(</span><span>data</span><span>=</span><span>h.after.reset</span><span>,</span><span>
</span><span>weight</span><span>=</span><span>param</span><span>$</span><span>trans.h2h.weight</span><span>,</span><span>
</span><span>bias</span><span>=</span><span>param</span><span>$</span><span>trans.h2h.bias</span><span>,</span><span>
</span><span>num.hidden</span><span>=</span><span>num.hidden</span><span>,</span><span>
</span><span>name</span><span>=</span><span>paste0</span><span>(</span><span>&quot;t&quot;</span><span>,</span><span> </span><span>seqidx</span><span>,</span><span> </span><span>&quot;.l&quot;</span><span>,</span><span> </span><span>layeridx</span><span>,</span><span> </span><span>&quot;.trans.h2h&quot;</span><span>))</span><span>
</span><span>h.trans</span><span> </span><span>&lt;-</span><span> </span><span>htrans.i2h</span><span> </span><span>+</span><span> </span><span>htrans.h2h</span><span>
</span><span>h.trans.active</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.Activation</span><span>(</span><span>h.trans</span><span>,</span><span> </span><span>act.type</span><span>=</span><span>&quot;tanh&quot;</span><span>)</span><span>
</span><span>next.h</span><span> </span><span>&lt;-</span><span> </span><span>prev.state</span><span>$</span><span>h</span><span> </span><span>+</span><span> </span><span>update.gate</span><span> </span><span>*</span><span> </span><span>(</span><span>h.trans.active</span><span> </span><span>-</span><span> </span><span>prev.state</span><span>$</span><span>h</span><span>)</span><span>
</span><span>return</span><span> </span><span>(</span><span>list</span><span>(</span><span>h</span><span>=</span><span>next.h</span><span>))</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<h3>Unrolling RNN Models</h3>
<p>Since MXNet has implemented low level layers, we can unroll the RNN model in the time dimension and then use the MXNet layers to construct the different RNN networks according the above defined RNN cells. After unrolling the RNN in time, the model is just like the common feedforward network unless the weights shared in different feedforward layers and specific different recurrent units. Specifically, we can use fully-connected layers and corresponding activations to represent the different types of connections. </p>
<p>The unroll function needs to unroll the recurrent model according to the predefined sequence length. The recurrent weights across time need to be shared and the network depth represents the number of recurrent layers.</p>
<p>We provide the unrolling functions which is suitable for tasks like character language model and PennTreeBank language model. For other tasks the unrolling functions are similiar. </p>
<p>First, we define the weights and states. <code>embed.weight</code> is the weights used for embedding layer, mapping from the one-hot input to a dense vector. <code>cls.weight</code> and <code>cls.bias</code> are the weights and bias for the final prediction at each time step. <code>param.cells</code> and <code>last.states</code> is the weights and states for each cell. The weights are shared and the states are different and will be updates over time.</p>
<div>
<pre><code data-lang="r"><span>lstm.unroll</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>num.lstm.layer</span><span>,</span><span> </span><span>seq.len</span><span>,</span><span> </span><span>input.size</span><span>,</span><span>
</span><span>num.hidden</span><span>,</span><span> </span><span>num.embed</span><span>,</span><span> </span><span>num.label</span><span>,</span><span> </span><span>dropout</span><span>=</span><span>0</span><span>.</span><span>)</span><span> </span><span>{</span><span>
</span><span>embed.weight</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.Variable</span><span>(</span><span>&quot;embed.weight&quot;</span><span>)</span><span>
</span><span>cls.weight</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.Variable</span><span>(</span><span>&quot;cls.weight&quot;</span><span>)</span><span>
</span><span>cls.bias</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.Variable</span><span>(</span><span>&quot;cls.bias&quot;</span><span>)</span><span>
</span><span>param.cells</span><span> </span><span>&lt;-</span><span> </span><span>lapply</span><span>(</span><span>1</span><span>:</span><span>num.lstm.layer</span><span>,</span><span> </span><span>function</span><span>(</span><span>i</span><span>)</span><span> </span><span>{</span><span>
</span><span>cell</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>(</span><span>i</span><span>2</span><span>h.weight</span><span> </span><span>=</span><span> </span><span>mx.symbol.Variable</span><span>(</span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.i2h.weight&quot;</span><span>)),</span><span>
</span><span>i</span><span>2</span><span>h.bias</span><span> </span><span>=</span><span> </span><span>mx.symbol.Variable</span><span>(</span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.i2h.bias&quot;</span><span>)),</span><span>
</span><span>h</span><span>2</span><span>h.weight</span><span> </span><span>=</span><span> </span><span>mx.symbol.Variable</span><span>(</span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.h2h.weight&quot;</span><span>)),</span><span>
</span><span>h</span><span>2</span><span>h.bias</span><span> </span><span>=</span><span> </span><span>mx.symbol.Variable</span><span>(</span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.h2h.bias&quot;</span><span>)))</span><span>
</span><span>return</span><span> </span><span>(</span><span>cell</span><span>)</span><span>
</span><span>})</span><span>
</span><span>last.states</span><span> </span><span>&lt;-</span><span> </span><span>lapply</span><span>(</span><span>1</span><span>:</span><span>num.lstm.layer</span><span>,</span><span> </span><span>function</span><span>(</span><span>i</span><span>)</span><span> </span><span>{</span><span>
</span><span>state</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>(</span><span>c</span><span>=</span><span>mx.symbol.Variable</span><span>(</span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.init.c&quot;</span><span>)),</span><span>
</span><span>h</span><span>=</span><span>mx.symbol.Variable</span><span>(</span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.init.h&quot;</span><span>)))</span><span>
</span><span>return</span><span> </span><span>(</span><span>state</span><span>)</span><span>
</span><span>})</span><span>
</span></code></pre>
</div>
<p>Then we unroll the RNN model in the time dimension and then use the MXNet layers to construct the RNN networks. Here the <code>mx.symbol.Embedding</code> is used to get the embedding vector for the specific task (char-rnn) shown there. At each time step, we share the weights <code>param.cells</code> and update the states <code>last.states</code>. Also, <code>last.hidden</code> is used to collect the outputs over time.</p>
<div>
<pre><code data-lang="r"><span> </span><span># embeding layer
</span><span> </span><span>label</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.Variable</span><span>(</span><span>&quot;label&quot;</span><span>)</span><span>
</span><span>data</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.Variable</span><span>(</span><span>&quot;data&quot;</span><span>)</span><span>
</span><span>embed</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.Embedding</span><span>(</span><span>data</span><span>=</span><span>data</span><span>,</span><span> </span><span>input_dim</span><span>=</span><span>input.size</span><span>,</span><span>
</span><span>weight</span><span>=</span><span>embed.weight</span><span>,</span><span> </span><span>output_dim</span><span>=</span><span>num.embed</span><span>,</span><span> </span><span>name</span><span>=</span><span>&quot;embed&quot;</span><span>)</span><span>
</span><span>wordvec</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.SliceChannel</span><span>(</span><span>data</span><span>=</span><span>embed</span><span>,</span><span> </span><span>num_outputs</span><span>=</span><span>seq.len</span><span>,</span><span> </span><span>squeeze_axis</span><span>=</span><span>1</span><span>)</span><span>
</span><span>last.hidden</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>()</span><span>
</span><span>for</span><span> </span><span>(</span><span>seqidx</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>seq.len</span><span>)</span><span> </span><span>{</span><span>
</span><span>hidden</span><span> </span><span>&lt;-</span><span> </span><span>wordvec</span><span>[[</span><span>seqidx</span><span>]]</span><span>
</span><span># stack lstm
</span><span> </span><span>for</span><span> </span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>num.lstm.layer</span><span>)</span><span> </span><span>{</span><span>
</span><span>dp</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>i</span><span>==</span><span>1</span><span>,</span><span> </span><span>0</span><span>,</span><span> </span><span>dropout</span><span>)</span><span>
</span><span>next.state</span><span> </span><span>&lt;-</span><span> </span><span>lstm</span><span>(</span><span>num.hidden</span><span>,</span><span> </span><span>indata</span><span>=</span><span>hidden</span><span>,</span><span>
</span><span>prev.state</span><span>=</span><span>last.states</span><span>[[</span><span>i</span><span>]],</span><span>
</span><span>param</span><span>=</span><span>param.cells</span><span>[[</span><span>i</span><span>]],</span><span>
</span><span>seqidx</span><span>=</span><span>seqidx</span><span>,</span><span> </span><span>layeridx</span><span>=</span><span>i</span><span>,</span><span>
</span><span>dropout</span><span>=</span><span>dp</span><span>)</span><span>
</span><span>hidden</span><span> </span><span>&lt;-</span><span> </span><span>next.state</span><span>$</span><span>h</span><span>
</span><span>last.states</span><span>[[</span><span>i</span><span>]]</span><span> </span><span>&lt;-</span><span> </span><span>next.state</span><span>
</span><span>}</span><span>
</span><span># decoder
</span><span> </span><span>if</span><span> </span><span>(</span><span>dropout</span><span> </span><span>></span><span> </span><span>0</span><span>)</span><span>
</span><span>hidden</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.Dropout</span><span>(</span><span>data</span><span>=</span><span>hidden</span><span>,</span><span> </span><span>p</span><span>=</span><span>dropout</span><span>)</span><span>
</span><span>last.hidden</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>last.hidden</span><span>,</span><span> </span><span>hidden</span><span>)</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<p>Finally, we need to construct the remain layers according to the different tasks. Take char-rnn as an example, <code>cls.weight</code> and <code>cls.bias</code> are used for the final prediction and then <code>mx.symbol.SoftmaxOutput</code> connnets the prediction and corresponding labels to back propagate though time. </p>
<div>
<pre><code data-lang="r"><span> </span><span>last.hidden</span><span>$</span><span>dim</span><span> </span><span>&lt;-</span><span> </span><span>0</span><span>
</span><span>last.hidden</span><span>$</span><span>num.args</span><span> </span><span>&lt;-</span><span> </span><span>seq.len</span><span>
</span><span>concat</span><span> </span><span>&lt;-</span><span>mxnet</span><span>:::</span><span>mx.varg.symbol.Concat</span><span>(</span><span>last.hidden</span><span>)</span><span>
</span><span>fc</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.FullyConnected</span><span>(</span><span>data</span><span>=</span><span>concat</span><span>,</span><span>
</span><span>weight</span><span>=</span><span>cls.weight</span><span>,</span><span>
</span><span>bias</span><span>=</span><span>cls.bias</span><span>,</span><span>
</span><span>num.hidden</span><span>=</span><span>num.label</span><span>)</span><span>
</span><span>label</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.transpose</span><span>(</span><span>data</span><span>=</span><span>label</span><span>)</span><span>
</span><span>label</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.Reshape</span><span>(</span><span>data</span><span>=</span><span>label</span><span>,</span><span> </span><span>target.shape</span><span>=</span><span>c</span><span>(</span><span>0</span><span>))</span><span>
</span><span>loss.all</span><span> </span><span>&lt;-</span><span> </span><span>mx.symbol.SoftmaxOutput</span><span>(</span><span>data</span><span>=</span><span>fc</span><span>,</span><span> </span><span>label</span><span>=</span><span>label</span><span>,</span><span> </span><span>name</span><span>=</span><span>&quot;sm&quot;</span><span>)</span><span>
</span><span>return</span><span> </span><span>(</span><span>loss.all</span><span>)</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<h3>Training RNN Models</h3>
<p>After implementing unrolled RNN models, we need to know how to train the recurrent models. We write the training code on our own using the low level symbol interfaces instead of the common FeedForward model. The reason why we implement by ourselves is two-fold. First, currently, existing data iterators do not support sequence data well. Second, since our input contains both the input data x and the states i.e h and we need set the gradients of them to zero at each epoch, the existing high level FeedForward interface is not appropriate for our tasks.</p>
<p>For the reasons above, we write the training code on our own to bind the network and train epoch by epoch using the low level symbol interfaces. You can refer to the code in <a target="_blank" href="https://github.com/dmlc/mxnet/blob/master/R-package/R/rnn_model.R">rnn_model.R</a>. The training method of three RNN models are the same except the initial states are <code>init.h</code> or <code>init.c</code> and <code>init.h</code>. So they can use the same training function.</p>
<p>The training codes mainly contain the function <code>setup.rnn.model</code> and the function <code>train.rnn</code>.</p>
<h4>Set Up RNN Models</h4>
<p><code>setup.rnn.model</code> is defined to initialize parameters and bind the network. The parameter <code>init.states.name</code> stores the name of the initial state. For custom RNN and GRU, <code>init.states.name</code> is end with <code>init.h</code>. For LSTM, <code>init.states.name</code> is end with <code>init.h</code> and <code>init.c</code>.</p>
<p>First, we set the dimension of the input including input data, label and initial states. Given the input dimension, we can use <code>mx.model.init.params.rnn</code> function to initialize parameters.</p>
<div>
<pre><code data-lang="r"><span>setup.rnn.model</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>rnn.sym</span><span>,</span><span> </span><span>ctx</span><span>,</span><span>
</span><span>num.rnn.layer</span><span>,</span><span> </span><span>seq.len</span><span>,</span><span>
</span><span>num.hidden</span><span>,</span><span> </span><span>num.embed</span><span>,</span><span> </span><span>num.label</span><span>,</span><span>
</span><span>batch.size</span><span>,</span><span> </span><span>input.size</span><span>,</span><span>
</span><span>init.states.name</span><span>,</span><span>
</span><span>initializer</span><span>=</span><span>mx.init.uniform</span><span>(</span><span>0.01</span><span>),</span><span>
</span><span>dropout</span><span>=</span><span>0</span><span>)</span><span> </span><span>{</span><span>
</span><span>arg.names</span><span> </span><span>&lt;-</span><span> </span><span>rnn.sym</span><span>$</span><span>arguments</span><span>
</span><span>input.shapes</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>()</span><span>
</span><span>for</span><span> </span><span>(</span><span>name</span><span> </span><span>in</span><span> </span><span>arg.names</span><span>)</span><span> </span><span>{</span><span>
</span><span>if</span><span> </span><span>(</span><span>name</span><span> </span><span>%in%</span><span> </span><span>init.states.name</span><span>)</span><span> </span><span>{</span><span>
</span><span>input.shapes</span><span>[[</span><span>name</span><span>]]</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>num.hidden</span><span>,</span><span> </span><span>batch.size</span><span>)</span><span>
</span><span>}</span><span>
</span><span>else</span><span> </span><span>if</span><span> </span><span>(</span><span>grepl</span><span>(</span><span>'data$'</span><span>,</span><span> </span><span>name</span><span>)</span><span> </span><span>||</span><span> </span><span>grepl</span><span>(</span><span>'label$'</span><span>,</span><span> </span><span>name</span><span>)</span><span> </span><span>)</span><span> </span><span>{</span><span>
</span><span>if</span><span> </span><span>(</span><span>seq.len</span><span> </span><span>==</span><span> </span><span>1</span><span>)</span><span> </span><span>{</span><span>
</span><span>input.shapes</span><span>[[</span><span>name</span><span>]]</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>batch.size</span><span>)</span><span>
</span><span>}</span><span> </span><span>else</span><span> </span><span>{</span><span>
</span><span>input.shapes</span><span>[[</span><span>name</span><span>]]</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>seq.len</span><span>,</span><span> </span><span>batch.size</span><span>)</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span><span>params</span><span> </span><span>&lt;-</span><span> </span><span>mx.model.init.params.rnn</span><span>(</span><span>rnn.sym</span><span>,</span><span> </span><span>input.shapes</span><span>,</span><span> </span><span>initializer</span><span>,</span><span> </span><span>mx.cpu</span><span>())</span><span>
</span></code></pre>
</div>
<p>Next, we use <code>mx.simple.bind</code> to bind the network, set the <code>arg.arrays</code>, <code>aux.arrays</code> and <code>grad.arrays</code>.</p>
<div>
<pre><code data-lang="r"><span> </span><span>args</span><span> </span><span>&lt;-</span><span> </span><span>input.shapes</span><span>
</span><span>args</span><span>$</span><span>symbol</span><span> </span><span>&lt;-</span><span> </span><span>rnn.sym</span><span>
</span><span>args</span><span>$</span><span>ctx</span><span> </span><span>&lt;-</span><span> </span><span>ctx</span><span>
</span><span>args</span><span>$</span><span>grad.req</span><span> </span><span>&lt;-</span><span> </span><span>&quot;add&quot;</span><span>
</span><span>rnn.exec</span><span> </span><span>&lt;-</span><span> </span><span>do.call</span><span>(</span><span>mx.simple.bind</span><span>,</span><span> </span><span>args</span><span>)</span><span>
</span><span>mx.exec.update.arg.arrays</span><span>(</span><span>rnn.exec</span><span>,</span><span> </span><span>params</span><span>$</span><span>arg.params</span><span>,</span><span> </span><span>match.name</span><span>=</span><span>TRUE</span><span>)</span><span>
</span><span>mx.exec.update.aux.arrays</span><span>(</span><span>rnn.exec</span><span>,</span><span> </span><span>params</span><span>$</span><span>aux.params</span><span>,</span><span> </span><span>match.name</span><span>=</span><span>TRUE</span><span>)</span><span>
</span><span>grad.arrays</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>()</span><span>
</span><span>for</span><span> </span><span>(</span><span>name</span><span> </span><span>in</span><span> </span><span>names</span><span>(</span><span>rnn.exec</span><span>$</span><span>ref.grad.arrays</span><span>))</span><span> </span><span>{</span><span>
</span><span>if</span><span> </span><span>(</span><span>is.param.name</span><span>(</span><span>name</span><span>))</span><span>
</span><span>grad.arrays</span><span>[[</span><span>name</span><span>]]</span><span> </span><span>&lt;-</span><span> </span><span>rnn.exec</span><span>$</span><span>ref.arg.arrays</span><span>[[</span><span>name</span><span>]]</span><span>*</span><span>0</span><span>
</span><span>}</span><span>
</span><span>mx.exec.update.grad.arrays</span><span>(</span><span>rnn.exec</span><span>,</span><span> </span><span>grad.arrays</span><span>,</span><span> </span><span>match.name</span><span>=</span><span>TRUE</span><span>)</span><span>
</span><span>return</span><span> </span><span>(</span><span>list</span><span>(</span><span>rnn.exec</span><span>=</span><span>rnn.exec</span><span>,</span><span> </span><span>symbol</span><span>=</span><span>rnn.sym</span><span>,</span><span>
</span><span>num.rnn.layer</span><span>=</span><span>num.rnn.layer</span><span>,</span><span> </span><span>num.hidden</span><span>=</span><span>num.hidden</span><span>,</span><span>
</span><span>seq.len</span><span>=</span><span>seq.len</span><span>,</span><span> </span><span>batch.size</span><span>=</span><span>batch.size</span><span>,</span><span>
</span><span>num.embed</span><span>=</span><span>num.embed</span><span>))</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<h4>Training RNN Models</h4>
<p><code>train.rnn</code> function is to train network epoch by epoch, update the parameters using <code>mx.exec.forward</code> and <code>mx.exec.backward</code>, which is similar to the function <code>mx.model.train</code> in <a target="_blank" href="https://github.com/dmlc/mxnet/blob/master/R-package/R/model.R">model.R</a>. One difference is that in <code>train.rnn</code>, at the beginning of each epoch, the initial states should be clear.</p>
<div>
<pre><code data-lang="r"><span>for</span><span> </span><span>(</span><span>name</span><span> </span><span>in</span><span> </span><span>init.states.name</span><span>)</span><span> </span><span>{</span><span>
</span><span>init.states</span><span>[[</span><span>name</span><span>]]</span><span> </span><span>&lt;-</span><span> </span><span>m</span><span>$</span><span>rnn.exec</span><span>$</span><span>ref.arg.arrays</span><span>[[</span><span>name</span><span>]]</span><span>*</span><span>0</span><span>
</span><span>}</span><span>
</span><span>mx.exec.update.arg.arrays</span><span>(</span><span>m</span><span>$</span><span>rnn.exec</span><span>,</span><span> </span><span>init.states</span><span>,</span><span> </span><span>match.name</span><span>=</span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<p>Also, at the end of the epoch, when updating the gradient of parameters, the gradient of initial <code>c</code> and initial <code>h</code> should be zero.</p>
<div>
<pre><code data-lang="r"><span># the gradient of initial c and inital h should be zero
</span><span>init.grad</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>()</span><span>
</span><span>for</span><span> </span><span>(</span><span>name</span><span> </span><span>in</span><span> </span><span>init.states.name</span><span>)</span><span> </span><span>{</span><span>
</span><span>init.grad</span><span>[[</span><span>name</span><span>]]</span><span> </span><span>&lt;-</span><span> </span><span>m</span><span>$</span><span>rnn.exec</span><span>$</span><span>ref.arg.arrays</span><span>[[</span><span>name</span><span>]]</span><span>*</span><span>0</span><span>
</span><span>}</span><span>
</span><span>mx.exec.update.grad.arrays</span><span>(</span><span>m</span><span>$</span><span>rnn.exec</span><span>,</span><span> </span><span>init.grad</span><span>,</span><span> </span><span>match.name</span><span>=</span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<h4>APIs of custom RNN, LSTM, GRU</h4>
<p><code>mx.rnn</code>, <code>mx.lstm</code>, <code>mx.gru</code> is the APIs for training custom RNN / LSTM / GRU unrolled model. They will first change the data into iterator and then get the unrolled rnn symbol. After getting the unrolled rnn symbol, the function sets up the training model, binds the network and then trains the model to get the final result.</p>
<p>We show the <code>mx.lstm</code> function, the other two are similar to it.</p>
<div>
<pre><code data-lang="r"><span>mx.lstm</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>train.data</span><span>,</span><span> </span><span>eval.data</span><span>=</span><span>NULL</span><span>,</span><span>
</span><span>num.lstm.layer</span><span>,</span><span> </span><span>seq.len</span><span>,</span><span>
</span><span>num.hidden</span><span>,</span><span> </span><span>num.embed</span><span>,</span><span> </span><span>num.label</span><span>,</span><span>
</span><span>batch.size</span><span>,</span><span> </span><span>input.size</span><span>,</span><span>
</span><span>ctx</span><span>=</span><span>mx.ctx.default</span><span>(),</span><span>
</span><span>num.round</span><span>=</span><span>10</span><span>,</span><span> </span><span>update.period</span><span>=</span><span>1</span><span>,</span><span>
</span><span>initializer</span><span>=</span><span>mx.init.uniform</span><span>(</span><span>0.01</span><span>),</span><span>
</span><span>dropout</span><span>=</span><span>0</span><span>,</span><span> </span><span>optimizer</span><span>=</span><span>'sgd'</span><span>,</span><span>
</span><span>...</span><span>)</span><span> </span><span>{</span><span>
</span><span># check data and change data into iterator
</span><span> </span><span>train.data</span><span> </span><span>&lt;-</span><span> </span><span>check.data</span><span>(</span><span>train.data</span><span>,</span><span> </span><span>batch.size</span><span>,</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>eval.data</span><span> </span><span>&lt;-</span><span> </span><span>check.data</span><span>(</span><span>eval.data</span><span>,</span><span> </span><span>batch.size</span><span>,</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span># get unrolled lstm symbol
</span><span> </span><span>rnn.sym</span><span> </span><span>&lt;-</span><span> </span><span>lstm.unroll</span><span>(</span><span>num.lstm.layer</span><span>=</span><span>num.lstm.layer</span><span>,</span><span>
</span><span>num.hidden</span><span>=</span><span>num.hidden</span><span>,</span><span>
</span><span>seq.len</span><span>=</span><span>seq.len</span><span>,</span><span>
</span><span>input.size</span><span>=</span><span>input.size</span><span>,</span><span>
</span><span>num.embed</span><span>=</span><span>num.embed</span><span>,</span><span>
</span><span>num.label</span><span>=</span><span>num.label</span><span>,</span><span>
</span><span>dropout</span><span>=</span><span>dropout</span><span>)</span><span>
</span><span>init.states.c</span><span> </span><span>&lt;-</span><span> </span><span>lapply</span><span>(</span><span>1</span><span>:</span><span>num.lstm.layer</span><span>,</span><span> </span><span>function</span><span>(</span><span>i</span><span>)</span><span> </span><span>{</span><span>
</span><span>state.c</span><span> </span><span>&lt;-</span><span> </span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.init.c&quot;</span><span>)</span><span>
</span><span>return</span><span> </span><span>(</span><span>state.c</span><span>)</span><span>
</span><span>})</span><span>
</span><span>init.states.h</span><span> </span><span>&lt;-</span><span> </span><span>lapply</span><span>(</span><span>1</span><span>:</span><span>num.lstm.layer</span><span>,</span><span> </span><span>function</span><span>(</span><span>i</span><span>)</span><span> </span><span>{</span><span>
</span><span>state.h</span><span> </span><span>&lt;-</span><span> </span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.init.h&quot;</span><span>)</span><span>
</span><span>return</span><span> </span><span>(</span><span>state.h</span><span>)</span><span>
</span><span>})</span><span>
</span><span>init.states.name</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>init.states.c</span><span>,</span><span> </span><span>init.states.h</span><span>)</span><span>
</span><span># set up lstm model
</span><span> </span><span>model</span><span> </span><span>&lt;-</span><span> </span><span>setup.rnn.model</span><span>(</span><span>rnn.sym</span><span>=</span><span>rnn.sym</span><span>,</span><span>
</span><span>ctx</span><span>=</span><span>ctx</span><span>,</span><span>
</span><span>num.rnn.layer</span><span>=</span><span>num.lstm.layer</span><span>,</span><span>
</span><span>seq.len</span><span>=</span><span>seq.len</span><span>,</span><span>
</span><span>num.hidden</span><span>=</span><span>num.hidden</span><span>,</span><span>
</span><span>num.embed</span><span>=</span><span>num.embed</span><span>,</span><span>
</span><span>num.label</span><span>=</span><span>num.label</span><span>,</span><span>
</span><span>batch.size</span><span>=</span><span>batch.size</span><span>,</span><span>
</span><span>input.size</span><span>=</span><span>input.size</span><span>,</span><span>
</span><span>init.states.name</span><span>=</span><span>init.states.name</span><span>,</span><span>
</span><span>initializer</span><span>=</span><span>initializer</span><span>,</span><span>
</span><span>dropout</span><span>=</span><span>dropout</span><span>)</span><span>
</span><span># train lstm model
</span><span> </span><span>model</span><span> </span><span>&lt;-</span><span> </span><span>train.rnn</span><span>(</span><span> </span><span>model</span><span>,</span><span> </span><span>train.data</span><span>,</span><span> </span><span>eval.data</span><span>,</span><span>
</span><span>num.round</span><span>=</span><span>num.round</span><span>,</span><span>
</span><span>update.period</span><span>=</span><span>update.period</span><span>,</span><span>
</span><span>ctx</span><span>=</span><span>ctx</span><span>,</span><span>
</span><span>init.states.name</span><span>=</span><span>init.states.name</span><span>,</span><span>
</span><span>...</span><span>)</span><span>
</span><span># change model into MXFeedForwardModel
</span><span> </span><span>model</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>(</span><span>symbol</span><span>=</span><span>model</span><span>$</span><span>symbol</span><span>,</span><span> </span><span>arg.params</span><span>=</span><span>model</span><span>$</span><span>rnn.exec</span><span>$</span><span>ref.arg.arrays</span><span>,</span><span> </span><span>aux.params</span><span>=</span><span>model</span><span>$</span><span>rnn.exec</span><span>$</span><span>ref.aux.arrays</span><span>)</span><span>
</span><span>return</span><span>(</span><span>structure</span><span>(</span><span>model</span><span>,</span><span> </span><span>class</span><span>=</span><span>&quot;MXFeedForwardModel&quot;</span><span>))</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<h3>RNN Inference Models</h3>
<p>In the tasks like character language model and PennTreeBank language model, we unroll rnn models with specific <code>seq.len</code> when training models. After we get the trained models, we may want to use it with different <code>seq.len</code> from the unrolled rnn models. For example, as for char-rnn model, we first train a char level language model with <code>seq.len=128</code>. Then we may want to generate a 100-length text from it. The weights of rnn cells are shared at each time step, so we only need to store a network with <code>seq.len=1</code> after training.</p>
<h4>RNN Inference APIs</h4>
<p><code>mx.rnn.inference</code>, <code>mx.lstm.inference</code>, <code>mx.gru.inference</code> is used to create a custom RNN / LSTM / GRU inference model with parameter <code>seq.len=1</code>. We show the codes of <code>mx.lstm.inference</code> and the other two are similar to it. The parameter <code>arg.params</code> stores the parameters of the trained unrolled rnn model. We build the network with <code>seq.len=1</code>. Then we set up the rnn model with initialization and binding. We use <code>mx.exec.update.arg.arrays</code> to set the <code>arg.arrays</code> to <code>arg.params</code> of the trained model. The initial states are cleared as well.</p>
<div>
<pre><code data-lang="r"><span>mx.lstm.inference</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>num.lstm.layer</span><span>,</span><span>
</span><span>input.size</span><span>,</span><span>
</span><span>num.hidden</span><span>,</span><span>
</span><span>num.embed</span><span>,</span><span>
</span><span>num.label</span><span>,</span><span>
</span><span>batch.size</span><span>=</span><span>1</span><span>,</span><span>
</span><span>arg.params</span><span>,</span><span>
</span><span>ctx</span><span>=</span><span>mx.cpu</span><span>(),</span><span>
</span><span>dropout</span><span>=</span><span>0</span><span>.</span><span>)</span><span> </span><span>{</span><span>
</span><span>sym</span><span> </span><span>&lt;-</span><span> </span><span>lstm.inference.symbol</span><span>(</span><span>num.lstm.layer</span><span>=</span><span>num.lstm.layer</span><span>,</span><span>
</span><span>input.size</span><span>=</span><span>input.size</span><span>,</span><span>
</span><span>num.hidden</span><span>=</span><span>num.hidden</span><span>,</span><span>
</span><span>num.embed</span><span>=</span><span>num.embed</span><span>,</span><span>
</span><span>num.label</span><span>=</span><span>num.label</span><span>,</span><span>
</span><span>dropout</span><span>=</span><span>dropout</span><span>)</span><span>
</span><span>init.states.c</span><span> </span><span>&lt;-</span><span> </span><span>lapply</span><span>(</span><span>1</span><span>:</span><span>num.lstm.layer</span><span>,</span><span> </span><span>function</span><span>(</span><span>i</span><span>)</span><span> </span><span>{</span><span>
</span><span>state.c</span><span> </span><span>&lt;-</span><span> </span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.init.c&quot;</span><span>)</span><span>
</span><span>return</span><span> </span><span>(</span><span>state.c</span><span>)</span><span>
</span><span>})</span><span>
</span><span>init.states.h</span><span> </span><span>&lt;-</span><span> </span><span>lapply</span><span>(</span><span>1</span><span>:</span><span>num.lstm.layer</span><span>,</span><span> </span><span>function</span><span>(</span><span>i</span><span>)</span><span> </span><span>{</span><span>
</span><span>state.h</span><span> </span><span>&lt;-</span><span> </span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.init.h&quot;</span><span>)</span><span>
</span><span>return</span><span> </span><span>(</span><span>state.h</span><span>)</span><span>
</span><span>})</span><span>
</span><span>init.states.name</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>init.states.c</span><span>,</span><span> </span><span>init.states.h</span><span>)</span><span>
</span><span>seq.len</span><span> </span><span>&lt;-</span><span> </span><span>1</span><span>
</span><span># set up lstm model
</span><span> </span><span>model</span><span> </span><span>&lt;-</span><span> </span><span>setup.rnn.model</span><span>(</span><span>rnn.sym</span><span>=</span><span>sym</span><span>,</span><span>
</span><span>ctx</span><span>=</span><span>ctx</span><span>,</span><span>
</span><span>num.rnn.layer</span><span>=</span><span>num.lstm.layer</span><span>,</span><span>
</span><span>seq.len</span><span>=</span><span>seq.len</span><span>,</span><span>
</span><span>num.hidden</span><span>=</span><span>num.hidden</span><span>,</span><span>
</span><span>num.embed</span><span>=</span><span>num.embed</span><span>,</span><span>
</span><span>num.label</span><span>=</span><span>num.label</span><span>,</span><span>
</span><span>batch.size</span><span>=</span><span>batch.size</span><span>,</span><span>
</span><span>input.size</span><span>=</span><span>input.size</span><span>,</span><span>
</span><span>init.states.name</span><span>=</span><span>init.states.name</span><span>,</span><span>
</span><span>initializer</span><span>=</span><span>mx.init.uniform</span><span>(</span><span>0.01</span><span>),</span><span>
</span><span>dropout</span><span>=</span><span>dropout</span><span>)</span><span>
</span><span>arg.names</span><span> </span><span>&lt;-</span><span> </span><span>names</span><span>(</span><span>model</span><span>$</span><span>rnn.exec</span><span>$</span><span>ref.arg.arrays</span><span>)</span><span>
</span><span>for</span><span> </span><span>(</span><span>k</span><span> </span><span>in</span><span> </span><span>names</span><span>(</span><span>arg.params</span><span>))</span><span> </span><span>{</span><span>
</span><span>if</span><span> </span><span>((</span><span>k</span><span> </span><span>%in%</span><span> </span><span>arg.names</span><span>)</span><span> </span><span>&amp;&amp;</span><span> </span><span>is.param.name</span><span>(</span><span>k</span><span>)</span><span> </span><span>)</span><span> </span><span>{</span><span>
</span><span>rnn.input</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>()</span><span>
</span><span>rnn.input</span><span>[[</span><span>k</span><span>]]</span><span> </span><span>&lt;-</span><span> </span><span>arg.params</span><span>[[</span><span>k</span><span>]]</span><span>
</span><span>mx.exec.update.arg.arrays</span><span>(</span><span>model</span><span>$</span><span>rnn.exec</span><span>,</span><span> </span><span>rnn.input</span><span>,</span><span> </span><span>match.name</span><span>=</span><span>TRUE</span><span>)</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span><span>init.states</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>()</span><span>
</span><span>for</span><span> </span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>num.lstm.layer</span><span>)</span><span> </span><span>{</span><span>
</span><span>init.states</span><span>[[</span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.init.c&quot;</span><span>)]]</span><span> </span><span>&lt;-</span><span> </span><span>model</span><span>$</span><span>rnn.exec</span><span>$</span><span>ref.arg.arrays</span><span>[[</span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.init.c&quot;</span><span>)]]</span><span>*</span><span>0</span><span>
</span><span>init.states</span><span>[[</span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.init.h&quot;</span><span>)]]</span><span> </span><span>&lt;-</span><span> </span><span>model</span><span>$</span><span>rnn.exec</span><span>$</span><span>ref.arg.arrays</span><span>[[</span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.init.h&quot;</span><span>)]]</span><span>*</span><span>0</span><span>
</span><span>}</span><span>
</span><span>mx.exec.update.arg.arrays</span><span>(</span><span>model</span><span>$</span><span>rnn.exec</span><span>,</span><span> </span><span>init.states</span><span>,</span><span> </span><span>match.name</span><span>=</span><span>TRUE</span><span>)</span><span>
</span><span>return</span><span> </span><span>(</span><span>model</span><span>)</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<h4>RNN Forward APIs</h4>
<p>After getting the inference model with <code>seq.len=1</code>, we use the forward functions (<code>mx.rnn.forward</code>, <code>mx.lstm.forward</code>, <code>mx.gru.forward</code>) to perform prediction in custom RNN / LSTM / GRU inference model. The parameter <code>new.seq</code> denotes that whether there is a new sequence and the previous states should be clear. Given the previous states and input data, we use <code>mx.exec.forward</code> to get the output. After forwarding, the states <code>c</code> and <code>h</code> should be stored to be the previous states for the next input.</p>
<div>
<pre><code data-lang="r"><span>mx.lstm.forward</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>model</span><span>,</span><span> </span><span>input.data</span><span>,</span><span> </span><span>new.seq</span><span>=</span><span>FALSE</span><span>)</span><span> </span><span>{</span><span>
</span><span>if</span><span> </span><span>(</span><span>new.seq</span><span> </span><span>==</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>{</span><span>
</span><span>init.states</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>()</span><span>
</span><span>for</span><span> </span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>model</span><span>$</span><span>num.rnn.layer</span><span>)</span><span> </span><span>{</span><span>
</span><span>init.states</span><span>[[</span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.init.c&quot;</span><span>)]]</span><span> </span><span>&lt;-</span><span> </span><span>model</span><span>$</span><span>rnn.exec</span><span>$</span><span>ref.arg.arrays</span><span>[[</span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.init.c&quot;</span><span>)]]</span><span>*</span><span>0</span><span>
</span><span>init.states</span><span>[[</span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.init.h&quot;</span><span>)]]</span><span> </span><span>&lt;-</span><span> </span><span>model</span><span>$</span><span>rnn.exec</span><span>$</span><span>ref.arg.arrays</span><span>[[</span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.init.h&quot;</span><span>)]]</span><span>*</span><span>0</span><span>
</span><span>}</span><span>
</span><span>mx.exec.update.arg.arrays</span><span>(</span><span>model</span><span>$</span><span>rnn.exec</span><span>,</span><span> </span><span>init.states</span><span>,</span><span> </span><span>match.name</span><span>=</span><span>TRUE</span><span>)</span><span>
</span><span>}</span><span>
</span><span>dim</span><span>(</span><span>input.data</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>model</span><span>$</span><span>batch.size</span><span>)</span><span>
</span><span>data</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>(</span><span>data</span><span>=</span><span>mx.nd.array</span><span>(</span><span>input.data</span><span>))</span><span>
</span><span>mx.exec.update.arg.arrays</span><span>(</span><span>model</span><span>$</span><span>rnn.exec</span><span>,</span><span> </span><span>data</span><span>,</span><span> </span><span>match.name</span><span>=</span><span>TRUE</span><span>)</span><span>
</span><span>mx.exec.forward</span><span>(</span><span>model</span><span>$</span><span>rnn.exec</span><span>,</span><span> </span><span>is.train</span><span>=</span><span>FALSE</span><span>)</span><span>
</span><span>init.states</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>()</span><span>
</span><span>for</span><span> </span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>model</span><span>$</span><span>num.rnn.layer</span><span>)</span><span> </span><span>{</span><span>
</span><span>init.states</span><span>[[</span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.init.c&quot;</span><span>)]]</span><span> </span><span>&lt;-</span><span> </span><span>model</span><span>$</span><span>rnn.exec</span><span>$</span><span>ref.outputs</span><span>[[</span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.last.c_output&quot;</span><span>)]]</span><span>
</span><span>init.states</span><span>[[</span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.init.h&quot;</span><span>)]]</span><span> </span><span>&lt;-</span><span> </span><span>model</span><span>$</span><span>rnn.exec</span><span>$</span><span>ref.outputs</span><span>[[</span><span>paste0</span><span>(</span><span>&quot;l&quot;</span><span>,</span><span> </span><span>i</span><span>,</span><span> </span><span>&quot;.last.h_output&quot;</span><span>)]]</span><span>
</span><span>}</span><span>
</span><span>mx.exec.update.arg.arrays</span><span>(</span><span>model</span><span>$</span><span>rnn.exec</span><span>,</span><span> </span><span>init.states</span><span>,</span><span> </span><span>match.name</span><span>=</span><span>TRUE</span><span>)</span><span>
</span><span>prob</span><span> </span><span>&lt;-</span><span> </span><span>model</span><span>$</span><span>rnn.exec</span><span>$</span><span>ref.outputs</span><span>[[</span><span>&quot;sm_output&quot;</span><span>]]</span><span>
</span><span>return</span><span> </span><span>(</span><span>list</span><span>(</span><span>prob</span><span>=</span><span>prob</span><span>,</span><span> </span><span>model</span><span>=</span><span>model</span><span>))</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<h3>RNN Example</h3>
<p>All in all, we can use the APIs <code>mx.lstm</code>, <code>mx.lstm.inference</code>, <code>mx.lstm.forward</code> to train LSTM models, create a LSTM inference model and use forward function to predict in LSTM inference model. We provide a char-LSTM example. This example aims to show how to use LSTM model to build a char level language model, and generate text from it. We use a tiny shakespeare text for demo purpose. more details can refer to <a target="_blank" href="http://mxnet.readthedocs.io/en/latest/packages/r/CharRnnModel.html">char-RNN</a> documentation.</p>
<h4>Load data</h4>
<p>First of all, we load in the data and preprocess it. We transfer the test into data feature matrix. The dimension of the matrix is <code>seq.len</code> * <code>num.seq</code>, the element of the matrix is the id of the char.</p>
<p>We can get <code>X.train</code> ans <code>X.val</code> which represent the training data and evaluation data.</p>
<div>
<pre><code data-lang>X.train &lt;- list(data=X.train.data, label=X.train.label)
X.val &lt;- list(data=X.val.data, label=X.val.label)
</code></pre>
</div>
<h4>Training Model</h4>
<p>In <code>mxnet</code>, we have a function called <code>mx.lstm</code> so that users can build a general lstm model. </p>
<p>Set basic network parameters.</p>
<div>
<pre><code data-lang="r"><span>batch.size</span><span> </span><span>=</span><span> </span><span>32</span><span>
</span><span>seq.len</span><span> </span><span>=</span><span> </span><span>32</span><span>
</span><span>num.hidden</span><span> </span><span>=</span><span> </span><span>256</span><span>
</span><span>num.embed</span><span> </span><span>=</span><span> </span><span>256</span><span>
</span><span>num.lstm.layer</span><span> </span><span>=</span><span> </span><span>2</span><span>
</span><span>num.round</span><span> </span><span>=</span><span> </span><span>5</span><span>
</span><span>learning.rate</span><span>=</span><span> </span><span>0.1</span><span>
</span><span>wd</span><span>=</span><span>0.00001</span><span>
</span><span>clip_gradient</span><span>=</span><span>1</span><span>
</span><span>update.period</span><span> </span><span>=</span><span> </span><span>1</span><span>
</span></code></pre>
</div>
<p>Training the LSTM model.</p>
<div>
<pre><code data-lang="r"><span>model</span><span> </span><span>&lt;-</span><span> </span><span>mx.lstm</span><span>(</span><span>X.train</span><span>,</span><span> </span><span>X.val</span><span>,</span><span> </span><span>ctx</span><span>=</span><span>mx.cpu</span><span>(),</span><span>
</span><span>num.round</span><span>=</span><span>num.round</span><span>,</span><span> </span><span>update.period</span><span>=</span><span>update.period</span><span>,</span><span>
</span><span>num.lstm.layer</span><span>=</span><span>num.lstm.layer</span><span>,</span><span> </span><span>seq.len</span><span>=</span><span>seq.len</span><span>,</span><span>
</span><span>num.hidden</span><span>=</span><span>num.hidden</span><span>,</span><span> </span><span>num.embed</span><span>=</span><span>num.embed</span><span>,</span><span> </span><span>num.label</span><span>=</span><span>vocab</span><span>,</span><span>
</span><span>batch.size</span><span>=</span><span>batch.size</span><span>,</span><span> </span><span>input.size</span><span>=</span><span>vocab</span><span>,</span><span>
</span><span>initializer</span><span>=</span><span>mx.init.uniform</span><span>(</span><span>0.1</span><span>),</span><span> </span><span>learning.rate</span><span>=</span><span>learning.rate</span><span>,</span><span>
</span><span>wd</span><span>=</span><span>wd</span><span>,</span><span>
</span><span>clip_gradient</span><span>=</span><span>clip_gradient</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code data-lang>## Epoch [31] Train: NLL=3.47213018872144, Perp=32.2052727363657
## ...
## Epoch [961] Train: NLL=2.32060007657895, Perp=10.181782322355
## Iter [1] Train: Time: 161.368175268173 sec, NLL=2.31135356537961, Perp=10.0880702804858
## Iter [1] Val: NLL=1.94184484060012, Perp=6.97160060607419
## Epoch [992] Train: NLL=1.84784553299322, Perp=6.34613225095329
## ...
## Epoch [1953] Train: NLL=1.70175791172558, Perp=5.48357857093351
## Iter [2] Train: Time: 163.582908391953 sec, NLL=1.70103940328978, Perp=5.47963998859367
## Iter [2] Val: NLL=1.74979316010449, Perp=5.75341251767988
## Epoch [1984] Train: NLL=1.59619252884271, Perp=4.9342097524275
## ...
## Epoch [2914] Train: NLL=1.54738185300295, Perp=4.69915099483974
## Iter [3] Train: Time: 163.555737018585 sec, NLL=1.54604189517013, Perp=4.69285854740519
## Iter [3] Val: NLL=1.67780240235925, Perp=5.35377758479576
## Epoch [2945] Train: NLL=1.48868466087876, Perp=4.43126307034767
## ...
## Epoch [3906] Train: NLL=1.474871752958, Perp=4.37047523646576
## Iter [4] Train: Time: 155.97230219841 sec, NLL=1.4744973925858, Perp=4.36883940994296
## Iter [4] Val: NLL=1.64488167325603, Perp=5.18039689118454
## Epoch [3937] Train: NLL=1.46355541021581, Perp=4.32129622881604
## ...
## Epoch [4898] Train: NLL=1.42900458455642, Perp=4.17454171976281
## Iter [5] Train: Time: 151.142980337143 sec, NLL=1.42909226256273, Perp=4.17490775130428
## Iter [5] Val: NLL=1.62716655804022, Perp=5.08943365437187
</code></pre>
</div>
<h4>Predict from the Pretrained Model</h4>
<p>We define the <code>choice</code> function to get random samples. We can use random output or fixed output by choosing largest probability.</p>
<div>
<pre><code data-lang="r"><span>make.output</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>prob</span><span>,</span><span> </span><span>sample</span><span>=</span><span>FALSE</span><span>)</span><span> </span><span>{</span><span>
</span><span>if</span><span> </span><span>(</span><span>!</span><span>sample</span><span>)</span><span> </span><span>{</span><span>
</span><span>idx</span><span> </span><span>&lt;-</span><span> </span><span>which.max</span><span>(</span><span>as.array</span><span>(</span><span>prob</span><span>))</span><span>
</span><span>}</span><span>
</span><span>else</span><span> </span><span>{</span><span>
</span><span>idx</span><span> </span><span>&lt;-</span><span> </span><span>choice</span><span>(</span><span>prob</span><span>)</span><span>
</span><span>}</span><span>
</span><span>return</span><span> </span><span>(</span><span>idx</span><span>)</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<p>In <code>mxnet</code>, we have a function called <code>mx.lstm.inference</code> so that users can build a inference from lstm model and then use function <code>mx.lstm.forward</code> to get forward output from the inference.</p>
<p>Build inference from model.</p>
<div>
<pre><code data-lang="r"><span>infer.model</span><span> </span><span>&lt;-</span><span> </span><span>mx.lstm.inference</span><span>(</span><span>num.lstm.layer</span><span>=</span><span>num.lstm.layer</span><span>,</span><span>
</span><span>input.size</span><span>=</span><span>vocab</span><span>,</span><span>
</span><span>num.hidden</span><span>=</span><span>num.hidden</span><span>,</span><span>
</span><span>num.embed</span><span>=</span><span>num.embed</span><span>,</span><span>
</span><span>num.label</span><span>=</span><span>vocab</span><span>,</span><span>
</span><span>arg.params</span><span>=</span><span>model</span><span>$</span><span>arg.params</span><span>,</span><span>
</span><span>ctx</span><span>=</span><span>mx.cpu</span><span>())</span><span>
</span></code></pre>
</div>
<p>Generate a sequence of 75 chars using function <code>mx.lstm.forward</code>.</p>
<div>
<pre><code data-lang="r"><span>start</span><span> </span><span>&lt;-</span><span> </span><span>'a'</span><span>
</span><span>seq.len</span><span> </span><span>&lt;-</span><span> </span><span>75</span><span>
</span><span>random.sample</span><span> </span><span>&lt;-</span><span> </span><span>TRUE</span><span>
</span><span>last.id</span><span> </span><span>&lt;-</span><span> </span><span>dic</span><span>[[</span><span>start</span><span>]]</span><span>
</span><span>out</span><span> </span><span>&lt;-</span><span> </span><span>&quot;a&quot;</span><span>
</span><span>for</span><span> </span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>(</span><span>1</span><span>:</span><span>(</span><span>seq.len</span><span>-1</span><span>)))</span><span> </span><span>{</span><span>
</span><span>input</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>last.id</span><span>-1</span><span>)</span><span>
</span><span>ret</span><span> </span><span>&lt;-</span><span> </span><span>mx.lstm.forward</span><span>(</span><span>infer.model</span><span>,</span><span> </span><span>input</span><span>,</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>infer.model</span><span> </span><span>&lt;-</span><span> </span><span>ret</span><span>$</span><span>model</span><span>
</span><span>prob</span><span> </span><span>&lt;-</span><span> </span><span>ret</span><span>$</span><span>prob</span><span>
</span><span>last.id</span><span> </span><span>&lt;-</span><span> </span><span>make.output</span><span>(</span><span>prob</span><span>,</span><span> </span><span>random.sample</span><span>)</span><span>
</span><span>out</span><span> </span><span>&lt;-</span><span> </span><span>paste0</span><span>(</span><span>out</span><span>,</span><span> </span><span>lookup.table</span><span>[[</span><span>last.id</span><span>]])</span><span>
</span><span>}</span><span>
</span><span>cat</span><span> </span><span>(</span><span>paste0</span><span>(</span><span>out</span><span>,</span><span> </span><span>&quot;\n&quot;</span><span>))</span><span>
</span></code></pre>
</div>
<p>The result:</p>
<div>
<pre><code data-lang>ah not a drobl greens
Settled asing lately sistering sounted to their hight
</code></pre>
</div>
<h2>Other things new</h2>
<p><a target="_blank" href="https://github.com/dmlc/mxnet/blob/master/R-package/R/optimizer.R">optimizer.R</a> in the R package is used to control the learning rate and update the weights during the training period. At past, only the sgd (Stochastic Gradient Descent) function was implemented. Right now, several other useful optimizer functions are added. The new added optimizers are:</p>
<ul>
<li>RMSProp optimizer</li>
<li>Adam optimizer</li>
<li>AdaGrad optimizer</li>
<li>AdaDelta optimizer</li>
</ul>
<h2>Reference</h2>
<p>[1] Hochreiter, Sepp, and Jürgen Schmidhuber. &quot;Long short-term memory.&quot;Neural computation 9.8 (1997): 1735-1780.</p>
<p>[2] Cho, Kyunghyun, et al. &quot;On the properties of neural machine translation: Encoder-decoder approaches.&quot; arXiv preprint arXiv:1409.1259 (2014).</p>
<p>[3] Graves, Alex. Supervised sequence labelling. Springer Berlin Heidelberg, 2012</p>
<p>[4] Chung, Junyoung, et al. &quot;Empirical evaluation of gated recurrent neural networks on sequence modeling.&quot; arXiv preprint arXiv:1412.3555 (2014).</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/recurrent-models-and-examples-with-mxnetr/"></div></div><div><a data-url="https://www.r-bloggers.com/recurrent-models-and-examples-with-mxnetr/" data-counturl="https://www.r-bloggers.com/recurrent-models-and-examples-with-mxnetr/" data-count="vertical" data-text="Recurrent Models and Examples with MXNetR" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://dmlc.ml/rstats/2016/08/19/recurrent-models-and-examples-with-mxnet-r.html"> DMLC(distributed machine learning common)</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/apWq7R1Bdh0"></a></div>