<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/imdb-genre-classification-using-deep-learning/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://flovv.github.io/Text_classification/"> Florian Teschner</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>The Internet Movie Database (Imdb) is a great source to get information about movies. Keras provides access to some part of the cleaned dataset (e.g. for sentiment classification). While sentiment classification is an interesting topic, I wanted to see if it is possible to identify a movie’s genre from its description.<br>
The image illustrates the task;</p>
<p><img alt="plot of chunk unnamed-chunk-4" src="https://i2.wp.com/flovv.github.io/figures/post25/imdb_classification.png?w=456" data-recalc-dims="1"></p>
<p>To see if that is possible I downloaded the raw data from an <a rel="nofollow" target="_blank">FU-Berlin ftp- server</a>. Most movies have multiple genres assigned (e.g. Action and Sci-fi.). I chose to randomly pick one genre in case of multiple assignments.</p>
<p>So the task at hand is to use a lengthy description to interfere a (noisy) label. Hence, the task is similar to the <a rel="nofollow" href="https://rstudio.github.io/keras/articles/examples/reuters_mlp.html" target="_blank">Reuters news categorization task</a>. I used the code as a guideline for the model.<br>
However, looking at the code, it becomes clear that data preprocessing part is skipped. In order to make it easy for a practitioner to create their own applications, I will try to detail the necessary preprocessing.<br>
The texts are represented as a vector of integers (indexes). So basically one builds a dictionary in which each index refers to a particular word.</p>
<figure>
<pre><code data-lang="r"><span>require</span><span>(</span><span>caret</span><span>)</span><span>
</span><span>require</span><span>(</span><span>keras</span><span>)</span><span>
</span><span>max_words</span><span> </span><span>&lt;-</span><span> </span><span>1500</span><span>
</span><span>### create a balanced dataset with equal numbers of observations for each class
</span><span>down_train</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>downSample</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>mm</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>mm</span><span>$</span><span>GenreFact</span><span>)</span><span>
</span><span>### preprocessing --- </span><span>tokenizer</span><span> </span><span>=</span><span> </span><span>keras</span><span>::</span><span>text_tokenizer</span><span>(</span><span>num_words</span><span> </span><span>=</span><span> </span><span>max_words</span><span>)</span><span>
</span><span>keras</span><span>::</span><span>fit_text_tokenizer</span><span>(</span><span>tokenizer</span><span>,</span><span> </span><span>mm</span><span>$</span><span>descr</span><span>)</span><span>
</span><span>sequences</span><span> </span><span>=</span><span> </span><span>tokenizer</span><span>$</span><span>texts_to_sequences</span><span>(</span><span>mm</span><span>$</span><span>descr</span><span>)</span><span>
</span><span>## split in training and test set.
</span><span>train</span><span> </span><span>&lt;-</span><span> </span><span>sample</span><span>(</span><span>1</span><span>:</span><span>length</span><span>(</span><span>sequences</span><span>),</span><span>size</span><span> </span><span>=</span><span> </span><span>0.95</span><span>*</span><span>length</span><span>(</span><span>sequences</span><span>),</span><span> </span><span>replace</span><span>=</span><span>F</span><span>)</span><span>
</span><span>x_test</span><span> </span><span>&lt;-</span><span> </span><span>sequences</span><span>[</span><span>-</span><span>train</span><span>]</span><span>
</span><span>x_train</span><span> </span><span>&lt;-</span><span> </span><span>sequences</span><span>[</span><span>train</span><span>]</span><span>
</span><span>### labels!
</span><span>y_train</span><span> </span><span>&lt;-</span><span> </span><span>mm</span><span>[</span><span>train</span><span>,]</span><span>$</span><span>GenreFact</span><span> </span><span>y_test</span><span> </span><span>&lt;-</span><span> </span><span>mm</span><span>[</span><span>-</span><span>train</span><span>,]</span><span>$</span><span>GenreFact</span><span>
</span><span>########## how many classes do we have?
</span><span>num_classes</span><span> </span><span>&lt;-</span><span> </span><span>length</span><span>(</span><span>unique</span><span>(</span><span>y_train</span><span>))</span><span> </span><span>+1</span><span>
</span><span>cat</span><span>(</span><span>num_classes</span><span>,</span><span> </span><span>'\n'</span><span>)</span><span>
</span><span>#'Vectorizing sequence data to a matrix which can be used an input matrix
</span><span>x_train</span><span> </span><span>&lt;-</span><span> </span><span>sequences_to_matrix</span><span>(</span><span>tokenizer</span><span>,</span><span> </span><span>x_train</span><span>,</span><span> </span><span>mode</span><span> </span><span>=</span><span> </span><span>'binary'</span><span>)</span><span>
</span><span>x_test</span><span> </span><span>&lt;-</span><span> </span><span>sequences_to_matrix</span><span>(</span><span>tokenizer</span><span>,</span><span> </span><span>x_test</span><span>,</span><span> </span><span>mode</span><span> </span><span>=</span><span> </span><span>'binary'</span><span>)</span><span>
</span><span>cat</span><span>(</span><span>'x_train shape:'</span><span>,</span><span> </span><span>dim</span><span>(</span><span>x_train</span><span>),</span><span> </span><span>'\n'</span><span>)</span><span>
</span><span>cat</span><span>(</span><span>'x_test shape:'</span><span>,</span><span> </span><span>dim</span><span>(</span><span>x_test</span><span>),</span><span> </span><span>'\n'</span><span>)</span><span>
</span><span>#'Convert class vector to binary class matrix',
</span><span># '(for use with categorical_crossentropy)\n')
</span><span>y_train</span><span> </span><span>&lt;-</span><span> </span><span>to_categorical</span><span>(</span><span>y_train</span><span>,</span><span> </span><span>num_classes</span><span>)</span><span>
</span><span>y_test</span><span> </span><span>&lt;-</span><span> </span><span>to_categorical</span><span>(</span><span>y_test</span><span>,</span><span> </span><span>num_classes</span><span>)</span></code></pre>
</figure>
<p>In order to get a trainable data, we first balance the dataset such that all classes have the same frequency.<br>
Then we preprocess the raw text descriptions in such an index based representation. As always, we split the dataset in test and training data (90%). Finally, we transform the index based representation into a matrix representation and hot-one-encode the classes.</p>
<p>After setting up the data, we can define the model. I tried different combinations (depth, dropouts, regularizers and input units) and the following layout seems to work the best:</p>
<figure>
<pre><code data-lang="r"><span>batch_size</span><span> </span><span>&lt;-</span><span> </span><span>64</span><span>
</span><span>epochs</span><span> </span><span>&lt;-</span><span> </span><span>200</span><span>
</span><span>model</span><span> </span><span>&lt;-</span><span> </span><span>keras_model_sequential</span><span>()</span><span>
</span><span>model</span><span> </span><span>%>%</span><span>
</span><span>layer_dense</span><span>(</span><span>units</span><span> </span><span>=</span><span> </span><span>512</span><span>,</span><span> </span><span>input_shape</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>max_words</span><span>),</span><span> </span><span>activation</span><span>=</span><span>&quot;relu&quot;</span><span>)</span><span> </span><span>%>%</span><span> </span><span>layer_dropout</span><span>(</span><span>rate</span><span> </span><span>=</span><span> </span><span>0.6</span><span>)</span><span> </span><span>%>%</span><span> </span><span>layer_dense</span><span>(</span><span>units</span><span>=</span><span>64</span><span>,</span><span> </span><span>activation</span><span> </span><span>=</span><span> </span><span>'relu'</span><span>,</span><span> </span><span>regularizer_l1</span><span>(</span><span>l</span><span>=</span><span>0.15</span><span>))</span><span> </span><span>%>%</span><span> </span><span>layer_dropout</span><span>(</span><span>rate</span><span> </span><span>=</span><span> </span><span>0.8</span><span>)</span><span> </span><span>%>%</span><span> </span><span>layer_dense</span><span>(</span><span>units</span><span>=</span><span>num_classes</span><span>,</span><span> </span><span>activation</span><span> </span><span>=</span><span> </span><span>'softmax'</span><span>)</span><span>
</span><span>summary</span><span>(</span><span>model</span><span>)</span><span>
</span><span>model</span><span> </span><span>%>%</span><span> </span><span>compile</span><span>(</span><span>
</span><span>loss</span><span> </span><span>=</span><span> </span><span>'categorical_crossentropy'</span><span>,</span><span>
</span><span>optimizer</span><span> </span><span>=</span><span> </span><span>'adam'</span><span>,</span><span>
</span><span>metrics</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>'accuracy'</span><span>)</span><span>
</span><span>)</span><span>
</span><span>hist</span><span> </span><span>&lt;-</span><span> </span><span>model</span><span> </span><span>%>%</span><span> </span><span>fit</span><span>(</span><span>
</span><span>x_train</span><span>,</span><span> </span><span>y_train</span><span>,</span><span>
</span><span>batch_size</span><span> </span><span>=</span><span> </span><span>batch_size</span><span>,</span><span>
</span><span>epochs</span><span> </span><span>=</span><span> </span><span>200</span><span>,</span><span>
</span><span>verbose</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span>
</span><span>validation_split</span><span> </span><span>=</span><span> </span><span>0.1</span><span>
</span><span>)</span><span>
</span><span>## using the holdout dataset!
</span><span>score</span><span> </span><span>&lt;-</span><span> </span><span>model</span><span> </span><span>%>%</span><span> </span><span>evaluate</span><span>(</span><span>
</span><span>x_test</span><span>,</span><span> </span><span>y_test</span><span>,</span><span>
</span><span>batch_size</span><span> </span><span>=</span><span> </span><span>batch_size</span><span>,</span><span>
</span><span>verbose</span><span> </span><span>=</span><span> </span><span>1</span><span>
</span><span>)</span><span>
</span><span>cat</span><span>(</span><span>'Test score:'</span><span>,</span><span> </span><span>score</span><span>[[</span><span>1</span><span>]],</span><span> </span><span>'\n'</span><span>)</span><span>
</span><span>cat</span><span>(</span><span>'Test accuracy'</span><span>,</span><span> </span><span>score</span><span>[[</span><span>2</span><span>]],</span><span> </span><span>'\n'</span><span>)</span></code></pre>
</figure>
<p>Finally, we plot the training progress and conclude that it is possible to train a classifier without too much effort.</p>
<p><img alt="plot of chunk unnamed-chunk-4" src="https://i2.wp.com/flovv.github.io/figures/post25/unnamed-chunk-4-1.png?w=456" data-recalc-dims="1"></p>
<p>I hope the short tutorial illustrated how to preprocess text in order to build a text-based deep-learning learning classifier. I am pretty sure that are better parameters to tune the model.<br>
If you want to implement such a model in production environment, I would recommend playing with the text-preprocessing parameters. The text-tokenizer and the text_to_sequence functions hold a lot of untapped value.</p>
<p>Good luck!</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://flovv.github.io/Text_classification/"> Florian Teschner</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/3_34tiSB1wY" height="1"></a></div>