<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/multilabel-classification-with-neuralnet-package/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.quantide.com/multilabel-classification-neuralnet-package/"> R blog | Quantide - R training &amp; consulting</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Some time ago I wrote an article on how to use a simple <a target="_blank" rel="nofollow" href="https://www.r-bloggers.com/fitting-a-neural-network-in-r-neuralnet-package/">neural network</a> in R with the <strong>neuralnet</strong> package to tackle a regression task. Since then, however, I turned my attention to other libraries such as <a target="_blank" rel="nofollow" href="https://www.r-bloggers.com/image-recognition-tutorial-in-r-using-deep-convolutional-neural-networks-mxnet-package/">MXNet</a>, mainly because I wanted something more than what the <strong>neuralnet </strong>package provides (for starters, convolutional neural networks and, why not, recurrent neural networks).</p>
<p>A few weeks ago, however, I was asked how to use the <strong>neuralnet</strong> package for making a <strong>multilabel classifier</strong>. I wrote a quick script as an example and thought I could write a short article on it, furthermore I think a classification tutorial using the <strong>neuralnet</strong> package could be complementary to the one I did on regression.</p>
<p>The <strong>neuralnet</strong> package is perhaps not the best option in R for using neural networks. If you ask why, for starters it does not recognize the typical formula <strong>y~.</strong>, it does not support factors, it does not provide a lot of models other than a standard MLP, and it has great competitors in the <strong>nnet</strong> package that seems to be better integrated in R and can be used with the <strong>caret</strong> package, and in the <strong>MXnet</strong> package that is a high level deep learning library which provides a wide variety of neural networks.</p>
<p>But still, I think there is some value in the ease of use of the <strong>neuralnet</strong> package, especially for a beginner, therefore I’ll be using it.</p>
<p>I’m going to be using both the <strong>neuralnet</strong> and, curiously enough, the <strong>nnet</strong> package. Let’s load them:</p>
<pre># load libs
require(neuralnet)
require(nnet)
require(ggplot2)
set.seed(10)</pre>
<p>
The need for the nnet package will be more clear later on, I promise <img alt="😉" class="wp-smiley" src="https://i2.wp.com/s.w.org/images/core/emoji/2/72x72/1f609.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<h2>The dataset</h2>
<p>I looked in the <a target="_blank" rel="nofollow" href="https://archive.ics.uci.edu/">UCI Machine Learning Repository</a><sup>1</sup> and found the <a target="_blank" rel="nofollow" href="https://archive.ics.uci.edu/ml/datasets/Wine">wine dataset</a>.</p>
<p>This dataset contains the results of a chemical analysis on 3 different kind of wines. The target variable is the label of the wine which is a <strong>factor</strong> with 3 (unordered) <strong>levels</strong>. The predictors are all continuous and represent 13 variables obtained as a result of chemical measurements.</p>
<p>This dataset seems not that hard to figure out, even the description says it is not quite challenging, but I think it is fine for practicing classification. Furthermore it is a little less boring than the <strong>iris</strong> dataset <img alt="😉" class="wp-smiley" src="https://i2.wp.com/s.w.org/images/core/emoji/2/72x72/1f609.png?w=456&ssl=1" data-recalc-dims="1"></p>
</div>
<div>
<h2>Loading the data</h2>
<p>Simply download the dataset from the link above and save it as a <strong>.csv</strong> file in your working directory. Then you can load it in R as follows:</p>
<pre># Load data and set variables names
wines &lt;- read.csv(&quot;wines.csv&quot;)
names(wines) &lt;- c(&quot;label&quot;,
                  &quot;Alcohol&quot;,
                  &quot;Malic_acid&quot;,
                  &quot;Ash&quot;,
                  &quot;Alcalinity_of_ash&quot;,
                  &quot;Magnesium&quot;,
                  &quot;Total_phenols&quot;,
                  &quot;Flavanoids&quot;,
                  &quot;Nonflavanoid_phenols&quot;,
                  &quot;Proanthocyanins&quot;,
                  &quot;Color_intensity&quot;,
                  &quot;Hue&quot;,
                  &quot;OD280_OD315_of_diluted_wines&quot;,
                  &quot;Proline&quot;)
head(wines)</pre>

<pre>##   label Alcohol Malic_acid  Ash Alcalinity_of_ash Magnesium Total_phenols
## 1     1   13.20       1.78 2.14              11.2       100          2.65
## 2     1   13.16       2.36 2.67              18.6       101          2.80
## 3     1   14.37       1.95 2.50              16.8       113          3.85
## 4     1   13.24       2.59 2.87              21.0       118          2.80
## 5     1   14.20       1.76 2.45              15.2       112          3.27
## 6     1   14.39       1.87 2.45              14.6        96          2.50
##   Flavanoids Nonflavanoid_phenols Proanthocyanins Color_intensity  Hue
## 1       2.76                 0.26            1.28            4.38 1.05
## 2       3.24                 0.30            2.81            5.68 1.03
## 3       3.49                 0.24            2.18            7.80 0.86
## 4       2.69                 0.39            1.82            4.32 1.04
## 5       3.39                 0.34            1.97            6.75 1.05
## 6       2.52                 0.30            1.98            5.25 1.02
##   OD280_OD315_of_diluted_wines Proline
## 1                         3.40    1050
## 2                         3.17    1185
## 3                         3.45    1480
## 4                         2.93     735
## 5                         2.85    1450
## 6                         3.58    1290</pre>
<p>
Let’s have a look at a plot of some of the variables:</p>
<pre>plt1 &lt;- ggplot(wines, aes(x = Alcohol, y = Magnesium, colour = as.factor(label))) +
    geom_point(size=3) +
    ggtitle(&quot;Wines&quot;)
plt2 &lt;- ggplot(wines, aes(x = Alcohol, y = Proline, colour = as.factor(label))) +
    geom_point(size=3) +
    ggtitle(&quot;Wines&quot;)
plt1</pre>
</div>
<div>
<p><img sizes="(max-width: 350px) 100vw, 350px" alt="immagine-1" class="wp-image-3327" src="https://i0.wp.com/www.quantide.com/wp-content/uploads/2016/11/Immagine-1-1-300x214.png?resize=350%2C250" data-recalc-dims="1"></p>
<pre>plt2</pre>
<p>
<img sizes="(max-width: 350px) 100vw, 350px" alt="immagine-2" class="wp-image-3329" src="https://i0.wp.com/www.quantide.com/wp-content/uploads/2016/11/Immagine-2-300x214.png?resize=350%2C250" data-recalc-dims="1"></p>
<p>Out of the blue I decided <a target="_blank" rel="nofollow" href="http://www.quantide.com/performing-principal-components-regression-pcr-in-r/">to run PCA</a> on the normalized dataset and found an interesting results worth displaying:</p>
<p><img sizes="(max-width: 350px) 100vw, 350px" alt="multilabel-neuralnet-3" class="wp-image-3330" src="https://i0.wp.com/www.quantide.com/wp-content/uploads/2016/11/Immagine-3-300x214.png?resize=350%2C250" data-recalc-dims="1"></p>
<div>
<p>Although the first two components amount to about 50% of the total variance, it looks like it could be interesting to fit some classifier using only those two components. But let’s not get distracted by these beautiful plots, let’s go on to the next step.</p>
</div>
<div>
<div>
<h2>Preprocessing</h2>
<p>During the preprocessing phase, I have to do at least the following two things:</p>
<ul>
<li>Encode the categorical variables.</li>
<li>Standardize the predictors.</li>
</ul>
<p>First of all, let’s encode our target variable. The encoding of the categorical variables is needed when using <strong>neuralnet</strong> since it does not like factors at all. It will shout at you if you try to feed in a factor (I am told <strong>nnet</strong> likes factors though).</p>
<p>In the wine dataset the variable <strong>label</strong> contains three different labels: 1,2 and 3.</p>
<p>The usual practice, as far as I know, is to encode categorical variables as a <strong>“one hot” vector</strong>. For instance, if I had three classes, like in this case, I’d need to replace the label variable with three variables like these:</p>
<pre>#   l1,l2,l3
#   1,0,0
#   0,0,1
#   ...</pre>
<p>
In this case the first observation would be labelled as a 1, the second would be labelled as a 2, and so on. Ironically, the <strong>nnet</strong> package provides a function to perform this encoding in a painless way:</p>
<pre># Encode as a one hot vector multilabel data
train &lt;- cbind(wines[, 2:14], class.ind(as.factor(wines$label)))
# Set labels name
names(train) &lt;- c(names(wines)[2:14],&quot;l1&quot;,&quot;l2&quot;,&quot;l3&quot;)</pre>
<p>
see? Very painless and fast! <img alt="😉" class="wp-smiley" src="https://i2.wp.com/s.w.org/images/core/emoji/2/72x72/1f609.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>By the way, since the predictors are all continuous, you do not need to encode any of them, however, in case you needed to, you could apply the same strategy applied above to all the categorical predictors. Unless of course you’d like to try some other kind of custom encoding.</p>
<p>Now let’s standardize the predictors in the <span><span data-mathml="&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;mo stretchy=&quot;false&quot;&gt;[&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;mo&gt;&amp;#x2212;&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy=&quot;false&quot;&gt;]&lt;/mo&gt;&lt;/math&gt;" tabindex="0"><span><span><span>[</span><span>0</span><span>−</span><span>1</span><span>] </span></span></span></span></span> interval by leveraging the <strong>lapply</strong> function:</p>
<pre># Scale data
scl &lt;- function(x){ (x - min(x))/(max(x) - min(x)) }
train[, 1:13] &lt;- data.frame(lapply(train[, 1:13], scl))
head(train)</pre>

<pre>##     Alcohol Malic_acid       Ash Alcalinity_of_ash Magnesium Total_phenols
## 1 0.5710526  0.2055336 0.4171123        0.03092784 0.3260870     0.5758621
## 2 0.5605263  0.3201581 0.7005348        0.41237113 0.3369565     0.6275862
## 3 0.8789474  0.2391304 0.6096257        0.31958763 0.4673913     0.9896552
## 4 0.5815789  0.3656126 0.8074866        0.53608247 0.5217391     0.6275862
## 5 0.8342105  0.2015810 0.5828877        0.23711340 0.4565217     0.7896552
## 6 0.8842105  0.2233202 0.5828877        0.20618557 0.2826087     0.5241379
##   Flavanoids Nonflavanoid_phenols Proanthocyanins Color_intensity
## 1  0.5105485            0.2452830       0.2744479       0.2645051
## 2  0.6118143            0.3207547       0.7570978       0.3754266
## 3  0.6645570            0.2075472       0.5583596       0.5563140
## 4  0.4957806            0.4905660       0.4447950       0.2593857
## 5  0.6434599            0.3962264       0.4921136       0.4667235
## 6  0.4599156            0.3207547       0.4952681       0.3387372
##         Hue OD280_OD315_of_diluted_wines   Proline l1 l2 l3
## 1 0.4634146                    0.7802198 0.5506419  1  0  0
## 2 0.4471545                    0.6959707 0.6469330  1  0  0
## 3 0.3089431                    0.7985348 0.8573466  1  0  0
## 4 0.4552846                    0.6080586 0.3259629  1  0  0
## 5 0.4634146                    0.5787546 0.8359486  1  0  0
## 6 0.4390244                    0.8461538 0.7218260  1  0  0</pre>
</div>
<div>
<h2>Fitting the model with neuralnet</h2>
<p>Now it is finally time to fit the model.</p>
<p>As you might remember from the old post I wrote, neuralnet does not like the formula <strong>y~.</strong>. Fear not, you can build the formula to be used in a simple step:</p>
<pre># Set up formula
n &lt;- names(train)
f &lt;- as.formula(paste(&quot;l1 + l2 + l3 ~&quot;, paste(n[!n %in% c(&quot;l1&quot;,&quot;l2&quot;,&quot;l3&quot;)], collapse = &quot; + &quot;)))
f</pre>

<pre>## l1 + l2 + l3 ~ Alcohol + Malic_acid + Ash + Alcalinity_of_ash + 
##     Magnesium + Total_phenols + Flavanoids + Nonflavanoid_phenols + 
##     Proanthocyanins + Color_intensity + Hue + OD280_OD315_of_diluted_wines + 
##     Proline</pre>
<p>
Note that the characters in the vector are not pasted to the right of the “~” symbol.</p>
<p>Just remember to check that the formula is indeed correct and then you are good to go.</p>
<p>Let’s train the neural network with the full dataset. It should take very little time to converge. If you did not standardize the predictors it could take a lot more though.</p>
<pre>nn &lt;- neuralnet(f,
                data = train,
                hidden = c(13, 10, 3),
                act.fct = &quot;logistic&quot;,
                linear.output = FALSE,
                lifesign = &quot;minimal&quot;)</pre>

<pre>## hidden: 13, 10, 3    thresh: 0.01    rep: 1/1    steps:      88  error: 0.03039  time: 0.1 secs</pre>
<p>
Note that I set the argument <strong>linear.output</strong> to <strong>FALSE</strong> in order to tell the model that I want to apply the activation function <strong>act.fct</strong> and that I am not doing a regression task. Then I set the activation function to <strong>logistic</strong> (which by the way is the default option) in order to apply the logistic function. The other available option is <strong>tanh</strong> but the model seems to perform a little worse with it so I opted for the default option. As far as I know these two are the only two available options, there is no “relu” function available although it seems to be a common activation function in other packages.</p>
<p>As far as the number of <strong>hidden neurons</strong>, I tried some combination and the one used seems to perform slightly better than the others (around 1% of accuracy difference in cross validation score).</p>
<p>By using the in-built plot method you can get a visual take on what is actually happening inside the model, however the plot is not that helpful I think</p>
<pre>plot(nn)</pre>
<p>
Let’s have a look at the accuracy on the training set:</p>
<pre># Compute predictions
pr.nn &lt;- compute(nn, train[, 1:13])

# Extract results
pr.nn_ &lt;- pr.nn$net.result
head(pr.nn_)</pre>

<pre>##              [,1]           [,2]              [,3]
## [1,] 0.9897528761 0.003171322443 0.000006987838514
## [2,] 0.9908394248 0.002331321781 0.000008693900073
## [3,] 0.9914977585 0.002103254765 0.000008649814003
## [4,] 0.9855622778 0.004418327885 0.000008738518880
## [5,] 0.9916175055 0.002119520153 0.000008319926342
## [6,] 0.9915542288 0.002144844815 0.000008337763696</pre>

<pre># Accuracy (training set)
original_values &lt;- max.col(train[, 14:16])
pr.nn_2 &lt;- max.col(pr.nn_)
mean(pr.nn_2 == original_values)</pre>

<pre>## [1] 1</pre>
<p>
100% not bad! But wait, this may be because our model over fitted the data, furthermore evaluating accuracy on the training set is kind of cheating since the model already “knows” (or should know) the answers. In order to assess the “true accuracy” of the model you need to perform some kind of cross validation.</p>
</div>
<div>
<h2>Cross validating the classifier</h2>
<p>Let’s <strong>crossvalidate</strong> the model using the evergreen <strong>10 fold cross validation</strong> with the following train and test split: 95% of the dataset will be used as training set while the remaining 5% as test set.</p>
<p>Just out of curiosity I decided to run a <strong>LOOCV</strong> round too. In case you’d like to run this cross validation technique, just set the proportion variable to 0.995: this will select just one observation for as test set and leave all the other observations as training set. Running LOOCV you should get similar results to the 10 fold cross validation.</p>
<pre># Set seed for reproducibility purposes
set.seed(500)
# 10 fold cross validation
k &lt;- 10
# Results from cv
outs &lt;- NULL
# Train test split proportions
proportion &lt;- 0.95 # Set to 0.995 for LOOCV

# Crossvalidate, go!
for(i in 1:k)
{
    index &lt;- sample(1:nrow(train), round(proportion*nrow(train)))
    train_cv &lt;- train[index, ]
    test_cv &lt;- train[-index, ]
    nn_cv &lt;- neuralnet(f,
                        data = train_cv,
                        hidden = c(13, 10, 3),
                        act.fct = &quot;logistic&quot;,
                        linear.output = FALSE)
    
    # Compute predictions
    pr.nn &lt;- compute(nn_cv, test_cv[, 1:13])
    # Extract results
    pr.nn_ &lt;- pr.nn$net.result
    # Accuracy (test set)
    original_values &lt;- max.col(test_cv[, 14:16])
    pr.nn_2 &lt;- max.col(pr.nn_)
    outs[i] &lt;- mean(pr.nn_2 == original_values)
}

mean(outs)</pre>

<pre>## [1] 0.9888888889</pre>
<p>
98.8%, awesome! Next time when you are invited to a relaxing evening that includes a wine tasting competition I think you should definitely bring your laptop as a contestant!</p>
<p>Aside from that poor taste joke, (I made it again!), indeed this dataset is not the most challenging, I think with some more tweaking a better cross validation score could be achieved. Nevertheless I hope you found this tutorial useful. A gist with the entire code for this tutorial can be found <a target="_blank" rel="nofollow" href="https://gist.github.com/mick001/5973654a443a79b5d1b911a22c00e487">here</a>.</p>
<p>Thank you for reading this article, please feel free to leave a comment if you have any questions or suggestions and share the post with others if you find it useful.</p>
</div>
<div>
<p>Notes:</p>
<p><sup>1</sup> Lichman, M. (2013). UCI Machine Learning Repository [<a target="_blank" rel="nofollow" href="http://archive.ics.uci.edu/ml">http://archive.ics.uci.edu/ml</a>]. Irvine, CA: University of California, School of Information and Computer Science.</p>
</div>
</div>
</div>
<p>The post <a target="_blank" rel="nofollow" href="http://www.quantide.com/multilabel-classification-neuralnet-package/">Multilabel classification with neuralnet package</a> appeared first on <a target="_blank" rel="nofollow" href="http://www.quantide.com/">Quantide – R training &amp; consulting</a>.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.quantide.com/multilabel-classification-neuralnet-package/"> R blog | Quantide - R training &amp; consulting</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/HQvgSgwbKDI"></a></div>