<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/weighted-linear-support-vector-machine/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://datascienceplus.com/weighted-linear-support-vector-machine/"> DataScience+</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Consider the spam vs ham data from <a target="_blank" rel="nofollow" href="https://archive.ics.uci.edu/ml/datasets/SMS+Spam+Collection"> this site</a>. Let us do some basic analysis on the data with R version 3.3.3, 64 bits on qn windows machine </p>
<pre>
setwd(&quot;your directory&quot;)
sms_data&lt;-read.csv(&quot;sms_spam.csv&quot;,stringsAsFactors = FALSE)
</pre>
<p>Next, check the proportion of spam and ham in your data set</p>
<pre>
prop.table(table(sms_data$type))
<em>ham spam 0.8659849 0.1340151 </em>
</pre>
<p>As you can see, the proportion of ham in the data set is almost 6.6 times higher than spam. Let us select some amount of data from this entire data set to train a model for our classifier.</p>
<pre>
simply_text&lt;-Corpus(VectorSource(sms_data$text))

cleaned_corpus&lt;-tm_map(simply_text, tolower)
cleaned_corpus&lt;-tm_map(cleaned_corpus,removeNumbers)
cleaned_corpus&lt;-tm_map(cleaned_corpus,removeWords,stopwords())

sms_dtm&lt;-DocumentTermMatrix(cleaned_corpus)
sms_train&lt;-cleaned_corpus[1:3000]
sms_test&lt;-cleaned_corpus[3001:5574]

freq_term=(findFreqTerms(sms_dtm,lowfreq=10, highfreq = Inf))

sms_freq_train&lt;-DocumentTermMatrix(sms_train,  list(dictionary=freq_term))
sms_freq_test&lt;-DocumentTermMatrix(sms_test,list(dictionary=freq_term))

print(NCOL(sms_freq_train))
print(NCOL(sms_freq_test))

y_train&lt;-factor(sms_data$type[1:3000])
y_test&lt;-factor(sms_data$type[3001:5574])
prop.table(table(y_train))
print(NCOL(sms_freq_train))
print(NCOL(sms_freq_test))
<em>[1] 856
[1] 856
ham spam 0.8636667 0.1363333 </em>
</pre>
<p>Notice that the proportion of spam and ham in the training data set is similar to that of the entire data. One of the widely used classifiers is Linear Support Vector Machine. From my last writing <a target="_blank" rel="nofollow" href="https://datascienceplus.com/understanding-linear-svm-with-r/"> on Linear Support Vector Machine</a>, you can find that in case of Linear SVM we solve the following optimization problem. </p>
<p>$$Minimize_{w,b}\frac{\vert\vert w\vert\vert^{2}}{2}$$<br>
subject to the constraint<br>
$$y_{i}(w^{T}x_{i}+b)\geq \frac{1}{\vert \vert w\vert\vert}~\forall ~x_{i}~in ~Training~Data~set$$.</p>
<p>Here \(w\) is a weight vector while \(b\) is a scalar also known as bias. For linearly inseparable case, we introduce some penalty \(0\leq \xi_{i} \leq 1\) in the objective function and our constraint.<br>
$$Minimize_{w,b}(\frac{\vert\vert w\vert\vert^{2}}{2}+ C\sum_{i=1}^{n}\xi_{i})$$<br>
subject to the constraints<br>
$$(1)~y_{i}(w^{T}x_{i}+b)\geq 1-\xi_{i}~\forall ~x_{i}~in ~Training~Data~set~of~size~n$$ and<br>
$$(2)~\xi_{i}\geq 0~\forall ~n~points~in~the~Training~Data~set$$ </p>
<p>\(C\) is a user defined parameter which is known as regularization parameter. The regularization parameter is a special parameter. It tells the optimizer what it should minimize more between \(\frac{\vert\vert w\vert\vert^{2}}{2}\) and \(\sum_{i=1}^{n}\xi_{i}\). For example if \(C=0\), then only \(\frac{\vert\vert w\vert\vert^{2}}{2}\) will be minimized whereas if \(C\) is a large value, then \(\sum_{i=1}^{n}\xi_{i}\) will be minimized. </p>
<p>In this example case where the class proportions are so skewed, the choice of the regularization parameter will be a concern. Notice that \(C\) is the scalar-weight of the penalty of mis-classification. Intuitively you can think that, since the proportion of ham is 6.6 times higher than spam samples in the training data, the linear SVM may get biased towards classifying hams more accurately as mis-classifying lots of ham will lead to more aggregated penalty. </p>
<p>Thus, instead of using Linear SVM directly on such data set, it is better to use weighted Linear SVM where instead of using one regularization parameter, we use two separate regularization parameters, \(C_{1}, C_{2}\) where \(C_{1}\) (respectively \(C_{2}\)) is the weight on penalty of mis-classifying a ham sample (respectively a spam sample). So this time our objective function becomes,</p>
<p>$$Minimize_{w,b}(\frac{\vert\vert w\vert\vert^{2}}{2}+ C_{1}\sum_{i=1}^{n_{1}}\xi_{i} +C_{2}\sum_{j=1}^{n_{2}}\xi_{j})$$<br>
where \(n_{1}\) (respective \(n_{2}\)) are the number of hams (respectively spams) in our training data.
</p>
<p>
<h2>
How to Fix the Values for \(C_{1},C_{2}\)<br>
</h2>
<p>One simple way is to set<br>
$$C_{1}=\frac{1}{n_{1}}$$ and<br>
$$C_{2}=\frac{1}{n_{2}}$$ </p>
<p>The R code for doing this is as follows:</p>
<pre>
y&lt;-((sms_data$type))
wts&lt;-1/table(y)
print(wts)
<em> ham spam 0.000207168 0.001338688 </em>
</pre>
<p>Now train your weighted linear SVM. An easy way of doing so is as follows:</p>
<pre>
library(e1071)
sms_freq_matrx&lt;-as.matrix(sms_freq_train)
sms_freq_dtm&lt;-as.data.frame(sms_freq_matrx)
sms_freq_matrx_test&lt;-as.matrix(sms_freq_test)
sms_freq_dtm_test&lt;-as.data.frame(sms_freq_matrx_test)
trained_model&lt;-svm(sms_freq_dtm, y_train, type=&quot;C-classification&quot;, kernel=&quot;linear&quot;, class.weights = wts)
</pre>
<p>Let us use the model for some predictions</p>
<pre>
y_predict&lt;-predict(trained_model, sms_freq_dtm_test)
library(gmodels)
CrossTable(y_predict,y_test,prop.chisq = FALSE)
<em> | y_test y_predict | ham | spam | Row Total | -------------|-----------|-----------|-----------|
ham | 2002 | 233 | 2235 | | 0.896 | 0.104 | 0.868 | | 0.895 | 0.689 | | | 0.778 | 0.091 | | -------------|-----------|-----------|-----------|
spam | 234 | 105 | 339 | | 0.690 | 0.310 | 0.132 | | 0.105 | 0.311 | | | 0.091 | 0.041 | | -------------|-----------|-----------|-----------|
Column Total | 2236 | 338 | 2574 | | 0.869 | 0.131 | | -------------|-----------|-----------|-----------|</em>
</pre>
<p>Of the 2236 hams, 2002 got correctly classified as ham while 234 has got mis-classified as spam. However, the case is not so shiny with spams. Of the 338 spams, 105 got correctly classified as spam while 233 has got mis-classified as ham. </p>
<p>Now let us see what happens without using the weights. </p>
<pre>
trained_model2&lt;-svm(sms_freq_dtm, y_train, type=&quot;C-classification&quot;, kernel=&quot;linear&quot;)
y_predict&lt;-predict(trained_model2, sms_freq_dtm_test)
library(gmodels)
CrossTable(y_predict,y_test,prop.chisq = FALSE)
<em> | y_test y_predict | ham | spam | Row Total | -------------|-----------|-----------|-----------|
ham | 1922 | 224 | 2146 | | 0.896 | 0.104 | 0.834 | | 0.860 | 0.663 | | | 0.747 | 0.087 | | -------------|-----------|-----------|-----------|
spam | 314 | 114 | 428 | | 0.734 | 0.266 | 0.166 | | 0.140 | 0.337 | | | 0.122 | 0.044 | | -------------|-----------|-----------|-----------|
Column Total | 2236 | 338 | 2574 | | 0.869 | 0.131 | | -------------|-----------|-----------|-----------|</em></pre>
<p>Well, the number of spams correctly classified increased but the number of ham correctly classified decreased.<br>
Let us repeat our experiment by increasing the values of \(C_{1}\) and \(C_{2}\). </p>
<pre>
wts&lt;-100/table(y)
print(wts)
<em> ham spam 0.0207168 0.1338688 </em>
</pre>
<p>The results presented below show that the mis-classification for spam has reduced and the accuracy for spam classification has increased. However, the accuracy for ham has decreased.</p>
<pre>
<em> | y_test y_predict | ham | spam | Row Total | -------------|-----------|-----------|-----------|
ham | 1908 | 219 | 2127 | | 0.897 | 0.103 | 0.826 | | 0.853 | 0.648 | | | 0.741 | 0.085 | | -------------|-----------|-----------|-----------|
spam | 328 | 119 | 447 | | 0.734 | 0.266 | 0.174 | | 0.147 | 0.352 | | | 0.127 | 0.046 | | -------------|-----------|-----------|-----------|
Column Total | 2236 | 338 | 2574 | | 0.869 | 0.131 | | -------------|-----------|-----------|-----------|</em>
</pre>
<p>We continued our experiments further by setting </p>
<pre>
wts&lt;-10/table(y)
print(wts)
<em> ham spam 0.00207168 0.01338688 </em>
</pre>
<p>The results are </p>
<pre>
<em> | y_test y_predict | ham | spam | Row Total | -------------|-----------|-----------|-----------|
ham | 1931 | 230 | 2161 | | 0.894 | 0.106 | 0.840 | | 0.864 | 0.680 | | | 0.750 | 0.089 | | -------------|-----------|-----------|-----------|
spam | 305 | 108 | 413 | | 0.738 | 0.262 | 0.160 | | 0.136 | 0.320 | | | 0.118 | 0.042 | | -------------|-----------|-----------|-----------|
Column Total | 2236 | 338 | 2574 | | 0.869 | 0.131 | | -------------|-----------|-----------|-----------|</em>
</pre>
<p>One can repeat the experiment with different values of wts and see the results.</p>
<ul></ul>
<p><strong>
<p>Related Post</p>
<p>
<ol>
<li><a title="Permanent Link toLogistic Regression Regularized with Optimization" target="_blank" rel="nofollow" href="https://datascienceplus.com/logistic-regression-regularized-with-optimization/">Logistic Regression Regularized with Optimization</a></li>
<li><a title="Permanent Link toAnalytical and Numerical Solutions to Linear Regression Problems" target="_blank" rel="nofollow" href="https://datascienceplus.com/analytical-and-numerical-solutions-to-linear-regression-problems/">Analytical and Numerical Solutions to Linear Regression Problems</a></li>
<li><a title="Permanent Link toHow to create a loop to run multiple regression models" target="_blank" rel="nofollow" href="https://datascienceplus.com/how-to-create-a-loop-to-run-multiple-regression-models/">How to create a loop to run multiple regression models</a></li>
<li><a title="Permanent Link toRegression model with auto correlated errors – Part 3, some astrology" target="_blank" rel="nofollow" href="https://datascienceplus.com/regression-model-with-auto-correlated-errors-part-3-some-astrology/">Regression model with auto correlated errors – Part 3, some astrology</a></li>
<li><a title="Permanent Link toRegression model with auto correlated errors – Part 2, the models" target="_blank" rel="nofollow" href="https://datascienceplus.com/regression-model-with-auto-correlated-errors-part-2-the-models/">Regression model with auto correlated errors – Part 2, the models</a></li>
</ol></strong>
<div>
<a target="_blank" rel="nofollow" href="http://feeds.feedburner.com/~ff/datascienceplus?a=Nr--vHO1_gE:qw9OESIuP2Q:D7DqB2pKExk"><img src="http://feeds.feedburner.com/~ff/datascienceplus?i=Nr--vHO1_gE:qw9OESIuP2Q:D7DqB2pKExk" border="0"></a> <a target="_blank" rel="nofollow" href="http://feeds.feedburner.com/~ff/datascienceplus?a=Nr--vHO1_gE:qw9OESIuP2Q:qj6IDK7rITs"><img src="http://feeds.feedburner.com/~ff/datascienceplus?d=qj6IDK7rITs" border="0"></a> <a target="_blank" rel="nofollow" href="http://feeds.feedburner.com/~ff/datascienceplus?a=Nr--vHO1_gE:qw9OESIuP2Q:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/datascienceplus?d=yIl2AUoC8zA" border="0"></a>
</div>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://datascienceplus.com/weighted-linear-support-vector-machine/"> DataScience+</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/TTqvtRkY-Cw"></a></div>