<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/introduction-to-r-for-data-science-session-8-appendix/"></div></div><div><a data-url="http://www.r-bloggers.com/introduction-to-r-for-data-science-session-8-appendix/" data-counturl="http://www.r-bloggers.com/introduction-to-r-for-data-science-session-8-appendix/" data-count="horizontal" data-text="Introduction to R for Data Science :: Session 8 [Appendix]" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.exactness.net/post/146230982005"> The Exactness of Mind</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><br></p>

<h2><b> Appendix to <a target="_blank" href="http://www.exactness.net/post/146228218945/introduction-to-r-for-data-science-session-8">Session 8: Intro to Text Mining in R, ML Estimation + Binomial Logistic Regression</a></b></h2>
<p>Welcome to Introduction to R for Data Science, Session 8: Intro to Text Mining in R, ML Estimation + Binomial Logistic Regression [<i>Web-scraping with tm.plugin.webmining. The tm package corpora structures: assessing document metadata and content. Typical corpus transformations and Term-Document Matrix production. A simple binomial regression model with tf-idf scores as features and its shortcommings due to sparse data. Reminder: Maximum Likelihood Estimation with Nelder-Mead from optim().<i>]</i></i></p>
<p>The course is co-organized by <a target="_blank" href="http://www.datascience.rs/">Data Science Serbia</a> and <a target="_blank" href="http://en.startit.rs/">Startit</a>. You will find all course material (R scripts, data sets, SlideShare presentations, readings) on these pages.</p>
<p>Check out the <a target="_blank" href="http://www.exactness.net/post/143651799365/introduction-to-r-for-data-science">Course Overview</a> to acess the learning material presented thus far.</p>
<p><a target="_blank" href="http://www.datascience.rs/kurs-r-za-data-science/">Data Science Serbia Course Pages [in Serbian]</a></p>
<p><a target="_blank" href="http://startit.rs/uvod-u-r-za-data-science/">Startit Course Pages [in Serbian]</a></p>
<figure data-orig-height="373" data-orig-width="500"><img alt="image" data-orig-height="373" src="http://i1.wp.com/66.media.tumblr.com/836ade3ef73dfc8cfae1df65587ba948/tumblr_inline_o93e37s2sZ1qa0hyw_540.png?resize=456%2C340" data-recalc-dims="1" data-orig-width="500"></figure>
<h2><b>Lecturers</b></h2>
<ul>
<li><a target="_blank" href="https://rs.linkedin.com/in/kovacbranko">dipl. ing Branko Kovač</a>, Data Analyst at <a target="_blank" href="http://cube.rs/">CUBE</a>, Data Science Mentor at <a target="_blank" href="https://www.springboard.com/">Springboard</a>, Data Science Serbia</li>
<li><a target="_blank" href="http://www.exactness.net/post/72461375500/services">Goran S. Milovanović, Phd</a>, <a target="_blank" href="http://www.diplomacy.edu/">DataScientist@DiploFoundation</a>, Data Science Mentor at <a target="_blank" href="https://www.springboard.com/">Springboard</a>, Data Science Serbia</li>
</ul>
<h2><b>Summary of Session 8, 17. June 2016 :: Intro to Text Mining in R + Binomial Logistic Regression.</b></h2>
<p><i>Intro to Text Mining in R + Binomial Logistic Regression. Intro to Text Mining in R + Binomiral Logistic Regression: Web-scraping with tm.plugin.webmining. The tm package corpora structures: assessing document metadata and content. Typical corpus transformations and Term-Document Matrix production. A simple binomial regression model with tf-idf scores as features and its shortcommings due to sparse data. Reminder: Maximum Likelihood Estimation with Nelder-Mead from optim().</i></p>
<p><a target="_blank" href="http://www.exactness.net/post/146230982005#script">Session 8 R Script</a></p>
<p><a target="_blank" href="http://www.exactness.net/post/146230982005#readings">Further Readings</a></p>
<h2><b>R script :: Session 8</b></h2>
<p><b>Split data into training and test</b></p>
<pre>#### w. training vs. test data set
# split into test and training
<a target="_blank" href="http://inside-r.org/r-doc/base/dim">dim</a>(dataSet)
choice &lt;- <a target="_blank" href="http://inside-r.org/r-doc/base/sample">sample</a>(1:475,250,<a target="_blank" href="http://inside-r.org/r-doc/base/replace">replace</a> = F)
test &lt;- <a target="_blank" href="http://inside-r.org/r-doc/base/which">which</a>(!(<a target="_blank" href="http://inside-r.org/r-doc/base/c">c</a>(1:475) %in% choice))
trainData &lt;- dataSet[choice,]
newData &lt;- dataSet[test,]
# check!
<a target="_blank" href="http://inside-r.org/r-doc/base/sum">sum</a>(dataSet$Category[choice])/length(choice) # proportion of dotCom in training
<a target="_blank" href="http://inside-r.org/r-doc/base/sum">sum</a>(dataSet$Category[test])/length(choice) # proportion of dotCom in test
 
# Binomial Logistic Regression: use glm w. logit link
bLRmodel &lt;- <a target="_blank" href="http://inside-r.org/r-doc/stats/glm">glm</a>(Category ~.,
                <a target="_blank" href="http://inside-r.org/r-doc/stats/family">family</a>=<a target="_blank" href="http://inside-r.org/r-doc/stats/binomial">binomial</a>(link='logit'),
                <a target="_blank" href="http://inside-r.org/r-doc/boot/control">control</a> = <a target="_blank" href="http://inside-r.org/r-doc/base/list">list</a>(maxit = 500),
                <a target="_blank" href="http://inside-r.org/r-doc/utils/data">data</a>=trainData)
 
sumLR &lt;- <a target="_blank" href="http://inside-r.org/r-doc/base/summary">summary</a>(bLRmodel)
sumLR
 
# Coefficients
sumLR$coefficients
<a target="_blank" href="http://inside-r.org/r-doc/base/class">class</a>(sumLR$coefficients)
coefLR &lt;- <a target="_blank" href="http://inside-r.org/r-doc/base/as.data.frame">as.data.frame</a>(sumLR$coefficients)
# Wald statistics significant? (this Wald z is normally distributed)
coefLR &lt;- coefLR[<a target="_blank" href="http://inside-r.org/r-doc/base/order">order</a>(-coefLR$Estimate), ]
w &lt;- <a target="_blank" href="http://inside-r.org/r-doc/base/which">which</a>((coefLR$`Pr(>|z|)` &lt; .05)&amp;(!(<a target="_blank" href="http://inside-r.org/r-doc/base/rownames">rownames</a>(coefLR) == &quot;(Intercept)&quot;)))
# which predictors worked?
<a target="_blank" href="http://inside-r.org/r-doc/base/rownames">rownames</a>(coefLR)[w]
 
# plot coefficients {ggplot2}
plotFrame &lt;- coefLR[w,]
plotFrame$Estimate &lt;- <a target="_blank" href="http://inside-r.org/r-doc/base/round">round</a>(plotFrame$Estimate,2)
plotFrame$Features &lt;- <a target="_blank" href="http://inside-r.org/r-doc/base/rownames">rownames</a>(plotFrame)
plotFrame &lt;- plotFrame[<a target="_blank" href="http://inside-r.org/r-doc/base/order">order</a>(-plotFrame$Estimate), ]
plotFrame$Features &lt;- <a target="_blank" href="http://inside-r.org/r-doc/base/factor">factor</a>(plotFrame$Features, <a target="_blank" href="http://inside-r.org/r-doc/base/levels">levels</a> = plotFrame$Features, <a target="_blank" href="http://inside-r.org/r-doc/base/ordered">ordered</a>=T)
<a target="_blank" href="http://inside-r.org/packages/cran/ggplot">ggplot</a>(<a target="_blank" href="http://inside-r.org/r-doc/utils/data">data</a> = plotFrame, aes(x = plotFrame$Features, y = plotFrame$Estimate)) +
  geom_line(<a target="_blank" href="http://inside-r.org/r-doc/grDevices/group">group</a>=1) + geom_point(color=&quot;red&quot;, size=2.5) + geom_point(color=&quot;white&quot;, size=2) +
  xlab(&quot;Features&quot;) + ylab(&quot;Regression Coefficients&quot;) +
  ggtitle(&quot;Logistic Regression: Coeficients (sig. Wald test)&quot;) +
  theme(axis.text.x = element_text(angle=90))
 
# fitted probabilities
<a target="_blank" href="http://inside-r.org/r-doc/stats/fitted">fitted</a>(bLRmodel)
<a target="_blank" href="http://inside-r.org/r-doc/graphics/hist">hist</a>(<a target="_blank" href="http://inside-r.org/r-doc/stats/fitted">fitted</a>(bLRmodel),50)
<a target="_blank" href="http://inside-r.org/r-doc/graphics/plot">plot</a>(<a target="_blank" href="http://inside-r.org/r-doc/stats/density">density</a>(<a target="_blank" href="http://inside-r.org/r-doc/stats/fitted">fitted</a>(bLRmodel)),
     main = &quot;Predicted Probabilities: Density&quot;)
<a target="_blank" href="http://inside-r.org/r-doc/graphics/polygon">polygon</a>(<a target="_blank" href="http://inside-r.org/r-doc/stats/density">density</a>(<a target="_blank" href="http://inside-r.org/r-doc/stats/fitted">fitted</a>(bLRmodel)), 
        <a target="_blank" href="http://inside-r.org/r-doc/base/col">col</a>=&quot;red&quot;, 
        border=&quot;black&quot;)
 
# Prediction from the model
predictions &lt;- <a target="_blank" href="http://inside-r.org/r-doc/stats/predict">predict</a>(bLRmodel,
                       newdata=newData,
                       type='response')
 
predictions &lt;- <a target="_blank" href="http://inside-r.org/r-doc/base/ifelse">ifelse</a>(predictions >= 0.5,1,0)
trueCategory &lt;- newData$Category
 
meanClasError &lt;- <a target="_blank" href="http://inside-r.org/r-doc/base/mean">mean</a>(predictions != trueCategory)
<a target="_blank" href="http://inside-r.org/packages/cran/accuracy">accuracy</a> &lt;- 1-meanClasError
<a target="_blank" href="http://inside-r.org/packages/cran/accuracy">accuracy</a> # probably rather poor..? - Why? - Think!
 
# Try to train a binomial regression model many times by randomly assigning 
# documents to the training and test data set
# What happens? Why?
 
# *Look* at your data set and *think* about it before actually modeling it.</pre>
<p><a title="Created by Pretty R at inside-R.org" target="_blank" href="http://www.inside-r.org/pretty-r">Created by Pretty R at inside-R.org</a></p>
<figure data-orig-height="373" data-orig-width="500"><img alt="image" src="http://i1.wp.com/66.media.tumblr.com/20c7f6054ecf358788f2c71fc8e8f3be/tumblr_inline_o93e38EZnU1qa0hyw_540.png?resize=456%2C340" data-orig-height="373" data-recalc-dims="1" data-orig-width="500"></figure>
<figure data-orig-height="373" data-orig-width="500"><img alt="image" data-orig-height="373" src="http://i1.wp.com/66.media.tumblr.com/836ade3ef73dfc8cfae1df65587ba948/tumblr_inline_o93e37s2sZ1qa0hyw_540.png?resize=456%2C340" data-recalc-dims="1" data-orig-width="500"></figure>
<h2><b>Readings :: Session 9: Binomial and Multinomial Logistic Regression [23. June, 2016, @Startit.rs, 19h CET]</b></h2>
<ul>
<li>Yves Croissant, <a target="_blank" href="https://cran.r-project.org/web/packages/mlogit/vignettes/mlogit.pdf"> Estimation of multinomial logit models in R: The mlogit Packages</a>
</li>
</ul>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/introduction-to-r-for-data-science-session-8-appendix/"></div></div><div><a data-url="http://www.r-bloggers.com/introduction-to-r-for-data-science-session-8-appendix/" data-counturl="http://www.r-bloggers.com/introduction-to-r-for-data-science-session-8-appendix/" data-count="vertical" data-text="Introduction to R for Data Science :: Session 8 [Appendix]" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.exactness.net/post/146230982005"> The Exactness of Mind</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/xCmYwSr1cuU"></a></div>