<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/linear-logistic-regression-in-r-dealing-with-unknown-factor-levels-in-test-data/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://feedproxy.google.com/~r/RDiscoveringPythonR/~3/CfUhRar0uRE/"> R – Discovering Python &amp; R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Let’s say you have data containing a categorical variable with 50 <em>levels</em>. When you divide the data into train and test sets, chances are you don’t have all 50 levels featuring in your training set.</p>
<p>This often happens when you divide the data set into train and test sets according to the distribution of the outcome variable. In doing so, chances are that our explanatory categorical variable might not be distributed exactly the same way in train and test sets – so much so that certain levels of this categorical variable are missing from the training set. The more levels there are to a categorical variable, it gets difficult for that variable to be similarly represented upon splitting the data.</p>
<p>Take for instance this example data set (<a rel="nofollow" href="https://github.com/anirudhjayaraman/shiny-fiesta/blob/master/train.csv" target="_blank">train.csv</a> + <a rel="nofollow" href="https://github.com/anirudhjayaraman/shiny-fiesta/blob/master/test.csv" target="_blank">test.csv</a>) which contains a categorical variable <code>var_b</code> that takes 349 unique levels. Our train data has 334 of these levels – on which the model is built – and hence 15 levels are excluded from our trained model. If you try making predictions on the test set with this model in R, it throws an error:<br>
<code>factor var_b has new levels 16060, 17300, 17980, 19060, 21420, 21820,<br>
25220, 29340, 30300, 33260, 34100, 38340, 39660, 44300, 45460<br>
</code><br>
If you’ve used R to model generalized linear class of models such as <em>linear</em>, <em>logit</em> or <em>probit</em> models, then chances are you’ve come across this problem – especially when you’re validating your trained model on test data.</p>
<p>The workaround to this problem is in the form of a function, <code>remove_missing_levels</code>  that I found <a rel="nofollow" href="https://stackoverflow.com/questions/4285214/predict-lm-with-an-unknown-factor-level-in-test-data" target="_blank">here</a> written by <a rel="nofollow" href="https://stackoverflow.com/users/4185785/pat-s" target="_blank">pat-s</a>. You need <em>magrittr</em> library installed and it can only work on <em>lm</em>, <em>glm</em> and <em>glmmPQL</em> objects.</p>
<p></p>
<p>Once you’ve sourced the above function in R, you can seamlessly proceed with using your trained model to make predictions on the test set. The code below demonstrates this for the data set shared above. You can find these codes in one of my <a rel="nofollow" href="https://github.com/anirudhjayaraman/shiny-fiesta/blob/master/factor_new_levels.R" target="_blank">github repos</a> and try it out yourself.</p>
<p><br> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/pythonandr.wordpress.com/2310/" target="_blank"><img border="0" alt="" src="http://feeds.wordpress.com/1.0/comments/pythonandr.wordpress.com/2310/"></a> <img border="0" alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" data-recalc-dims="1"><img width="1" alt="" src="http://feeds.feedburner.com/~r/RDiscoveringPythonR/~4/CfUhRar0uRE" height="1"></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://feedproxy.google.com/~r/RDiscoveringPythonR/~3/CfUhRar0uRE/"> R – Discovering Python &amp; R</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/QQptJGI379c" height="1"></a></div>