<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/example-of-overfitting/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://matloff.wordpress.com/2018/11/16/example-of-overfitting/"> Mad (Data) Scientist</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>I occasionally see queries on various social media as to overfitting — what is it?, etc. I’ll post an example here. (I mentioned it at <a rel="nofollow" href="http://heather.cs.ucdavis.edu/BARUG.polyanNA.pdf" target="_blank">my talk the other night</a> on our novel approach to missing values, but had a bug in the code. Here is the correct account.)</p>
<p>The dataset is <strong>prgeng</strong>, on wages of programmers and engineers in Silicon Valley as of the 2000 Census. It’s included in our <a rel="nofollow" href="http://github.com/matloff/polyreg" target="_blank"><strong>polyreg</strong> package</a>, which we developed as <a rel="nofollow" href="http://heather.cs.ucdavis.edu/BARUG.polyanNA.pdf" target="_blank">an alternative to neural networks</a>. But it is quite useful in its own right, as  it makes it very convenient to fit multivariate polynomial models. (Not as easy as it sounds; e.g. one must avoid cross products of orthogonal dummy variables, powers of those variables, etc.)</p>
<p>First I split the data into training and test sets:</p>
<p> </p>
<pre>> set.seed(88888)
> getPE()
> pe1 &lt;- pe[,c(1,2,4,6,7,12:16,3)]
> testidxs &lt;- sample(1:nrow(pe1),1000)
> testset &lt;- pe1[testidxs,]
> trainset &lt;- pe1[-testidxs,]
</pre>
<p>As a base, I fit an ordinary degree-1 model and found the mean absolute prediction error:</p>
<pre>> lmout &lt;- lm(wageinc ~ .,data=trainset)
> predvals &lt;- predict(lmout,testset[,-11])
> mean(abs(predvals - testset[,11]))
[1] 25456.98
</pre>
<p>Next, I tried a quadratic model:</p>
<pre>> pfout &lt;- polyFit(trainset,deg=2)
> mean(abs(predict(pfout,testset[,-11]) 
   - testset[,11]))
[1] 24249.83
</pre>
<p>Note that, though originally there were 10 predictors, now with polynomial terms we have 46.</p>
<p>I kept going:</p>
<table border="1">
<tbody>
<tr>
<td>deg</td>
<td>MAPE</td>
<td># terms</td>
</tr>
<tr>
<td>3</td>
<td>23951.69</td>
<td>118</td>
</tr>
<tr>
<td>4</td>
<td>23974.76</td>
<td>226</td>
</tr>
<tr>
<td>5</td>
<td>24340.85</td>
<td>371</td>
</tr>
<tr>
<td>6</td>
<td>24554.34</td>
<td>551</td>
</tr>
<tr>
<td>7</td>
<td>36463.61</td>
<td>767</td>
</tr>
<tr>
<td>8</td>
<td>74296.09</td>
<td>1019</td>
</tr>
</tbody>
</table>
<p>One must keep in mind the effect of sampling variation, and repeated trials would be useful here, but it seems that the data can stand at least a cubic fit and possibly as much as degree 5 or even 6. To be conservative, it would seem wise to stop at degree 3. That’s also consistent with the old Tukey rule of thumb that we should have p &lt;-sqrt(n), n being about 20,000 here.</p>
<p>In any event, the effects of overfitting here are dramatic, starting at degree 7.</p>
<p>It should be noted that I made no attempt to clean the data, nor to truncate predicted values at 0, etc.</p>
<p>It should also be noted that, starting at degree 4, R emitted warnings, “prediction from a rank-deficient fit may be misleading.” It is well known that at high degrees, polynomial terms can have multicollinearity issues.</p>
<p>Indeed, this is a major point we make in our arXiv paper cited above. There we argue that neural networks are polynomial models in disguise, with the effective degree of the polynomial increasing a lot at each successive layer, and thus multicollinearity increasing from layer to layer. We’ve confirmed this empirically. We surmise that this is a major cause of convergence problems in NNs.</p>
<p>Finally, whenever I talk about polynomials and NNs, I hear the standard (and correct) concern that polynomial grow rapidly at the edges of the data. True, but I would point out that if you accept NNs = polynomials, then the same is true for NNs.</p>
<p>We’re still working on the polynomials/NNs project. More developments to be announced soon. But for those who are wondering about overfitting, the data here should make the point.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://matloff.wordpress.com/2018/11/16/example-of-overfitting/"> Mad (Data) Scientist</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/oZYxQN5lKwI" height="1"></a></div>