<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/complete-subset-regressions-simple-and-powerful/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=https%3A%2F%2Finsightr.wordpress.com%2F2017%2F06%2F01%2Fcomplete-subset-regressions-simple-and-powerfull%2F"> R – insightR</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div><strong>By Gabriel Vasconcelos</strong></div>
<div></div>
<p>The complete subset regressions (CSR) is a forecasting method proposed by Elliott, Gargano and Timmermann in 2013. It is as very simple but powerful technique. Suppose you have a set of variables and you want to forecast one of them using information from the others. If your variables are highly correlated and the variable you want to predict is noisy you will have collinearity problems and in-sample overfitting because the model will try to fit the noise.</p>
<p><span></span></p>
<p>These problems may be solved if you estimate a smaller model using only a subset of the explanatory variables, however, if you do not know which variables are important you may loose information. What if we estimate models from many different subsets and combine their forecasts? Even better, what if we estimate models for all possible combinations of variables? This would be a good solution, however, if you have only 20 variables, the number of regressions would be more the 1 million. The CSR is a solution between using only one subset and all possible subsets. Instead of estimating all possible combinations of models, we fix a value <img title="k" alt="k" class="latex" src="https://s0.wp.com/latex.php?latex=k&bg=ffffff&%23038;fg=333333&%23038;s=0"> and estimate all possible models with <img title="k" alt="k" class="latex" src="https://s0.wp.com/latex.php?latex=k&bg=ffffff&%23038;fg=333333&%23038;s=0"> variables. Then we compute the forecasts from all these models to get our final result. Naturaly, <img title="k" alt="k" class="latex" src="https://s0.wp.com/latex.php?latex=k&bg=ffffff&%23038;fg=333333&%23038;s=0"> must be smaller than the number of variables you have. Let us review the steps:</p>
<ul>
<li>Suppose you have <img title="K" alt="K" class="latex" src="https://s0.wp.com/latex.php?latex=K&bg=ffffff&%23038;fg=333333&%23038;s=0"> explanatory variables. Estimate all possible models with <img title="k&lt;K" alt="k&lt;K" class="latex" src="https://s0.wp.com/latex.php?latex=k%3CK&bg=ffffff&%23038;fg=333333&%23038;s=0"> variables,</li>
<li>Compute the forecasts for all the models,</li>
<li>Compute the average of all the forecasts to have the final result.</li>
</ul>
<h2>Aplication</h2>
<p>We are going to generate data from a linear model where the explanatory variables, <img title="X" alt="X" class="latex" src="https://s0.wp.com/latex.php?latex=X&bg=ffffff&%23038;fg=333333&%23038;s=0">, are draw from a multivariate normal distribution. The coefficients are generated from a normal distribution and multiplied by a parameter <img title="alpha" alt="alpha" class="latex" src="https://s0.wp.com/latex.php?latex=%5Calpha&bg=ffffff&%23038;fg=333333&%23038;s=0"> to ensure the dependent variable <img title="y" alt="y" class="latex" src="https://s0.wp.com/latex.php?latex=y&bg=ffffff&%23038;fg=333333&%23038;s=0"> has a significant random component. The value for <img title="K" alt="K" class="latex" src="https://s0.wp.com/latex.php?latex=K&bg=ffffff&%23038;fg=333333&%23038;s=0"> is 10 and the CSR is estimated with <img title="k=4" alt="k=4" class="latex" src="https://s0.wp.com/latex.php?latex=k%3D4&bg=ffffff&%23038;fg=333333&%23038;s=0">.</p>
<pre>set.seed(1) # = Seed for replication = #
K = 10 # = Number of Variables = #
T = 300 # = Number of Observations = #

# = Generate covariance matrix = #
library(mvtnorm)
D = diag(0.1, K)
P = matrix(rnorm(K * K), K, K)
sigma = t(P)%*%D%*%P

alpha = 0.1
beta = rnorm(K) * alpha # = coefficients = #
X = rmvnorm(T, sigma = sigma) # = Explanatory Variables = #
u = rnorm(T) # = Error = #
y = X%*%beta + u # = Generate y = #

# = Break data into in-sample and out-of-sample = #
y.in = y[1:200]
y.out = y[-c(1:200)]
X.in = X[1:200, ]
X.out = X[-c(1:200), ]
</pre>
<pre># = Estimate model by OLS to compare = #
model = lm(y.in ~ X.in)
pred = cbind(1, X.out)%*%coef(model)

# = CSR = #
k = 4
models = combn(K, k) # = Gives all combinations with 4 variables = #
csr.fitted = rep(0, length(y.in)) # = Store in-sample fitted values = #
csr.pred = rep(0, length(y.out)) # = Store forecast = #
for(i in 1:ncol(models)){
  m = lm(y.in ~ X.in[ ,models[ ,i]])
  csr.fitted = csr.fitted + fitted(m)
  csr.pred = csr.pred + cbind(1, X.out[ ,models[ ,i]])%*%coef(m)
}

R2ols = 1 - var(y.in - fitted(model))/var(y.in) # = R2 OLS = #
R2csr = 1 - var(y.in - csr.fitted/ncol(models))/var(y.in) # = R2 CSR = #
c(R2ols, R2csr)
</pre>
<pre>## [1] 0.1815733 0.1461342
</pre>
<pre># = In-sample fit = #
plot(y.in, type=&quot;l&quot;)
lines(fitted(model), col=2)
lines(csr.fitted/ncol(models), col=4)
</pre>
<p><img alt="plot of chunk unnamed-chunk-7" src="https://i0.wp.com/i.imgur.com/0bJ2wwO.png"></p>
<pre># = Out-of-sample fit = #
plot(y.out, type=&quot;l&quot;)
lines(pred, col = 2)
lines(csr.pred/ncol(models), col = 4)
</pre>
<p><img alt="plot of chunk unnamed-chunk-7" src="https://i2.wp.com/i.imgur.com/AD2ryDa.png"></p>
<pre># = MAPE = #
MAPEols=mean(abs(y.out - pred))
MAPEcsr=mean(abs(y.out - csr.pred/ncol(models)))
c(MAPEols, MAPEcsr)
</pre>
<pre>## [1] 0.8820019 0.8446682
</pre>
<p>The main conclusion from the results is that the CSR gives up some in-sample performance to improve the forecasts. In fact, the CSR is very robust to overfitting and you should definitively add it to your collection to use when you believe that you are having this type of problem. Most modern forecasting techniques have the same idea of accepting some bias in-sample to have a more parsimonious or stable model out-of-sample.</p>
<p>This is the simplest implementation possible for the CSR. In some cases you may have fixed controls you want to include in all regressions such as lags of the dependent variable. The CSR computational costs increase fast with the number of variables. If you are in a high-dimensional framework you may need to do some type of pre-selection of the variables to reduce the problem’s size.</p>
<h2>References</h2>
<p>Elliott, Graham, Antonio Gargano, and Allan Timmermann. “Complete subset regressions.” Journal of Econometrics 177.2 (2013): 357-373.</p>
<p> <a target="_blank" rel="nofollow" href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=http%3A%2F%2Ffeeds.wordpress.com%2F1.0%2Fgocomments%2Finsightr.wordpress.com%2F238%2F"><img alt="" src="http://feeds.wordpress.com/1.0/comments/insightr.wordpress.com/238/" border="0"></a> <img alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" border="0" data-recalc-dims="1"></p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=https%3A%2F%2Finsightr.wordpress.com%2F2017%2F06%2F01%2Fcomplete-subset-regressions-simple-and-powerfull%2F"> R – insightR</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=https%3A%2F%2Ffeedburner.google.com%2Ffb%2Fa%2Fmailverify%3Furi%3DRBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=https%3A%2F%2Fwww.r-project.org%2F">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=https%3A%2F%2Fwww.r-users.com%2F">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/wPbalAJvjAw"></a></div>