<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/regression-analysis-essentials-for-machine-learning/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.sthda.com/english/wiki/regression-analysis-essentials-for-machine-learning"> Easy Guides</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p></p>
<div>
<p><strong>Regression analysis</strong> consists of a set of <em>machine learning</em> methods that allow us to predict a continuous outcome variable (y) based on the value of one or multiple predictor variables (x).</p>
<p>Briefly, the goal of regression model is to build a mathematical equation that defines y as a function of the x variables. Next, this equation can be used to predict the outcome (y) on the basis of new values of the predictor variables (x).</p>
<p><a rel="nofollow" href="http://www.sthda.com/english/articles/40-regression-analysis/165-linear-regression-essentials-in-r/" target="_blank"><strong>Linear regression</strong></a> is the most simple and popular technique for predicting a continuous variable. It assumes a linear relationship between the outcome and the predictor variables.</p>
<p>The linear regression equation can be written as <code>y = b0 + b*x + e</code>, where:</p>
<ul>
<li>b0 is the intercept,</li>
<li>b is the regression weight or coefficient associated with the predictor variable x.</li>
<li>e is the residual error</li>
</ul>
<p>Technically, the linear regression coefficients are detetermined so that the error in predicting the outcome value is minimized. This method of computing the beta coefficients is called the <a rel="nofollow" href="http://www.sthda.com/english/articles/40-regression-analysis/165-linear-regression-essentials-in-r/" target="_blank"><strong>Ordinary Least Squares</strong></a> method.</p>
<p>When you have multiple predictor variables, say x1 and x2, the regression equation can be written as <code>y = b0 + b1*x1 + b2*x2 +e</code>. In some situations, there might be an <a rel="nofollow" href="http://www.sthda.com/english/articles/40-regression-analysis/164-interaction-effect-in-multiple-regression-essentials/" target="_blank"><strong>interaction effect</strong></a> between some predictors, that is for example, increasing the value of a predictor variable x1 may increase the effectiveness of the predictor x2 in explaining the variation in the outcome variable.</p>
<p>Note also that, linear regression models can incorporate both continuous and <a rel="nofollow" href="http://www.sthda.com/english/articles/40-regression-analysis/163-regression-with-categorical-variables-dummy-coding-essentials-in-r/" target="_blank"><strong>categorical predictor variables</strong></a>.</p>
<p>When you build the linear regression model, you need to <a rel="nofollow" href="http://www.sthda.com/english/articles/39-regression-model-diagnostics/" target="_blank"><strong>diagnostic</strong></a> whether linear model is suitable for your data.</p>
<p>In some cases, the relationship between the outcome and the predictor variables is not linear. In these situations, you need to build a <strong>non-linear regression</strong>, such as <a rel="nofollow" href="http://www.sthda.com/english/articles/40-regression-analysis/162-nonlinear-regression-essentials-in-r-polynomial-and-spline-regression-models/" target="_blank"><em>polynomial and spline regression</em></a>.</p>
<p>When you have multiple predictors in the regression model, you might want to select the best combination of predictor variables to build an optimal predictive model. This process called <a rel="nofollow" href="http://www.sthda.com/english/articles/37-model-selection-essentials-in-r/" target="_blank"><strong>model selection</strong></a>, consists of comparing multiple models containing different sets of predictors in order to select the best performing model that minimize the prediction error. Linear model selection approaches include <a rel="nofollow" href="http://www.sthda.com/english/articles/37-model-selection-essentials-in-r/155-best-subsets-regression-essentials-in-r/" target="_blank"><strong>best subsets regression</strong></a> and <a rel="nofollow" href="http://www.sthda.com/english/articles/37-model-selection-essentials-in-r/154-stepwise-regression-essentials-in-r/" target="_blank"><strong>stepwise regression</strong></a></p>
<p>In some situations, such as in genomic fields, you might have a large multivariate data set containing some correlated predictors. In this case, the information, in the original data set, can be summarized into few new variables (called principal components) that are a linear combination of the original variables. This few principal components can be used to build a linear model, which might be more performant for your data. This approach is know as <strong>principal component-based methods</strong>, which include: <a rel="nofollow" href="http://www.sthda.com/english/articles/37-model-selection-essentials-in-r/152-principal-component-and-partial-least-squares-regression-essentials/" target="_blank"><strong>principal component regression</strong> and <strong>partial least squares regression</strong></a>.</p>
<p>An alternative method to simplify a large multivariate model is to use <a rel="nofollow" href="http://www.sthda.com/english/articles/37-model-selection-essentials-in-r/153-penalized-regression-essentials-ridge-lasso-elastic-net/" target="_blank"><strong>penalized regression</strong></a>, which penalizes the model for having too many variables. The most well known penalized regression include <strong>ridge regression</strong> and the <strong>lasso regression</strong>.</p>
<p>You can apply all these different regression models on your data, compare the models and finally select the best approach that explains well your data. To do so, you need some statistical metrics to compare the performance of the different models in explaining your data and in predicting the outcome of new test data.</p>
<p>The best model is defined as the model that has the lowest prediction error. The <a rel="nofollow" href="http://www.sthda.com/english/articles/38-regression-model-validation/158-regression-model-accuracy-metrics-r-square-aic-bic-cp-and-more/" target="_blank">most popular metrics for comparing regression models</a>, include:</p>
<ul>
<li><strong>Root Mean Squared Error</strong>, which measures the model prediction error. It corresponds to the average difference between the observed known values of the outcome and the predicted value by the model. RMSE is computed as <code>RMSE = mean((observeds - predicteds)^2) %>% sqrt()</code>. The lower the RMSE, the better the model.</li>
<li><strong>Adjusted R-square</strong>, representing the proportion of variation (i.e., information), in your data, explained by the model. This corresponds to the overall quality of the model. The higher the adjusted R2, the better the model</li>
</ul>
<p>Note that, the above mentioned metrics should be computed on a new test data that has not been used to train (i.e. build) the model. If you have a large data set, with many records, you can randomly split the data into training set (80% for building the predictive model) and test set or validation set (20% for evaluating the model performance).</p>
<p>One of the most robust and popular approach for estimating a model performance is <a rel="nofollow" href="http://www.sthda.com/english/articles/38-regression-model-validation/157-cross-validation-essentials-in-r/" target="_blank"><strong>k-fold cross-validation</strong></a>. It can be applied even on a small data set. k-fold cross-validation works as follow:</p>
<ol>
<li>Randomly split the data set into k-subsets (or k-fold) (for example 5 subsets)</li>
<li>Reserve one subset and train the model on all other subsets</li>
<li>Test the model on the reserved subset and record the prediction error</li>
<li>Repeat this process until each of the k subsets has served as the test set.</li>
<li>Compute the average of the k recorded errors. This is called the cross-validation error serving as the performance metric for the model.</li>
</ol>
<p>Taken together, the best model is the model that has the lowest cross-validation error, RMSE.</p>
<p>In this Part, you will learn different methods for regression analysis and we’ll provide practical example in <strong>R</strong>.</p>
<p>The content is organized as follow:</p>
<ol>
<li><a rel="nofollow" href="http://www.sthda.com/english/articles/40-regression-analysis/" target="_blank">Regression Analysis</a>
<ul>
<li><a rel="nofollow" href="http://www.sthda.com/english/articles/40-regression-analysis/165-linear-regression-essentials-in-r/" target="_blank">Linear Regression</a></li>
<li><a rel="nofollow" href="http://www.sthda.com/english/articles/40-regression-analysis/164-interaction-effect-in-multiple-regression-essentials/" target="_blank">Interaction Effect in Multiple Regression</a></li>
<li><a rel="nofollow" href="http://www.sthda.com/english/articles/40-regression-analysis/163-regression-with-categorical-variables-dummy-coding-essentials-in-r/" target="_blank">Regression with Categorical Variables: Dummy Coding Essentials in R</a></li>
<li><a rel="nofollow" href="http://www.sthda.com/english/articles/40-regression-analysis/162-nonlinear-regression-essentials-in-r-polynomial-and-spline-regression-models/" target="_blank">Nonlinear Regression Essentials in R: Polynomial and Spline Regression Models</a></li>
</ul>
</li>
<li><a rel="nofollow" href="http://www.sthda.com/english/articles/39-regression-model-diagnostics/" target="_blank">Regression Model Diagnostics</a>
<ul>
<li><a rel="nofollow" href="http://www.sthda.com/english/articles/39-regression-model-diagnostics/161-linear-regression-assumptions-and-diagnostics-in-r-essentials/" target="_blank">Linear Regression Assumptions and Diagnostics in R</a></li>
<li><a rel="nofollow" href="http://www.sthda.com/english/articles/39-regression-model-diagnostics/160-multicollinearity-essentials-and-vif-in-r/" target="_blank">Multicollinearity Essentials and VIF in R</a></li>
<li><a rel="nofollow" href="http://www.sthda.com/english/articles/39-regression-model-diagnostics/159-confounding-variable-essentials/" target="_blank">Confounding Variable Essentials</a></li>
</ul>
</li>
<li><a rel="nofollow" href="http://www.sthda.com/english/articles/38-regression-model-validation/" target="_blank">Regression Model Validation</a>
<ul>
<li><a rel="nofollow" href="http://www.sthda.com/english/articles/38-regression-model-validation/158-regression-model-accuracy-metrics-r-square-aic-bic-cp-and-more/" target="_blank">Regression Model Accuracy Metrics: R-square, AIC, BIC, Cp and more</a></li>
<li><a rel="nofollow" href="http://www.sthda.com/english/articles/38-regression-model-validation/157-cross-validation-essentials-in-r/" target="_blank">Cross-Validation Essentials in R</a></li>
<li><a rel="nofollow" href="http://www.sthda.com/english/articles/38-regression-model-validation/156-bootstrap-resampling-essentials-in-r/" target="_blank">Bootstrap Resampling Essentials in R</a></li>
</ul>
</li>
<li><a rel="nofollow" href="http://www.sthda.com/english/articles/37-model-selection-essentials-in-r/" target="_blank">Model Selection Essentials in R</a>
<ul>
<li><a rel="nofollow" href="http://www.sthda.com/english/articles/37-model-selection-essentials-in-r/155-best-subsets-regression-essentials-in-r/" target="_blank">Best Subsets Regression Essentials in R</a></li>
<li><a rel="nofollow" href="http://www.sthda.com/english/articles/37-model-selection-essentials-in-r/154-stepwise-regression-essentials-in-r/" target="_blank">Stepwise Regression Essentials in R</a></li>
<li><a rel="nofollow" href="http://www.sthda.com/english/articles/37-model-selection-essentials-in-r/153-penalized-regression-essentials-ridge-lasso-elastic-net/" target="_blank">Penalized Regression Essentials: Ridge, Lasso &amp; Elastic Net</a></li>
<li><a rel="nofollow" href="http://www.sthda.com/english/articles/37-model-selection-essentials-in-r/152-principal-component-and-partial-least-squares-regression-essentials/" target="_blank">Principal Component and Partial Least Squares Regression Essentials</a></li>
</ul>
</li>
</ol>
</div>
<p></p>
<p></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.sthda.com/english/wiki/regression-analysis-essentials-for-machine-learning"> Easy Guides</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/yl9TGROzhog" height="1"></a></div>