<span><span lang="en">It is important to mention that the present posts began as a personal way of practicing R programming and machine learning. <span>Subsequently feedback from the community, urged me to continue performing these exercises and sharing them.</span><span> </span></span></span><br><span><span lang="en"><span>The bibliography and corresponding authors are cited at all times and it is a way of honoring them and giving them the credit they deserve for their work.</span></span></span><br><br><span><span lang="en"><span>We will develop a linear regression</span></span></span><span><span lang="en"><span><span><span lang="en"><span> example, </span></span></span></span></span></span><span><span lang="en"><span><span><span lang="en"><span><span lang="en"><span>including simple linear regression, multiple linear regression, linear regression with term interaction, linear regression with higher order terms, and finally with a transformation.</span></span></span></span></span> The exercise was originally published in <i>&quot;An Introduction to Statistical Learning. With applications in R&quot; </i>by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani. Springer 2015.</span></span></span><br><br><span><span lang="en"><span><span><span lang="en"><span><span><span lang="en"><span><span lang="en"><span>The example we will develop is about predicting the median house value on the Boston Housing dataset.</span></span></span></span></span></span></span></span> </span></span></span><br><span><span lang="en"><span><br></span></span></span><span><span lang="en"><span><span lang="en"><span><span lang="en"><span><span lang="en"><span><span lang="en"><span>We will carry out the exercise verbatim as published in the aforementioned reference</span></span></span></span></span></span><span lang="en"><span> and o</span></span></span></span></span></span><span lang="en"><span>nly with slight changes in the coding style.</span></span></span><br><br><span><span lang="en"><span>For more details on the models, algorithms and parameters interpretation, it is recommended to check the aforementioned reference or any other bibliography of your choice.</span></span></span><br><br><span><span lang="en"><span><br><br>### install and load required packages<br>#install.packages(&quot;ISLR&quot;)<br>#install.packages(&quot;car&quot;)<br><br>library(ISLR)<br>library(car)<br> </span></span></span><br><span><span lang="en"><span>### MASS library contains the Boston dataset <br>library(MASS)<br><br>### explore dataset<br>#fix(Boston)<br>str(Boston)<br>summary(Boston)<br>names(Boston)<br><br>### we will seek to predict target: medv (median house value)<br>### using 13 predictors<br><br>### we will start with $medv as target and $lstat as predictor<br>### create the model<br>lm.fit &lt;- lm(medv~lstat, data=Boston)<br><br>### explore the model parameters<br>lm.fit<br>summary(lm.fit)<br>names(lm.fit)<br>coef(lm.fit)<br><br>### explore confidence interval for the coefficient estimates<br>confint(lm.fit)<br><br>### use predict() to produce confidence intervals and prediction<br>### intervals for the prediction of $medv for a given value of<br>### $lstat<br>predict (lm.fit , data.frame(lstat = c(5 ,10 ,15) ),<br>          interval = &quot;confidence&quot;)<br>predict (lm.fit , data.frame(lstat = c(5 ,10 ,15) ),<br>          interval = &quot;prediction&quot;)<br> </span></span></span><br><span><span lang="en"><span>### 95% confidence interval associated with $lstat=10 is<br>### (24.47, 25.63)<br>### 95% prediction interval is (12.828, 37.28)<br>### center around a predicted value of 25.05 for $medv when<br>### $lstat = 10<br><br>### now plot $medv and lstat along with least squares regression<br>### line<br>plot(Boston$lstat , Boston$medv, main=&quot;Boston Housing Data&quot;,<br>      xlab=&quot;Percent of households with low<br>     socioeconomic status&quot;, ylab= &quot;Median house value&quot;)<br><br>abline(lm.fit, col = &quot;red&quot;, lwd = 3)</span></span></span><br><span><span lang="en"><span><br></span></span></span><br><div><a href="https://2.bp.blogspot.com/-unPiJifdOP0/WRX_5-HLzkI/AAAAAAAAC-8/_k5kCdsV-_MVRVKybFGhpOgxa-wDY8pSQCLcB/s1600/housing_1.png"></a></div><span><span lang="en"><span></span></span></span><br><span><span lang="en"><span></span></span></span><br><span><span lang="en"><span><br>### explore some diagnostic plots<br>par(mfrow = c(2, 2))<br>plot(lm.fit)<br><br>plot(predict(lm.fit), residuals(lm.fit))<br>plot(predict(lm.fit), rstudent(lm.fit))</span></span></span><br><div><a href="https://3.bp.blogspot.com/-ihmZgswShfM/WRYABxpagfI/AAAAAAAAC_A/FL6g3zYnUx41Tq7ZlFSv8TU9ICsLdW8eQCLcB/s1600/housing_2.png"></a></div><span><span lang="en"><span><br> </span></span></span><br><span><span lang="en"><span>### leverage statistics hatvalues()<br>plot(hatvalues(lm.fit))<br>which.max(hatvalues(lm.fit))</span></span></span><br><br><div><a href="https://1.bp.blogspot.com/-8c1vhQ9ZUnY/WRYAKuEedJI/AAAAAAAAC_E/vIp2Lgecre8-3n5sO9E6IxVbTGxkM045ACLcB/s1600/housing_3.png"></a></div><br><span><span lang="en"><span>### multiple linear regression<br>### two predictors<br>lm.fit &lt;- lm(medv ~ lstat + age , data = Boston)<br>summary(lm.fit)<br><br>### all 13 predictors<br>lm.fit &lt;- lm(medv ~. , data = Boston)<br>summary(lm.fit)<br><br>### R^2<br>summary(lm.fit)$r.sq<br>### RSE<br>summary(lm.fit)$sigma<br><br>### use vif() to compute variance inflation factors<br>### vif is part of car library<br>vif(lm.fit)<br><br>### in the last regression $age has a high p-value<br>### run regression using all predictors but one: $age<br>lm.fit1 &lt;- lm(medv ~. -age, data = Boston)<br>summary(lm.fit1)<br><br>### interaction terms<br>summary(lm(medv ~ lstat*age , data = Boston ))<br><br>### non-linear transformation of the predictors<br>lm.fit2 &lt;- lm(medv ~lstat + I(lstat ^2), data = Boston)<br>summary(lm.fit2)<br><br>### a near cero p-value associated with the quadratic term<br>### leads to an improved model<br><br>### use anova() to further quantify the extent to which quadratic<br>### fit is superior to the linear fit<br>lm.fit &lt;- lm(medv ~ lstat, data = Boston)<br>anova(lm.fit, lm.fit2)<br><br>### null hypothesis is that the two models fit data equally well<br>### alternative hypothesis is that the full model is superior<br>### F-statistic is 135 and p-value ~ 0 is clear evidence </span></span></span><br><span><span lang="en"><span>### that model containing the predictors lstat and lstat^2<br>### is far superior<br>### we saw in the above plots evidence of nonlinearity<br><br><br>### now use higher order polynomials with poly()<br>lm.fit5 = lm(medv ~ poly(lstat, 5), data = Boston)<br>summary(lm.fit5)<br><br>### additional polynomial terms leads to an improvement in the<br>### model fit<br><br>### now try a log transformation for $rm<br>summary (lm(medv ~ log(rm) , data = Boston ) )</span></span></span><br><br><br><br><span><span lang="en"><span><span lang="en"><span><span><span lang="en"><span>You can get the example in:</span></span></span></span></span></span></span></span><br><span><span lang="en"><span><span lang="en"><span><span><span lang="en"><span><a target="_blank" href="https://github.com/pakinja/Data-R-Value/tree/master/MachineLearning_LinearRegression_BostonHousing">https://github.com/pakinja/Data-R-Value</a></span></span></span></span></span></span></span></span><br>