<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/dealing-with-the-problem-of-multicollinearity-in-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://r-posts.com/dealing-with-the-problem-of-multicollinearity-in-r/"> R-posts.com</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Imagine a situation where you are asked to predict the tourism revenue for a country, let’s say India. In this case, your output or dependent or response variable will be total revenue earned (in USD) in a given year. But, what about independent or predictor variables? </p>
<p>You have been provided with two sets of predictor variables and you have to choose one of the sets to predict your output. The first set consists of three variables:</p>
<ul>
<li>X1 = Total number of tourists visiting the country</li>
<li>X2 = Government spending on tourism marketing</li>
<li>X3 = a*X1 + b*X2 + c, where a, b and c are some constants</li>
</ul>
<p>The second set also consists of three variables:</p>
<ul>
<li>X1 = Total number of tourists visiting the country</li>
<li>X2 = Government spending on tourism marketing</li>
<li>X3 = Average currency exchange rate </li>
</ul>
<p>Which of the two sets do you think provides us more information in predicting our output?</p>
<p>I am sure, you will agree with me that the second set provides us more information in predicting the output because the second set has three variables which are different from each other and each of the variables provides different information (we can infer this intuitively at this moment). Moreover, none of the three variables is directly derived from the other variables in the system. Alternatively, we can also say that none of the variables is a linear combination of other variables in the system.</p>
<p>In the first set of variables, only two variables provide us relevant information; while, the third variable is nothing but a linear combination of other two variables. If we were to directly develop a model without including this variable, our model would have considered this combination and estimated coefficients accordingly. </p>
<p>Now, this effect in the first set of variables is called multicollinearity. Variables in the first set are strongly correlated to each other (if not all, at least some variables are correlated with other variables). Model developed using the first set of variables may not provide as accurate results as the second one because we are missing out on relevant variables/information in the first set. Therefore, it becomes important to study multicollinearity and the techniques to detect and tackle its effect in regression models.</p>
<p>According to Wikipedia, <em>“Collinearity is a linear association between two explanatory variables. Two variables are perfectly collinear if there is an exact linear relationship between them. For example, X</em><em>1</em><em> and X</em><em>2</em><em> are perfectly collinear if there exist parameters λ</em><em>0</em><em> and λ</em><em>1</em><em> such that, for all observations i, we have</em></p>
<p><em>X</em><em>2i = </em><em>λ</em><em>0 </em><em>+ λ</em><em>1 </em><em>* X</em><em>1i</em></p>
<p><em>Multicollinearity refers to a situation in which two or more explanatory variables in a multiple regression model are highly linearly related.”</em></p>
<p>We saw an example of exactly what the Wikipedia definition is describing.</p>
<p>Perfect multicollinearity occurs when one independent variable is an exact linear combination of other variables. For example, you already have X and Y as independent variables and you add another variable, Z = a*X + b*Y, to the set of independent variables. Now, this new variable, Z, does not add any significant or different value than provided by X or Y. The model can adjust itself to set the parameters that this combination is taken care of while determining the coefficients.</p>
<p>Multicollinearity may arise from several factors. Inclusion or incorrect use of dummy variables in the system may lead to multicollinearity. The other reason could be the usage of derived variables, i.e., one variable is computed from other variables in the system. This is similar to the example we took at the beginning of the article. The other reason could be taking variables which are similar in nature or which provide similar information or the variables which have very high correlation among each other. </p>
<p>Multicollinearity may not possess problem at an overall level, but it strongly impacts the individual variables and their predictive power. You may not be able to identify which are statistically significant variables in your model. Moreover, you will be working with a set of variables which provide you similar output or variables which are redundant with respect to other variables. </p>
<ul>
<li>It becomes difficult to identify statistically significant variables. Since your model will become very sensitive to the sample you choose to run the model, different samples may show different statistically significant variables.</li>
<li>Because of multicollinearity, regression coefficients cannot be estimated precisely because the standard errors tend to be very high. Value and even sign of regression coefficients may change when different samples are chosen from the data.</li>
<li>Model becomes very sensitive to addition or deletion of any independent variable. If you add a variable which is orthogonal to the existing variable, your variable may throw completely different results. Deletion of a variable may also significantly impact the overall results.</li>
<li>Confidence intervals tend to become wider because of which we may not be able to reject the NULL hypothesis. The NULL hypothesis states that the true population coefficient is zero.</li>
</ul>
<p>Now, moving on to how to detect the presence of multicollinearity in the system.</p>
<p>There are multiple ways to detect the presence of multicollinearity among the independent or explanatory variables.</p>
<ul>
<li>The first and most rudimentary way is to create a pair-wise correlation plot among different variables. In most of the cases, variables will have some bit of correlation among each other, but high correlation coefficient may be a point of concern for us. It may indicate the presence of multicollinearity among variables.</li>
</ul>
<ul>
<li>Large variations in regression coefficients on addition or deletion of new explanatory or independent variables can indicate the presence of multicollinearity. The other thing could be significant change in the regression coefficients from sample to sample. With different samples, different statistically significant variables may come out.</li>
<li>The other method can be to use tolerance or variance inflation factor (VIF). </li>
</ul>
<p>VIF = 1 / Tolerance</p>
<p>VIF = 1/ (1 – R square)</p>
<p>VIF of over 10 indicates that the variables have high correlation among each other. Usually, VIF value of less than 4 is considered good for a model.</p>
<ul>
<li>The model may have very high R-square value but most of the coefficients are not statistically significant. This kind of a scenario may reflect multicollinearity in the system.</li>
<li>Farrar-Glauber test is one of the statistical test used to detect multicollinearity. This comprises of three further tests. The first, Chi-square test, examines whether multicollinearity is present in the system. The second test, F-test, determines which regressors or explanatory variables are collinear. The third test, t-test, determines the type or pattern of multicollinearity. </li>
</ul>
<p>We will now use some of these techniques and try their implementation in R.</p>
<p>We will use CPS_85_Wages data which consists of a random sample of 534 persons from the CPS (Current Population Survey). The data provides information on wages and other characteristics of the workers. (<a rel="nofollow" href="http://lib.stat.cmu.edu/datasets/CPS_85_Wages" target="_blank">Link</a> – <a rel="nofollow" href="http://lib.stat.cmu.edu/datasets/CPS_85_Wages" target="_blank">http://lib.stat.cmu.edu/datasets/CPS_85_Wages</a>). You can go through the data details on the link provided.</p>
<p>In this data, we will predict wages from other variables in the data.</p>
<table class="wp-block-table">
<tbody>
<tr>
<td>> data1 = read.csv(file.choose(), header = T)> head(data1)  Education South Sex Experience Union  Wage Age Race Occupation Sector Marr1         8 0 1         21 0 5.10 35  2 6 1 12         9 0 1         42 0 4.95 57  3 6 1 13        12 0  0 1     0 6.67 19 3         6 1 04        12 0  0 4     0 4.00 22 3         6 0 05        12 0  0 17     0 7.50 35 3         6 0 16        13 0  0 9     1 13.07 28 3         6 0 0> str(data1)’data.frame’: 534 obs. of  11 variables: $ Education : int  8 9 12 12 12 13 10 12 16 12 … $ South     : int 0 0 0 0 0 0 1 0 0 0 … $ Sex       : int 1 1 0 0 0 0 0 0 0 0 … $ Experience: int  21 42 1 4 17 9 27 9 11 9 … $ Union     : int 0 0 0 0 0 1 0 0 0 0 … $ Wage      : num 5.1 4.95 6.67 4 7.5 … $ Age       : int 35 57 19 22 35 28 43 27 33 27 … $ Race      : int 2 3 3 3 3 3 3 3 3 3 … $ Occupation: int  6 6 6 6 6 6 6 6 6 6 … $ Sector    : int 1 1 1 0 0 0 0 0 1 0 … $ Marr      : int 1 1 0 0 1 0 0 0 1 0 …</td>
</tr>
</tbody>
</table>
<p>The above results show the sample view of data and the variables present in the data. Now, let’s fit the linear regression model and analyze the results.</p>
<table class="wp-block-table">
<tbody>
<tr>
<td>> fit_model1 = lm(log(data1$Wage) ~., data = data1)> summary(fit_model1) Call:lm(formula = log(data1$Wage) ~ ., data = data1) Residuals:     Min     1Q Median       3Q Max -2.16246 -0.29163 -0.00469  0.29981 1.98248  Coefficients:             Estimate Std. Error t value Pr(>|t|)    (Intercept)  1.078596 0.687514   1.569 0.117291 Education    0.179366 0.110756   1.619 0.105949 South       -0.102360 0.042823  -2.390 0.017187 * Sex         -0.221997 0.039907  -5.563 4.24e-08 ***Experience   0.095822 0.110799   0.865 0.387531 Union        0.200483 0.052475   3.821 0.000149 ***Age         -0.085444 0.110730  -0.772 0.440671 Race         0.050406 0.028531   1.767 0.077865 . Occupation  -0.007417 0.013109  -0.566 0.571761 Sector       0.091458 0.038736   2.361 0.018589 * Marr         0.076611 0.041931   1.827 0.068259 . —Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 0.4398 on 523 degrees of freedomMultiple R-squared:  0.3185, Adjusted R-squared:  0.3054 F-statistic: 24.44 on 10 and 523 DF,  p-value: &lt; 2.2e-16</td>
</tr>
</tbody>
</table>
<p>The linear regression results show that the model is statistically significant as the F-statistic has high value and p-value for model is less than 0.05. However, on closer examination we observe that four variables – Education, Experience, Age and Occupation are not statistically significant; while, two variables Race and Marr (martial status) are significant at 10% level. Now, let’s plot the model diagnostics to validate the assumptions of the model.</p>
<table class="wp-block-table">
<tbody>
<tr>
<td>> plot(fit_model1)<br>Hit </td></tr></tbody></table>