<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/more-on-orthogonal-regression/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://davegiles.blogspot.com/2016/12/more-on-orthogonal-regression.html"> Econometrics Beat: Dave Giles' Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div dir="ltr">
<div>Some time ago I wrote <a target="_blank" rel="nofollow" href="http://davegiles.blogspot.ca/2014/11/orthogonal-regression-first-steps.html">a post</a> about orthogonal regression. This is where we fit a regression line so that we minimize the sum of the squares of the <i>orthogonal</i> (rather than <i>vertical</i>) distances from the data points to the regression line.</div>
<div></div>
<div>Subsequently, I received the following email comment:</div>
<blockquote><p><span><span>“Thanks for this blog post. I enjoyed reading it. I’m wondering how straightforward you think this would be to extend orthogonal regression to the case of two independent variables? Assume both independent variables are meaningfully measured in the same units.”</span></span></p></blockquote>
<div>Well, we don’t have to make the latter assumption about units in order to answer this question. And we don’t have to limit ourselves to just two regressors. Let’s suppose that we have p of them.</div>
<div></div>
<div>In fact, I hint at the answer to the question posed above towards the end of my earlier post, when I say,<span> “<span>Finally, it will come as no surprise to hear that there’s a close connection between orthogonal least squares and principal components analysis.”</span></span><br><span><span><br></span></span><span><span>What was I referring to, exactly?</span></span><br><a name="more"></a><span>Well, just recall how we define the Principal Components of a multivariate set of data. Suppose that the data are in the form of an (n x p) matrix, X. There are n observations, and p variables. An orthogonal transformation is applied to X. This results in r (le p) new variables that are linearly uncorrelated.  These are the principal components (PC’s) of the data, and they are ordered as follow. The first PC accounts for the most of the variability in the original data. The second PC accounts for the maximum amount of the <i>remaining</i> variability in the data, subject to the constraint that it is uncorrelated with (<i>i.e.</i>, <i>orthogonal to</i>) the first PC. </span><br><span><br></span><span>Note how <i>orthogonality</i> has crept into the story!</span><br><span><br></span><span>We then continue – the third PC accounts for the maximum amount of the remaining variability in the data, subject to the constraint that it is orthogonal to <i>both</i> the first and second PC’s. <i>etc.</i></span><br><span><i><br></i></span><span>You’ll find examples of PC analysis being used in a statistically descriptive way in some earlier posts of mine – <i>e.g.</i>, <a target="_blank" rel="nofollow" href="http://davegiles.blogspot.ca/2011/06/even-better-life.html">here</a> and <a target="_blank" rel="nofollow" href="http://davegiles.blogspot.ca/2011/06/on-correlation-between-crime-and.html">here</a>.</span><br><span><br></span><span>We can use (some of) the PC’s of the regressor data as explanatory variables in a regression model. A useful reference for this can be found <a target="_blank" rel="nofollow" href="https://onlinecourses.science.psu.edu/stat505/node/49">here</a>. Note that, by construction, these transformed explanatory variables will have zero multicollinearity.</span><br><span><br></span><span>So, in the multivariate case, orthogonal regression is just least squares regression using a sub-set of the principal components of the original regressor matrix as the explanatory variables. We also sometimes call it Total Least Squares.</span><br><span><br></span><span>In this <a target="_blank" rel="nofollow" href="http://davegiles.blogspot.ca/2011/09/estimating-models-with-under-sized.html">earlier post</a> I talked about using Principal Components Regression (PCR) in the context of simultaneous equations models. The problem there was that we can’t construct the 2SLS estimator if the sample size is smaller than the total number of predetermined variables in the entire system. (This used to be referred to as the “under-sized sample” problem.) One solution was to use a few of the principal components of the matrix of data on the predetermined variables, instead of all of the latter variables, at the first stage of 2SLS. (Usually, just the first few principal components will capture almost all of the variability in the original data.)</span><br><span><br></span><span>There are some useful discussions of this that you might want to refer to. For instance, </span><span><span><a target="_blank" rel="nofollow" href="http://zoonek.free.fr/">Vincent Zoonekynd</a> has a nice illustration <a target="_blank" rel="nofollow" href="http://zoonek2.free.fr/UNIX/48_R/09.html#7">here</a>. I particularly recommend two other pieces that discuss PCR using R – </span></span><a target="_blank" rel="nofollow" href="http://blog.revolutionanalytics.com/2016/05/principal-components-tutorial.html">this post</a>, <span>“</span><span>Principal components regression in R, an operational tutorial”, </span>by John Mount, on the <a target="_blank" rel="nofollow" href="http://blog.revolutionanalytics.com/">Revolutions blog</a>; and <a target="_blank" rel="nofollow" href="http://www.quantide.com/performing-principal-components-regression-pcr-in-r/">this post</a>, “Performing principal components regression (PCR) in R”, by Michy Alice, on the <a target="_blank" rel="nofollow" href="http://www.quantide.com/">Quantide</a> site.
<p>PCR also gets a brief mention in this <a target="_blank" rel="nofollow" href="http://davegiles.blogspot.ca/2012/10/what-i-learned-last-week.html">earlier post</a> of mine – see the discussion of the last paper mentioned in that post.</div>
<div>So, the bottom line is that while my introductory post dealt with just the single-regressor case, it’s straightforward to apply orthogonal <i>multiple</i> regression – it’s just regression using the first few principal components of the  regressor matrix.
</div>
<p><center>© 2016, David E. Giles </center></div>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://davegiles.blogspot.com/2016/12/more-on-orthogonal-regression.html"> Econometrics Beat: Dave Giles' Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/NVMQiJUxZgk"></a></div>