<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/gradient-descent/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://jean9208.github.io/gradient_descent/"> --Jean Arreola--</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<h2>Trying gradient descent for linear regression</h2>
<p>The best way to learn an algorith is to code it. So here it is, my take on Gradient Descent Algorithm for simple linear regression.</p>
<p>First, we fit a simple linear model with lm for comparison with gradient descent values.</p>
<div>
<pre><code><span>#Load libraries
</span><span>
</span><span>library</span><span>(</span><span>dplyr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>highcharter</span><span>)</span><span>
</span><span>#Scaling length variables from iris dataset.
</span><span>
</span><span>iris_demo</span><span> </span><span>&lt;-</span><span> </span><span>iris</span><span>[,</span><span>c</span><span>(</span><span>&quot;Sepal.Length&quot;</span><span>,</span><span>&quot;Petal.Length&quot;</span><span>)]</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>sepal_length</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>scale</span><span>(</span><span>Sepal.Length</span><span>)),</span><span>
</span><span>petal_length</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>scale</span><span>(</span><span>Petal.Length</span><span>)))</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>sepal_length</span><span>,</span><span>petal_length</span><span>)</span><span>
</span><span>#Fit a simple linear model to compare coefficients.
</span><span>
</span><span>regression</span><span> </span><span>&lt;-</span><span> </span><span>lm</span><span>(</span><span>iris_demo</span><span>$</span><span>petal_length</span><span>~</span><span>iris_demo</span><span>$</span><span>sepal_length</span><span>)</span><span>
</span><span>coef</span><span>(</span><span>regression</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##            (Intercept) iris_demo$sepal_length 
##           4.643867e-16           8.717538e-01
</code></pre>
</div>
<div>
<pre><code><span>iris_demo_reg</span><span> </span><span>&lt;-</span><span> </span><span>iris_demo</span><span>
</span><span>iris_demo_reg</span><span>$</span><span>reg</span><span> </span><span>&lt;-</span><span> </span><span>predict</span><span>(</span><span>regression</span><span>,</span><span>iris_demo</span><span>)</span><span>
</span><span>#Plot the model with highcharter
</span><span>
</span><span>highchart</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>hc_add_series</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>iris_demo_reg</span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>&quot;scatter&quot;</span><span>,</span><span> </span><span>hcaes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>sepal_length</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>petal_length</span><span>),</span><span> </span><span>name</span><span> </span><span>=</span><span> </span><span>&quot;Sepal Length VS Petal Length&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>hc_add_series</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>iris_demo_reg</span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>&quot;line&quot;</span><span>,</span><span> </span><span>hcaes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>sepal_length</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>reg</span><span>),</span><span> </span><span>name</span><span> </span><span>=</span><span> </span><span>&quot;Linear Regression&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>hc_title</span><span>(</span><span>text</span><span> </span><span>=</span><span> </span><span>&quot;Linear Regression&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><iframe height="500" src="https://jean9208.github.io/html_wg/gradient_descent/linear_regression.html"></iframe></p>
<p><a target="_blank" rel="nofollow" href="https://jean9208.github.io/html_wg/gradient_descent/linear_regression.html">open</a></p>
<p>We will try to acomplish the same coefficients, this time using Gradient Descent.</p>
<div>
<pre><code><span>library</span><span>(</span><span>tidyr</span><span>)</span><span>
</span><span>set.seed</span><span>(</span><span>135</span><span>)</span><span> </span><span>#To reproduce results
</span><span>
</span><span>#Auxiliary function
</span><span>
</span><span># y = mx + b
</span><span>
</span><span>reg</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>m</span><span>,</span><span>b</span><span>,</span><span>x</span><span>)</span><span> </span><span>return</span><span>(</span><span>m</span><span> </span><span>*</span><span> </span><span>x</span><span> </span><span>+</span><span> </span><span>b</span><span>)</span><span>
</span><span>#Starting point
</span><span>
</span><span>b</span><span> </span><span>&lt;-</span><span> </span><span>runif</span><span>(</span><span>1</span><span>)</span><span>
</span><span>m</span><span> </span><span>&lt;-</span><span> </span><span>runif</span><span>(</span><span>1</span><span>)</span><span>
</span><span>#Gradient descent function
</span><span>
</span><span>gradient_desc</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>b</span><span>,</span><span> </span><span>m</span><span>,</span><span> </span><span>data</span><span>,</span><span> </span><span>learning_rate</span><span> </span><span>=</span><span> </span><span>0.01</span><span>){</span><span> </span><span># Small steps
</span><span> </span><span># Column names = Code easier to understand
</span><span> </span><span>colnames</span><span>(</span><span>data</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;x&quot;</span><span>,</span><span>&quot;y&quot;</span><span>)</span><span>
</span><span>#Values for first iteration
</span><span> </span><span>b_iter</span><span> </span><span>&lt;-</span><span> </span><span>0</span><span> </span><span>m_iter</span><span> </span><span>&lt;-</span><span> </span><span>0</span><span>
</span><span>n</span><span> </span><span>&lt;-</span><span> </span><span>nrow</span><span>(</span><span>data</span><span>)</span><span>
</span><span># Compute the gradient for Mean Squared Error function
</span><span> </span><span>for</span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>n</span><span>){</span><span>
</span><span># Partial derivative for b
</span><span> </span><span>b_iter</span><span> </span><span>&lt;-</span><span> </span><span>b_iter</span><span> </span><span>+</span><span> </span><span>(</span><span>-2</span><span>/</span><span>n</span><span>)</span><span> </span><span>*</span><span> </span><span>(</span><span>data</span><span>$</span><span>y</span><span>[</span><span>i</span><span>]</span><span> </span><span>-</span><span> </span><span>((</span><span>m</span><span> </span><span>*</span><span> </span><span>data</span><span>$</span><span>x</span><span>[</span><span>i</span><span>])</span><span> </span><span>+</span><span> </span><span>b</span><span>))</span><span>
</span><span># Partial derivative for m
</span><span> </span><span>m_iter</span><span> </span><span>&lt;-</span><span> </span><span>m_iter</span><span> </span><span>+</span><span> </span><span>(</span><span>-2</span><span>/</span><span>n</span><span>)</span><span> </span><span>*</span><span> </span><span>data</span><span>$</span><span>x</span><span>[</span><span>i</span><span>]</span><span> </span><span>*</span><span> </span><span>(</span><span>data</span><span>$</span><span>y</span><span>[</span><span>i</span><span>]</span><span> </span><span>-</span><span> </span><span>((</span><span>m</span><span> </span><span>*</span><span> </span><span>data</span><span>$</span><span>x</span><span>[</span><span>i</span><span>])</span><span> </span><span>+</span><span> </span><span>b</span><span>))</span><span>
</span><span>}</span><span>
</span><span># Move to the OPPOSITE direction of the derivative
</span><span> </span><span>new_b</span><span> </span><span>&lt;-</span><span> </span><span>b</span><span> </span><span>-</span><span> </span><span>(</span><span>learning_rate</span><span> </span><span>*</span><span> </span><span>b_iter</span><span>)</span><span>
</span><span>new_m</span><span> </span><span>&lt;-</span><span> </span><span>m</span><span> </span><span>-</span><span> </span><span>(</span><span>learning_rate</span><span> </span><span>*</span><span> </span><span>m_iter</span><span>)</span><span>
</span><span># Replace values and return
</span><span> </span><span>new</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>(</span><span>new_b</span><span>,</span><span>new_m</span><span>)</span><span>
</span><span>return</span><span>(</span><span>new</span><span>)</span><span>
</span><span>}</span><span>
</span><span># I need to store some values to make the motion plot
</span><span>
</span><span>vect_m</span><span> </span><span>&lt;-</span><span> </span><span>m</span><span>
</span><span>vect_b</span><span> </span><span>&lt;-</span><span> </span><span>b</span><span>
</span><span># Iterate to obtain better parameters
</span><span>
</span><span>for</span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>1000</span><span>){</span><span>
</span><span>if</span><span>(</span><span>i</span><span> </span><span>%in%</span><span> </span><span>c</span><span>(</span><span>1</span><span>,</span><span>100</span><span>,</span><span>250</span><span>,</span><span>500</span><span>)){</span><span> </span><span># I keep some values in the iteration for the plot
</span><span> </span><span>vect_m</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>vect_m</span><span>,</span><span>m</span><span>)</span><span>
</span><span>vect_b</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>vect_b</span><span>,</span><span>b</span><span>)</span><span>
</span><span>}</span><span> </span><span>x</span><span> </span><span>&lt;-</span><span> </span><span>gradient_desc</span><span>(</span><span>b</span><span>,</span><span>m</span><span>,</span><span>iris_demo</span><span>)</span><span>
</span><span>b</span><span> </span><span>&lt;-</span><span> </span><span>x</span><span>[[</span><span>1</span><span>]]</span><span>
</span><span>m</span><span> </span><span>&lt;-</span><span> </span><span>x</span><span>[[</span><span>2</span><span>]]</span><span>
</span><span>}</span><span>
</span><span>print</span><span>(</span><span>paste0</span><span>(</span><span>&quot;m = &quot;</span><span>,</span><span> </span><span>m</span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] &quot;m = 0.871753774273602&quot;
</code></pre>
</div>
<div>
<pre><code><span>print</span><span>(</span><span>paste0</span><span>(</span><span>&quot;b = &quot;</span><span>,</span><span> </span><span>b</span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] &quot;b = 5.52239677041512e-10&quot;
</code></pre>
</div>
<p>The difference in the coefficients is minimal.</p>
<p>We can see how the iterations work in the next plot:</p>
<div>
<pre><code><span>#Compute new values
</span><span>
</span><span>iris_demo</span><span>$</span><span>preit</span><span> </span><span>&lt;-</span><span> </span><span>reg</span><span>(</span><span>vect_m</span><span>[</span><span>1</span><span>],</span><span>vect_b</span><span>[</span><span>1</span><span>],</span><span>iris_demo</span><span>$</span><span>sepal_length</span><span>)</span><span>
</span><span>iris_demo</span><span>$</span><span>it1</span><span> </span><span>&lt;-</span><span> </span><span>reg</span><span>(</span><span>vect_m</span><span>[</span><span>2</span><span>],</span><span>vect_b</span><span>[</span><span>2</span><span>],</span><span>iris_demo</span><span>$</span><span>sepal_length</span><span>)</span><span>
</span><span>iris_demo</span><span>$</span><span>it100</span><span> </span><span>&lt;-</span><span> </span><span>reg</span><span>(</span><span>vect_m</span><span>[</span><span>3</span><span>],</span><span>vect_b</span><span>[</span><span>3</span><span>],</span><span>iris_demo</span><span>$</span><span>sepal_length</span><span>)</span><span>
</span><span>iris_demo</span><span>$</span><span>it250</span><span> </span><span>&lt;-</span><span> </span><span>reg</span><span>(</span><span>vect_m</span><span>[</span><span>4</span><span>],</span><span>vect_b</span><span>[</span><span>4</span><span>],</span><span>iris_demo</span><span>$</span><span>sepal_length</span><span>)</span><span>
</span><span>iris_demo</span><span>$</span><span>it500</span><span> </span><span>&lt;-</span><span> </span><span>reg</span><span>(</span><span>vect_m</span><span>[</span><span>5</span><span>],</span><span>vect_b</span><span>[</span><span>5</span><span>],</span><span>iris_demo</span><span>$</span><span>sepal_length</span><span>)</span><span>
</span><span>iris_demo</span><span>$</span><span>finalit</span><span> </span><span>&lt;-</span><span> </span><span>reg</span><span>(</span><span>m</span><span>,</span><span>b</span><span>,</span><span>iris_demo</span><span>$</span><span>sepal_length</span><span>)</span><span>
</span><span>iris_gathered</span><span> </span><span>&lt;-</span><span> </span><span>iris_demo</span><span> </span><span>%>%</span><span> </span><span>gather</span><span>(</span><span>key</span><span> </span><span>=</span><span> </span><span>gr</span><span>,</span><span> </span><span>value</span><span> </span><span>=</span><span> </span><span>val</span><span>,</span><span> </span><span>preit</span><span>:</span><span>finalit</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>-</span><span>petal_length</span><span>)</span><span> </span><span>%>%</span><span> </span><span>distinct</span><span>()</span><span>
</span><span>iris_start</span><span> </span><span>&lt;-</span><span> </span><span>iris_gathered</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>gr</span><span> </span><span>==</span><span> </span><span>&quot;preit&quot;</span><span>)</span><span>
</span><span>iris_seq</span><span> </span><span>&lt;-</span><span> </span><span>iris_gathered</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>sepal_length</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>do</span><span>(</span><span>sequence</span><span> </span><span>=</span><span> </span><span>list_parse</span><span>(</span><span>select</span><span>(</span><span>.</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>val</span><span>)))</span><span>
</span><span>iris_data</span><span> </span><span>&lt;-</span><span> </span><span>left_join</span><span>(</span><span>iris_start</span><span>,</span><span> </span><span>iris_seq</span><span>)</span><span>
</span><span>#Motion Plot
</span><span>
</span><span>irhc2</span><span> </span><span>&lt;-</span><span> </span><span>highchart</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>hc_add_series</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>iris_data</span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>&quot;line&quot;</span><span>,</span><span> </span><span>hcaes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>sepal_length</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>val</span><span>),</span><span> </span><span>name</span><span> </span><span>=</span><span> </span><span>&quot;Gradient Descent&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>hc_motion</span><span>(</span><span>enabled</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>series</span><span> </span><span>=</span><span> </span><span>0</span><span>,</span><span> </span><span>startIndex</span><span> </span><span>=</span><span> </span><span>0</span><span>,</span><span>
</span><span>labels</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Iteration 1&quot;</span><span>,</span><span>&quot;Iteration 100&quot;</span><span>,</span><span>&quot;Iteration 250&quot;</span><span>,</span><span>&quot;Iteration 500&quot;</span><span>,</span><span>&quot;Final Iteration&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>hc_add_series</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>iris_demo_reg</span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>&quot;scatter&quot;</span><span>,</span><span> </span><span>hcaes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>sepal_length</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>petal_length</span><span>),</span><span> </span><span>name</span><span> </span><span>=</span><span> </span><span>&quot;Sepal Length VS Petal Length&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>hc_title</span><span>(</span><span>text</span><span> </span><span>=</span><span> </span><span>&quot;Gradient Descent Iterations&quot;</span><span>)</span><span>
</span><span>irhc2</span><span>
</span></code></pre>
</div>
<p><iframe height="500" src="https://jean9208.github.io/html_wg/gradient_descent/gradient_descent_iterations.html"></iframe></p>
<p><a target="_blank" rel="nofollow" href="https://jean9208.github.io/html_wg/gradient_descent/gradient_descent_iterations.html">open</a></p>
<p>Maybe in a future post we can try a multivariate regression model!</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://jean9208.github.io/gradient_descent/"> --Jean Arreola--</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/1iN1svD4XUg"></a></div>