<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/deep-learning-from-first-principles-in-python-r-and-octave-part-2/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/"> R – Giga thoughts …</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><em>“What does the world outside your head really ‘look’ like? Not only is there no color, there’s also no sound: the compression and expansion of air is picked up by the ears, and turned into electrical signals. The brain then presents these signals to us as mellifluous tones and swishes and clatters and jangles. Reality is also odorless: there’s no such thing as smell outside our brains. Molecules floating through the air bind to receptors in our nose and are interpreted as different smells by our brain. The real world is not full of rich sensory events; instead, our brains light up the world with their own sensuality.”</em><br>
The Brain: The Story of You” by David Eagleman</p>
<p>“<em>The world is Maya, illusory. The ultimate reality, the Brahman, is all-pervading and all-permeating, which is colourless, odourless, tasteless, nameless and formless</em>“<br>
Bhagavad Gita</p>
<h2>1. Introduction</h2>
<p>This post is a follow-up post to my earlier post <a rel="nofollow" href="https://gigadom.wordpress.com/2018/01/04/deep-learning-from-basic-principles-in-python-r-and-octave-part-1/" target="_blank">Deep Learning from first principles in Python, R and Octave-Part 1</a>. In the first part, I implemented Logistic Regression, in vectorized Python,R and Octave, with a wannabe Neural Network (a Neural Network with no hidden layers). In this second part, I implement a regular, but somewhat primitive Neural Network (a Neural Network with just 1 hidden layer). The 2nd part implements classification of manually created datasets, where the different clusters of the 2 classes are not linearly separable.</p>
<p>Neural Network perform really well in learning all sorts of non-linear boundaries between classes. Initially logistic regression is used perform the classification and the decision boundary is plotted. Vanilla logistic regression performs quite poorly. Using SVMs with a radial basis kernel would have performed much better in creating non-linear boundaries. To see R and Python implementations of SVMs take a look at my post <a rel="nofollow" href="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/" target="_blank">Practical Machine Learning with R and Python – Part 4</a>.</p>
<p>You could also check out my book on Amazon <a rel="nofollow" href="https://gigadom.wordpress.com/2017/12/05/my-book-practical-machine-learning-with-r-and-python-on-amazon/" target="_blank">Practical Machine Learning with R and Python – Machine Learning in Stereo</a>,  in which I implement several Machine Learning algorithms on regression and classification, along with other necessary metrics that are used in Machine Learning.</p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/untitled.png" data-orig-size="1074,230" data-medium-file="https://gigadom.files.wordpress.com/2018/01/untitled.png?w=300" data-attachment-id="6213" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/untitled.png?w=450&h=145&fit=676%2C145" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/untitled-117/" data-large-file="https://gigadom.files.wordpress.com/2018/01/untitled.png?w=450&amp;h=145&amp;fit=676%2C145" alt="" data-image-title="Untitled" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6213" data-recalc-dims="1"></p>
<p>You can clone and fork this R Markdown file along with the vectorized implementations of the 3 layer Neural Network for Python, R and Octave from Github <a rel="nofollow" href="https://github.com/tvganesh/DeepLearning-Part2" target="_blank">DeepLearning-Part2</a></p>
<h3>2. The 3 layer Neural Network</h3>
<p>A simple representation of a 3 layer Neural Network (NN) with 1 hidden layer is shown below.<br>
<img data-orig-file="https://gigadom.files.wordpress.com/2018/01/dl.png" data-orig-size="1280,720" data-medium-file="https://gigadom.files.wordpress.com/2018/01/dl.png?w=300" data-attachment-id="6181" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/dl.png?w=450&h=380&fit=676%2C380" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/dl/" data-large-file="https://gigadom.files.wordpress.com/2018/01/dl.png?w=450&amp;h=380&amp;fit=676%2C380" alt="" data-image-title="dl" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6181" data-recalc-dims="1"><br>
In the above Neural Network, there are 2 input features at the input layer, 3 hidden units at the hidden layer and 1 output layer as it deals with binary classification. The activation unit at the hidden layer can be a tanh, sigmoid, relu etc. At the output layer the activation is a sigmoid to handle binary classification</p>
<p># Superscript indicates layer 1<br>
<img alt="z_{11} = w_{11}^{1}x_{1} + w_{21}^{1}x_{2} + b_{1}" title="z_{11} = w_{11}^{1}x_{1} + w_{21}^{1}x_{2} + b_{1}" src="https://s0.wp.com/latex.php?latex=z_%7B11%7D+%3D+w_%7B11%7D%5E%7B1%7Dx_%7B1%7D+%2B+w_%7B21%7D%5E%7B1%7Dx_%7B2%7D+%2B+b_%7B1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="z_{12} = w_{12}^{1}x_{1} + w_{22}^{1}x_{2} + b_{1}" title="z_{12} = w_{12}^{1}x_{1} + w_{22}^{1}x_{2} + b_{1}" src="https://s0.wp.com/latex.php?latex=z_%7B12%7D+%3D+w_%7B12%7D%5E%7B1%7Dx_%7B1%7D+%2B+w_%7B22%7D%5E%7B1%7Dx_%7B2%7D+%2B+b_%7B1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="z_{13} = w_{13}^{1}x_{1} + w_{23}^{1}x_{2} + b_{1}" title="z_{13} = w_{13}^{1}x_{1} + w_{23}^{1}x_{2} + b_{1}" src="https://s0.wp.com/latex.php?latex=z_%7B13%7D+%3D+w_%7B13%7D%5E%7B1%7Dx_%7B1%7D+%2B+w_%7B23%7D%5E%7B1%7Dx_%7B2%7D+%2B+b_%7B1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>Also <img alt="a_{11} = tanh(z_{11})" title="a_{11} = tanh(z_{11})" src="https://s0.wp.com/latex.php?latex=a_%7B11%7D+%3D+tanh%28z_%7B11%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="a_{12} = tanh(z_{12})" title="a_{12} = tanh(z_{12})" src="https://s0.wp.com/latex.php?latex=a_%7B12%7D+%3D+tanh%28z_%7B12%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="a_{13} = tanh(z_{13})" title="a_{13} = tanh(z_{13})" src="https://s0.wp.com/latex.php?latex=a_%7B13%7D+%3D+tanh%28z_%7B13%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p># Superscript indicates layer 2<br>
<img alt="z_{21} = w_{11}^{2}a_{11} + w_{21}^{2}a_{12} + w_{31}^{2}a_{13} + b_{2}" title="z_{21} = w_{11}^{2}a_{11} + w_{21}^{2}a_{12} + w_{31}^{2}a_{13} + b_{2}" src="https://s0.wp.com/latex.php?latex=z_%7B21%7D+%3D+w_%7B11%7D%5E%7B2%7Da_%7B11%7D+%2B+w_%7B21%7D%5E%7B2%7Da_%7B12%7D+%2B+w_%7B31%7D%5E%7B2%7Da_%7B13%7D+%2B+b_%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="a_{21} = sigmoid(z21)" title="a_{21} = sigmoid(z21)" src="https://s0.wp.com/latex.php?latex=a_%7B21%7D+%3D+sigmoid%28z21%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>Hence<br>
<img alt="Z1= begin{pmatrix}  z11\  z12\  z13  end{pmatrix} =begin{pmatrix}  w_{11}^{1} &amp; w_{21}^{1} \  w_{12}^{1} &amp; w_{22}^{1} \  w_{13}^{1} &amp; w_{23}^{1}  end{pmatrix} * begin{pmatrix}  x1\  x2  end{pmatrix} + b_{1}" title="Z1= begin{pmatrix}  z11\  z12\  z13  end{pmatrix} =begin{pmatrix}  w_{11}^{1} &amp; w_{21}^{1} \  w_{12}^{1} &amp; w_{22}^{1} \  w_{13}^{1} &amp; w_{23}^{1}  end{pmatrix} * begin{pmatrix}  x1\  x2  end{pmatrix} + b_{1}" src="https://s0.wp.com/latex.php?latex=Z1%3D+%5Cbegin%7Bpmatrix%7D++z11%5C%5C++z12%5C%5C++z13++%5Cend%7Bpmatrix%7D+%3D%5Cbegin%7Bpmatrix%7D++w_%7B11%7D%5E%7B1%7D+%26+w_%7B21%7D%5E%7B1%7D+%5C%5C++w_%7B12%7D%5E%7B1%7D+%26+w_%7B22%7D%5E%7B1%7D+%5C%5C++w_%7B13%7D%5E%7B1%7D+%26+w_%7B23%7D%5E%7B1%7D++%5Cend%7Bpmatrix%7D+%2A+%5Cbegin%7Bpmatrix%7D++x1%5C%5C++x2++%5Cend%7Bpmatrix%7D+%2B+b_%7B1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
And<br>
<img alt="A1= begin{pmatrix}  a11\  a12\  a13  end{pmatrix} = begin{pmatrix}  tanh(z11)\  tanh(z12)\  tanh(z13)  end{pmatrix}" title="A1= begin{pmatrix}  a11\  a12\  a13  end{pmatrix} = begin{pmatrix}  tanh(z11)\  tanh(z12)\  tanh(z13)  end{pmatrix}" src="https://s0.wp.com/latex.php?latex=A1%3D+%5Cbegin%7Bpmatrix%7D++a11%5C%5C++a12%5C%5C++a13++%5Cend%7Bpmatrix%7D+%3D+%5Cbegin%7Bpmatrix%7D++tanh%28z11%29%5C%5C++tanh%28z12%29%5C%5C++tanh%28z13%29++%5Cend%7Bpmatrix%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>Similarly<br>
<img alt="Z2= z_{21}  = begin{pmatrix}  w_{11}^{2} &amp; w_{21}^{2} &amp; w_{31}^{2}  end{pmatrix} *begin{pmatrix}  z_{11}\  z_{12}\  z_{13}  end{pmatrix} +b_{2}" title="Z2= z_{21}  = begin{pmatrix}  w_{11}^{2} &amp; w_{21}^{2} &amp; w_{31}^{2}  end{pmatrix} *begin{pmatrix}  z_{11}\  z_{12}\  z_{13}  end{pmatrix} +b_{2}" src="https://s0.wp.com/latex.php?latex=Z2%3D+z_%7B21%7D++%3D+%5Cbegin%7Bpmatrix%7D++w_%7B11%7D%5E%7B2%7D+%26+w_%7B21%7D%5E%7B2%7D+%26+w_%7B31%7D%5E%7B2%7D++%5Cend%7Bpmatrix%7D+%2A%5Cbegin%7Bpmatrix%7D++z_%7B11%7D%5C%5C++z_%7B12%7D%5C%5C++z_%7B13%7D++%5Cend%7Bpmatrix%7D+%2Bb_%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
and <img alt="A2 = a_{21} = sigmoid(z_{21})" title="A2 = a_{21} = sigmoid(z_{21})" src="https://s0.wp.com/latex.php?latex=A2+%3D+a_%7B21%7D+%3D+sigmoid%28z_%7B21%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>These equations can be written as<br>
<img alt="Z1 = W1 * X + b1" title="Z1 = W1 * X + b1" src="https://s0.wp.com/latex.php?latex=Z1+%3D+W1+%2A+X+%2B+b1&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="A1 = tanh(Z1)" title="A1 = tanh(Z1)" src="https://s0.wp.com/latex.php?latex=A1+%3D+tanh%28Z1%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="Z2 = W2 * A1 + b2" title="Z2 = W2 * A1 + b2" src="https://s0.wp.com/latex.php?latex=Z2+%3D+W2+%2A+A1+%2B+b2&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="A2 = sigmoid(Z2)" title="A2 = sigmoid(Z2)" src="https://s0.wp.com/latex.php?latex=A2+%3D+sigmoid%28Z2%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p><strong>I) Some important results</strong> (a memory refresher!)<br>
<img alt="d/dx(e^{x}) = e^{x}" title="d/dx(e^{x}) = e^{x}" src="https://s0.wp.com/latex.php?latex=d%2Fdx%28e%5E%7Bx%7D%29+%3D+e%5E%7Bx%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> and <img alt="d/dx(e^{-x}) = -e^{-x}" title="d/dx(e^{-x}) = -e^{-x}" src="https://s0.wp.com/latex.php?latex=d%2Fdx%28e%5E%7B-x%7D%29+%3D+-e%5E%7B-x%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> -(a) and<br>
<img alt="sinhx = (e^{x} - e^{-x})/2" title="sinhx = (e^{x} - e^{-x})/2" src="https://s0.wp.com/latex.php?latex=sinhx+%3D+%28e%5E%7Bx%7D+-+e%5E%7B-x%7D%29%2F2&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> and <img alt="coshx = (e^{x} + e^{-x})/2" title="coshx = (e^{x} + e^{-x})/2" src="https://s0.wp.com/latex.php?latex=coshx+%3D+%28e%5E%7Bx%7D+%2B+e%5E%7B-x%7D%29%2F2&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Using (a) we can shown that <img alt="d/dx(sinhx) = coshx" title="d/dx(sinhx) = coshx" src="https://s0.wp.com/latex.php?latex=d%2Fdx%28sinhx%29+%3D+coshx&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> and <img alt="d/dx(coshx) = sinhx" title="d/dx(coshx) = sinhx" src="https://s0.wp.com/latex.php?latex=d%2Fdx%28coshx%29+%3D+sinhx&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> (b)<br>
Now <img alt="d/dx(f(x)/g(x)) = (g(x)*d/dx(f(x)) - f(x)*d/dx(g(x)))/g(x)^{2}" title="d/dx(f(x)/g(x)) = (g(x)*d/dx(f(x)) - f(x)*d/dx(g(x)))/g(x)^{2}" src="https://s0.wp.com/latex.php?latex=d%2Fdx%28f%28x%29%2Fg%28x%29%29+%3D+%28g%28x%29%2Ad%2Fdx%28f%28x%29%29+-+f%28x%29%2Ad%2Fdx%28g%28x%29%29%29%2Fg%28x%29%5E%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> -(c)</p>
<p>Since <img alt="tanhx =z= sinhx/coshx" title="tanhx =z= sinhx/coshx" src="https://s0.wp.com/latex.php?latex=tanhx+%3Dz%3D+sinhx%2Fcoshx&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> and using (b) we get<br>
<img alt="tanhx = (coshx*d/dx(sinhx) - coshx*d/dx(sinhx))/(1-sinhx^{2})" title="tanhx = (coshx*d/dx(sinhx) - coshx*d/dx(sinhx))/(1-sinhx^{2})" src="https://s0.wp.com/latex.php?latex=tanhx+%3D+%28coshx%2Ad%2Fdx%28sinhx%29+-+coshx%2Ad%2Fdx%28sinhx%29%29%2F%281-sinhx%5E%7B2%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Using the values of the derivatives of sinhx and coshx from (b) above we get<br>
<img alt="d/dx(tanhx) = (coshx^{2} - sinhx{2})/coshx{2} = 1 - tanhx^{2}" title="d/dx(tanhx) = (coshx^{2} - sinhx{2})/coshx{2} = 1 - tanhx^{2}" src="https://s0.wp.com/latex.php?latex=d%2Fdx%28tanhx%29+%3D+%28coshx%5E%7B2%7D+-+sinhx%7B2%7D%29%2Fcoshx%7B2%7D+%3D+1+-+tanhx%5E%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Since <img alt="tanhx =z" title="tanhx =z" src="https://s0.wp.com/latex.php?latex=tanhx+%3Dz&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="d/dx(tanhx) = 1 - tanhx^{2}= 1 - z^{2}" title="d/dx(tanhx) = 1 - tanhx^{2}= 1 - z^{2}" src="https://s0.wp.com/latex.php?latex=d%2Fdx%28tanhx%29+%3D+1+-+tanhx%5E%7B2%7D%3D+1+-+z%5E%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> -(d)</p>
<p><strong>II) Derivatives</strong><br>
<img alt="L=-(Ylog(A2) + (1-Y)log(1-A2))" title="L=-(Ylog(A2) + (1-Y)log(1-A2))" src="https://s0.wp.com/latex.php?latex=L%3D-%28Ylog%28A2%29+%2B+%281-Y%29log%281-A2%29%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="dL/dA2 = -(Y/A2 + (1-Y)/(1-A2))" title="dL/dA2 = -(Y/A2 + (1-Y)/(1-A2))" src="https://s0.wp.com/latex.php?latex=dL%2FdA2+%3D+-%28Y%2FA2+%2B+%281-Y%29%2F%281-A2%29%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Since <img alt="A2 = sigmoid(Z2)" title="A2 = sigmoid(Z2)" src="https://s0.wp.com/latex.php?latex=A2+%3D+sigmoid%28Z2%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> therefore <img alt="dA2/dZ2 = A2(1-A2)" title="dA2/dZ2 = A2(1-A2)" src="https://s0.wp.com/latex.php?latex=dA2%2FdZ2+%3D+A2%281-A2%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> see <a rel="nofollow" href="https://gigadom.wordpress.com/2018/01/04/deep-learning-from-basic-principles-in-python-r-and-octave-part-1/" target="_blank">Part1</a><br>
<img alt="Z2 = W2A1 +b2" title="Z2 = W2A1 +b2" src="https://s0.wp.com/latex.php?latex=Z2+%3D+W2A1+%2Bb2&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="dZ2/dW2 = A1" title="dZ2/dW2 = A1" src="https://s0.wp.com/latex.php?latex=dZ2%2FdW2+%3D+A1&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="dZ2/db2 = 1" title="dZ2/db2 = 1" src="https://s0.wp.com/latex.php?latex=dZ2%2Fdb2+%3D+1&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="A1 = tanh(Z1)" title="A1 = tanh(Z1)" src="https://s0.wp.com/latex.php?latex=A1+%3D+tanh%28Z1%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> and <img alt="dA1/dZ1 = 1 - A1^{2}" title="dA1/dZ1 = 1 - A1^{2}" src="https://s0.wp.com/latex.php?latex=dA1%2FdZ1+%3D+1+-+A1%5E%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="Z1 = W1X + b1" title="Z1 = W1X + b1" src="https://s0.wp.com/latex.php?latex=Z1+%3D+W1X+%2B+b1&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="dZ1/dW1 = X" title="dZ1/dW1 = X" src="https://s0.wp.com/latex.php?latex=dZ1%2FdW1+%3D+X&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="dZ1/db1 = 1" title="dZ1/db1 = 1" src="https://s0.wp.com/latex.php?latex=dZ1%2Fdb1+%3D+1&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p><strong>III) Back propagation</strong><br>
Using the derivatives from II) we can derive the following results using Chain Rule<br>
<img alt="partial L/partial Z2 = partial L/partial A2 * partial A2/partial Z2 " title="partial L/partial Z2 = partial L/partial A2 * partial A2/partial Z2 " src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+Z2+%3D+%5Cpartial+L%2F%5Cpartial+A2+%2A+%5Cpartial+A2%2F%5Cpartial+Z2+&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="= -(Y/A2 + (1-Y)/(1-A2)) * A2(1-A2) = A2 - Y" title="= -(Y/A2 + (1-Y)/(1-A2)) * A2(1-A2) = A2 - Y" src="https://s0.wp.com/latex.php?latex=%3D+-%28Y%2FA2+%2B+%281-Y%29%2F%281-A2%29%29+%2A+A2%281-A2%29+%3D+A2+-+Y&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="partial L/partial W2 = partial L/partial A2 * partial A2/partial Z2 * partial Z2/partial W2" title="partial L/partial W2 = partial L/partial A2 * partial A2/partial Z2 * partial Z2/partial W2" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+W2+%3D+%5Cpartial+L%2F%5Cpartial+A2+%2A+%5Cpartial+A2%2F%5Cpartial+Z2+%2A+%5Cpartial+Z2%2F%5Cpartial+W2&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="= (A2-Y) *A1" title="= (A2-Y) *A1" src="https://s0.wp.com/latex.php?latex=%3D+%28A2-Y%29+%2AA1&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> -(A)<br>
<img alt="partial L/partial b2 = partial L/partial A2 * partial A2/partial Z2 * partial Z2/partial b2 = (A2-Y)" title="partial L/partial b2 = partial L/partial A2 * partial A2/partial Z2 * partial Z2/partial b2 = (A2-Y)" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+b2+%3D+%5Cpartial+L%2F%5Cpartial+A2+%2A+%5Cpartial+A2%2F%5Cpartial+Z2+%2A+%5Cpartial+Z2%2F%5Cpartial+b2+%3D+%28A2-Y%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> -(B)</p>
<p><img alt="partial L/partial Z1 = partial L/partial A2 * partial A2/partial Z2 * partial Z2/partial A1 *partial A1/partial Z1 = (A2-Y) * W2 * (1-A1^{2})" title="partial L/partial Z1 = partial L/partial A2 * partial A2/partial Z2 * partial Z2/partial A1 *partial A1/partial Z1 = (A2-Y) * W2 * (1-A1^{2})" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+Z1+%3D+%5Cpartial+L%2F%5Cpartial+A2+%2A+%5Cpartial+A2%2F%5Cpartial+Z2+%2A+%5Cpartial+Z2%2F%5Cpartial+A1+%2A%5Cpartial+A1%2F%5Cpartial+Z1+%3D+%28A2-Y%29+%2A+W2+%2A+%281-A1%5E%7B2%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="partial L/partial W1 = partial L/partial A2 * partial A2/partial Z2 * partial Z2/partial A1 *partial A1/partial Z1 *partial Z1/partial W1 " title="partial L/partial W1 = partial L/partial A2 * partial A2/partial Z2 * partial Z2/partial A1 *partial A1/partial Z1 *partial Z1/partial W1 " src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+W1+%3D+%5Cpartial+L%2F%5Cpartial+A2+%2A+%5Cpartial+A2%2F%5Cpartial+Z2+%2A+%5Cpartial+Z2%2F%5Cpartial+A1+%2A%5Cpartial+A1%2F%5Cpartial+Z1+%2A%5Cpartial+Z1%2F%5Cpartial+W1+&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="=(A2-Y) * W2 * (1-A1^{2}) * X" title="=(A2-Y) * W2 * (1-A1^{2}) * X" src="https://s0.wp.com/latex.php?latex=%3D%28A2-Y%29+%2A+W2+%2A+%281-A1%5E%7B2%7D%29+%2A+X&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> -(C)<br>
<img alt="partial L/partial b1 = partial L/partial A2 * partial A2/partial Z2 * partial Z2/partial A1 *dA1/dZ1 *dZ1/db1" title="partial L/partial b1 = partial L/partial A2 * partial A2/partial Z2 * partial Z2/partial A1 *dA1/dZ1 *dZ1/db1" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+b1+%3D+%5Cpartial+L%2F%5Cpartial+A2+%2A+%5Cpartial+A2%2F%5Cpartial+Z2+%2A+%5Cpartial+Z2%2F%5Cpartial+A1+%2AdA1%2FdZ1+%2AdZ1%2Fdb1&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="= (A2-Y) * W2 * (1-A1^{2})" title="= (A2-Y) * W2 * (1-A1^{2})" src="https://s0.wp.com/latex.php?latex=%3D+%28A2-Y%29+%2A+W2+%2A+%281-A1%5E%7B2%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> -(D)</p>
<p><strong>IV) Gradient Descent</strong><br>
The key computations in the backward cycle are<br>
<img alt="W1 = W1-learningRate * partial L/partial W1" title="W1 = W1-learningRate * partial L/partial W1" src="https://s0.wp.com/latex.php?latex=W1+%3D+W1-learningRate+%2A+%5Cpartial+L%2F%5Cpartial+W1&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> – From (C)<br>
<img alt="b1 = b1-learningRate * partial L/partial b1" title="b1 = b1-learningRate * partial L/partial b1" src="https://s0.wp.com/latex.php?latex=b1+%3D+b1-learningRate+%2A+%5Cpartial+L%2F%5Cpartial+b1&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> – From (D)<br>
<img alt="W2 = W2-learningRate * partial L/partial W2" title="W2 = W2-learningRate * partial L/partial W2" src="https://s0.wp.com/latex.php?latex=W2+%3D+W2-learningRate+%2A+%5Cpartial+L%2F%5Cpartial+W2&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> – From (A)<br>
<img alt="b2 = b2-learningRate * partial L/partial b2" title="b2 = b2-learningRate * partial L/partial b2" src="https://s0.wp.com/latex.php?latex=b2+%3D+b2-learningRate+%2A+%5Cpartial+L%2F%5Cpartial+b2&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> – From (B)</p>
<p>The weights and biases (W1,b1,W2,b2) are updated for each iteration thus minimizing the loss/cost.</p>
<p>These derivations can be represented pictorially using the computation graph (from the book Deep Learning by Ian Goodfellow, Joshua Bengio and Aaron Courville)<br>
<img data-orig-file="https://gigadom.files.wordpress.com/2018/01/dl-1.png" data-orig-size="1280,720" data-medium-file="https://gigadom.files.wordpress.com/2018/01/dl-1.png?w=300" data-attachment-id="6184" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/dl-1.png?w=450&h=380&fit=676%2C380" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/dl-1/" data-large-file="https://gigadom.files.wordpress.com/2018/01/dl-1.png?w=450&amp;h=380&amp;fit=676%2C380" alt="" data-image-title="dl.1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6184" data-recalc-dims="1"></p>
<div>
<h3>3. Manually create a data set that is not lineary separable</h3>
<p>Initially I create a dataset with 2 classes which has around 9 clusters that cannot be separated by linear boundaries. <strong>Note</strong>: <em>This data set is saved as data.csv and is used for the R and Octave Neural networks to see how they perform on the same dataset.</em></p>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> matplotlib.colors
<span>import</span> sklearn.linear_model

<span>from</span> sklearn.model_selection <span>import</span> train_test_split
<span>from</span> sklearn.datasets <span>import</span> make_classification, make_blobs
<span>from</span> matplotlib.colors <span>import</span> ListedColormap
<span>import</span> sklearn
<span>import</span> sklearn.datasets


colors=[<span>'black'</span>,<span>'gold'</span>]
cmap = matplotlib.colors.ListedColormap(colors)
X, y = make_blobs(n_samples = <span>400</span>, n_features = <span>2</span>, centers = <span>7</span>,
                       cluster_std = <span>1.3</span>, random_state = <span>4</span>)
<span>#Create 2 classes</span>
y=y.reshape(<span>400</span>,<span>1</span>)
y = y % <span>2
</span>#Plot the figure
plt.figure()
plt.title(<span>'Non-linearly separable classes'</span>)
plt.scatter(X[:,<span>0</span>], X[:,<span>1</span>], c=y,
           marker= <span>'o'</span>, s=<span>50</span>,cmap=cmap)
plt.savefig(<span>'fig1.png'</span>, bbox_inches=<span>'tight'</span>)</code></pre>
</div>
<div>
<h2><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig11.png?w=456" data-orig-size="558,435" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig11.png?w=456?w=300" data-attachment-id="6187" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig11.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/fig1-13/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig11.png?w=456?w=558" alt="" data-image-title="fig1" sizes="(max-width: 558px) 100vw, 558px" data-comments-opened="1" class="wp-image-6187" data-recalc-dims="1"></h2>
<h3>4. Logistic Regression</h3>
<p>On the above created dataset, classification with logistic regression is performed, and the decision boundary is plotted. It can be seen that logistic regression performs quite poorly</p>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> matplotlib.colors
<span>import</span> sklearn.linear_model

<span>from</span> sklearn.model_selection <span>import</span> train_test_split
<span>from</span> sklearn.datasets <span>import</span> make_classification, make_blobs
<span>from</span> matplotlib.colors <span>import</span> ListedColormap
<span>import</span> sklearn
<span>import</span> sklearn.datasets

<span>#from DLfunctions import plot_decision_boundary</span>
execfile(<span>&quot;./DLfunctions.py&quot;</span>) <span># Since import does not work in Rmd!!!</span>

colors=[<span>'black'</span>,<span>'gold'</span>]
cmap = matplotlib.colors.ListedColormap(colors)
X, y = make_blobs(n_samples = <span>400</span>, n_features = <span>2</span>, centers = <span>7</span>,
                       cluster_std = <span>1.3</span>, random_state = <span>4</span>)
<span>#Create 2 classes</span>
y=y.reshape(<span>400</span>,<span>1</span>)
y = y % <span>2</span>

<span># Train the logistic regression classifier</span>
clf = sklearn.linear_model.LogisticRegressionCV();
clf.fit(X, y);

<span># Plot the decision boundary for logistic regression</span>
plot_decision_boundary_n(<span>lambda</span> x: clf.predict(x), X.T, y.T,<span>&quot;fig2.png&quot;</span>)
</code><code></code></pre>
</div>
<div>
<h2><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig2.png?w=456" data-orig-size="578,455" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig2.png?w=456?w=300" data-attachment-id="6188" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig2.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/fig2-11/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig2.png?w=456?w=578" alt="" data-image-title="fig2" sizes="(max-width: 578px) 100vw, 578px" data-comments-opened="1" class="wp-image-6188" data-recalc-dims="1"></h2>
<h3>5. The 3 layer Neural Network in Python (vectorized)</h3>
<p>The vectorized implementation is included below. Note that in the case of Python a learning rate of 0.5 and 3 hidden units performs very well.</p>
<pre><code><span>## Random data set with 9 clusters</span>
<span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> sklearn.linear_model
<span>import</span> pandas <span>as</span> pd

<span>from</span> sklearn.datasets <span>import</span> make_classification, make_blobs
execfile(<span>&quot;./DLfunctions.py&quot;</span>) <span># Since import does not work in Rmd!!!</span>

X1, Y1 = make_blobs(n_samples = <span>400</span>, n_features = <span>2</span>, centers = <span>9</span>,
                       cluster_std = <span>1.3</span>, random_state = <span>4</span>)
<span>#Create 2 classes</span>
Y1=Y1.reshape(<span>400</span>,<span>1</span>)
Y1 = Y1 % <span>2</span>
X2=X1.T
Y2=Y1.T

#Perform gradient descent
parameters,costs = computeNN(X2, Y2, numHidden = <span>4</span>, learningRate=<span>0.5</span>, numIterations = <span>10000</span>)
plot_decision_boundary(<span>lambda</span> x: predict(parameters, x.T), X2, Y2,str(<span>4</span>),str(<span>0.5</span>),<span>&quot;fig3.png&quot;</span>)</code></pre>
<pre><code>## Cost after iteration 0: 0.692669
## Cost after iteration 1000: 0.246650
## Cost after iteration 2000: 0.227801
## Cost after iteration 3000: 0.226809
## Cost after iteration 4000: 0.226518
## Cost after iteration 5000: 0.226331
## Cost after iteration 6000: 0.226194
## Cost after iteration 7000: 0.226085
## Cost after iteration 8000: 0.225994
## Cost after iteration 9000: 0.225915</code></pre>
</div>
<div>
<h2> <img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig31.png?w=456" data-orig-size="590,455" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig31.png?w=456?w=300" data-attachment-id="6191" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig31.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/fig3-10/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig31.png?w=456?w=590" alt="" data-image-title="fig3" sizes="(max-width: 590px) 100vw, 590px" data-comments-opened="1" class="wp-image-6191" data-recalc-dims="1"></h2>
<h3>6. The 3 layer Neural Network in R (vectorized)</h3>
<p>For this the dataset created by Python is saved  to see how R performs on the same dataset. The vectorized implementation of a Neural Network was just a little more interesting as R does not have a similar package like ‘numpy’. While numpy handles broadcasting implicitly, in R I had to use the ‘sweep’ command to broadcast. The implementaion is included below. Note that since the initialization with random weights is slightly different, R performs best with a learning rate of 0.1 and with 6 hidden units</p>
<pre><code><span>source</span><span>(</span><span>&quot;DLfunctions2_1.R&quot;</span><span>)</span></code></pre>
<pre><code><span>z</span> <span>&lt;-</span> <span>as.matrix</span><span>(</span><span>read.csv</span><span>(</span><span>&quot;data.csv&quot;</span>,<span>header</span><span>=</span><span>FALSE</span><span>)</span><span>)</span> <span># </span>
<span>x</span> <span>&lt;-</span> <span>z</span><span>[</span>,<span>1</span><span>:</span><span>2</span><span>]</span>
<span>y</span> <span>&lt;-</span> <span>z</span><span>[</span>,<span>3</span><span>]</span>
<span>x1</span> <span>&lt;-</span> <span>t</span><span>(</span><span>x</span><span>)</span>
<span>y1</span> <span>&lt;-</span> <span>t</span><span>(</span><span>y</span><span>)
</span>#Perform gradient descent
<span>nn</span> <span>&lt;-</span><span>computeNN</span><span>(</span><span>x1</span>, <span>y1</span>, <span>6</span>, <span>learningRate</span><span>=</span><span>0.1</span>,<span>numIterations</span><span>=</span><span>10000</span><span>)</span> <span># Good</span></code></pre>
<pre><code>## [1] 0.7075341
## [1] 0.2606695
## [1] 0.2198039
## [1] 0.2091238
## [1] 0.211146
## [1] 0.2108461
## [1] 0.2105351
## [1] 0.210211
## [1] 0.2099104
## [1] 0.2096437
## [1] 0.209409</code></pre>
<pre><code><span>plotDecisionBoundary</span><span>(</span><span>z</span>,<span>nn</span>,<span>6</span>,<span>0.1</span><span>)</span></code></pre>
<p><img> <img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig1-1.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig1-1.png?w=300" data-attachment-id="6192" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig1-1.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/fig1-1-6/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig1-1.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig1-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6192" data-recalc-dims="1"></p>
<h3> 7.  The 3 layer Neural Network in Octave (vectorized)</h3>
<p>This uses the same dataset that was generated using Python code.<br>
<code>source(&quot;DL-function2.m&quot;)<br>
data=csvread(&quot;data.csv&quot;);<br>
X=data(:,1:2);<br>
Y=data(:,3);<br>
# Make sure that the model parameters are correct. Take the transpose of X &amp; Y</code><br>
<code>#Perform gradient descent<br>
[W1,b1,W2,b2,costs]= computeNN(X', Y',4, learningRate=0.5, numIterations = 10000);</code></p>
</div>
<div>
<h2><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/figa.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/01/figa.png?w=300" data-attachment-id="6194" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/figa.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/figa/" data-large-file="https://gigadom.files.wordpress.com/2018/01/figa.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="figa" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6194" data-recalc-dims="1"></h2>
<h3>8a. Performance  for different learning rates (Python)</h3>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> sklearn.linear_model
<span>import</span> pandas <span>as</span> pd

<span>from</span> sklearn.datasets <span>import</span> make_classification, make_blobs
execfile(<span>&quot;./DLfunctions.py&quot;</span>) <span># Since import does not work in Rmd!!!</span>
# Create data
X1, Y1 = make_blobs(n_samples = <span>400</span>, n_features = <span>2</span>, centers = <span>9</span>,
                       cluster_std = <span>1.3</span>, random_state = <span>4</span>)
<span>#Create 2 classes</span>
Y1=Y1.reshape(<span>400</span>,<span>1</span>)
Y1 = Y1 % <span>2</span>
X2=X1.T
Y2=Y1.T
# Create a list of learning rates
learningRate=[<span>0.5</span>,<span>1.2</span>,<span>3.0</span>]
df=pd.DataFrame()
#Compute costs for each learning rate
<span>for</span> lr <span>in</span> learningRate:
   parameters,costs = computeNN(X2, Y2, numHidden = <span>4</span>, learningRate=lr, numIterations = <span>10000</span>)
   <span>print</span>(costs)
   df1=pd.DataFrame(costs)
   df=pd.concat([df,df1],axis=<span>1</span>)
<span>#Set the iterations</span>
iterations=[<span>0</span>,<span>1000</span>,<span>2000</span>,<span>3000</span>,<span>4000</span>,<span>5000</span>,<span>6000</span>,<span>7000</span>,<span>8000</span>,<span>9000</span>]   
#Create data frame
<span>#Set index</span>
df1=df.set_index([iterations])
df1.columns=[<span>0.5</span>,<span>1.2</span>,<span>3.0</span>]
fig=df1.plot()
fig=plt.title(<span>&quot;Cost vs No of Iterations for different learning rates&quot;</span>)
plt.savefig(<span>'fig4.png'</span>, bbox_inches=<span>'tight'</span>)</code></pre>
</div>
<div>
<h2><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig4.png?w=456" data-orig-size="565,435" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig4.png?w=456?w=300" data-attachment-id="6195" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig4.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/fig4-6/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig4.png?w=456?w=565" alt="" data-image-title="fig4" sizes="(max-width: 565px) 100vw, 565px" data-comments-opened="1" class="wp-image-6195" data-recalc-dims="1"></h2>
<h3>8b. Performance  for different hidden units (Python)</h3>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> sklearn.linear_model
<span>import</span> pandas <span>as</span> pd

<span>from</span> sklearn.datasets <span>import</span> make_classification, make_blobs
execfile(<span>&quot;./DLfunctions.py&quot;</span>) <span># Since import does not work in Rmd!!!</span>
#Create data set
X1, Y1 = make_blobs(n_samples = <span>400</span>, n_features = <span>2</span>, centers = <span>9</span>,
                       cluster_std = <span>1.3</span>, random_state = <span>4</span>)
<span>#Create 2 classes</span>
Y1=Y1.reshape(<span>400</span>,<span>1</span>)
Y1 = Y1 % <span>2</span>
X2=X1.T
Y2=Y1.T
# Make a list of hidden unis
numHidden=[<span>3</span>,<span>5</span>,<span>7</span>]
df=pd.DataFrame()
#Compute costs for different hidden units
<span>for</span> numHid <span>in</span> numHidden:
   parameters,costs = computeNN(X2, Y2, numHidden = numHid, learningRate=<span>1.2</span>, numIterations = <span>10000</span>)
   <span>print</span>(costs)
   df1=pd.DataFrame(costs)
   df=pd.concat([df,df1],axis=<span>1</span>)
<span>#Set the iterations</span>
iterations=[<span>0</span>,<span>1000</span>,<span>2000</span>,<span>3000</span>,<span>4000</span>,<span>5000</span>,<span>6000</span>,<span>7000</span>,<span>8000</span>,<span>9000</span>]   
<span>#Set index</span>
df1=df.set_index([iterations])
df1.columns=[<span>3</span>,<span>5</span>,<span>7</span>]
#Plot
fig=df1.plot()
fig=plt.title(<span>&quot;Cost vs No of Iterations for different no of hidden units&quot;</span>)
plt.savefig(<span>'fig5.png'</span>, bbox_inches=<span>'tight'</span>)</code></pre>
</div>
<div>
<h2><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig5.png?w=456" data-orig-size="565,435" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig5.png?w=456?w=300" data-attachment-id="6196" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig5.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/fig5-5/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig5.png?w=456?w=565" alt="" data-image-title="fig5" sizes="(max-width: 565px) 100vw, 565px" data-comments-opened="1" class="wp-image-6196" data-recalc-dims="1"></h2>
<h3>9a. Performance  for different learning rates (R)</h3>
<pre><code><span>source</span><span>(</span><span>&quot;DLfunctions2_1.R&quot;</span><span>)
</span># Read data
<span>z</span> <span>&lt;-</span> <span>as.matrix</span><span>(</span><span>read.csv</span><span>(</span><span>&quot;data.csv&quot;</span>,<span>header</span><span>=</span><span>FALSE</span><span>)</span><span>)</span> <span># </span>
<span>x</span> <span>&lt;-</span> <span>z</span><span>[</span>,<span>1</span><span>:</span><span>2</span><span>]</span>
<span>y</span> <span>&lt;-</span> <span>z</span><span>[</span>,<span>3</span><span>]</span>
<span>x1</span> <span>&lt;-</span> <span>t</span><span>(</span><span>x</span><span>)</span>
<span>y1</span> <span>&lt;-</span> <span>t</span><span>(</span><span>y</span><span>)
</span>#Loop through learning rates and compute costs
<span>learningRate</span> <span>&lt;-</span><span>c</span><span>(</span><span>0.1</span>,<span>1.2</span>,<span>3.0</span><span>)</span>
<span>df</span> <span>&lt;-</span> <span>NULL</span>
<span>for</span><span>(</span><span>i</span> <span>in</span> <span>seq_along</span><span>(</span><span>learningRate</span><span>)</span><span>)</span><span>{</span>
   <span>nn</span> <span>&lt;-</span>  <span>computeNN</span><span>(</span><span>x1</span>, <span>y1</span>, <span>6</span>, <span>learningRate</span><span>=</span><span>learningRate</span><span>[</span><span>i</span><span>]</span>,<span>numIterations</span><span>=</span><span>10000</span><span>)</span> 
   <span>cost</span> <span>&lt;-</span> <span>nn</span><span>$</span><span>costs</span>
   <span>df</span> <span>&lt;-</span> <span>cbind</span><span>(</span><span>df</span>,<span>cost</span><span>)</span>
  
<span>}</span>      </code></pre>
<pre><code><span>
#Create dataframe
df</span> <span>&lt;-</span> <span>data.frame</span><span>(</span><span>df</span><span>)</span> 
<span>iterations</span><span>=</span><span>seq</span><span>(</span><span>0</span>,<span>10000</span>,<span>by</span><span>=</span><span>1000</span><span>)</span>
<span>df</span> <span>&lt;-</span> <span>cbind</span><span>(</span><span>iterations</span>,<span>df</span><span>)</span>
<span>names</span><span>(</span><span>df</span><span>)</span> <span>&lt;-</span> <span>c</span><span>(</span><span>&quot;iterations&quot;</span>,<span>&quot;0.5&quot;</span>,<span>&quot;1.2&quot;</span>,<span>&quot;3.0&quot;</span><span>)</span>
<span>library</span><span>(</span><span>reshape2</span><span>)</span></code></pre>
<pre><code><span>df1</span> <span>&lt;-</span> <span>melt</span><span>(</span><span>df</span>,<span>id</span><span>=</span><span>&quot;iterations&quot;</span><span>)</span>  # Melt the data
#Plot  
<span>ggplot</span><span>(</span><span>df1</span><span>)</span> <span>+</span> <span>geom_line</span><span>(</span><span>aes</span><span>(</span><span>x</span><span>=</span><span>iterations</span>,<span>y</span><span>=</span><span>value</span>,<span>colour</span><span>=</span><span>variable</span><span>)</span>,<span>size</span><span>=</span><span>1</span><span>)</span>  <span>+</span> 
    <span>xlab</span><span>(</span><span>&quot;Iterations&quot;</span><span>)</span> <span>+</span>
    <span>ylab</span><span>(</span><span>'Cost'</span><span>)</span> <span>+</span> <span>ggtitle</span><span>(</span><span>&quot;Cost vs No iterations for different learning rates&quot;</span><span>)</span></code></pre>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig2-1.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig2-1.png?w=300" data-attachment-id="6198" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig2-1.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/fig2-1-6/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig2-1.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig2-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6198" data-recalc-dims="1"></p>
</div>
<div>
<h3>9b. Performance  for different hidden units (R)</h3>
<pre><code><span>source</span><span>(</span><span>&quot;DLfunctions2_1.R&quot;</span><span>)</span>
<span># Loop through Num hidden units</span>
<span>numHidden</span> <span>&lt;-</span><span>c</span><span>(</span><span>4</span>,<span>6</span>,<span>9</span><span>)</span>
<span>df</span> <span>&lt;-</span> <span>NULL</span>
<span>for</span><span>(</span><span>i</span> <span>in</span> <span>seq_along</span><span>(</span><span>numHidden</span><span>)</span><span>)</span><span>{</span>
    <span>nn</span> <span>&lt;-</span>  <span>computeNN</span><span>(</span><span>x1</span>, <span>y1</span>, <span>numHidden</span><span>[</span><span>i</span><span>]</span>, <span>learningRate</span><span>=</span><span>0.1</span>,<span>numIterations</span><span>=</span><span>10000</span><span>)</span> 
    <span>cost</span> <span>&lt;-</span> <span>nn</span><span>$</span><span>costs</span>
    <span>df</span> <span>&lt;-</span> <span>cbind</span><span>(</span><span>df</span>,<span>cost</span><span>)</span>
    
<span>}</span>      </code></pre>
<pre><code><span>df</span> <span>&lt;-</span> <span>data.frame</span><span>(</span><span>df</span><span>)</span> 
<span>iterations</span><span>=</span><span>seq</span><span>(</span><span>0</span>,<span>10000</span>,<span>by</span><span>=</span><span>1000</span><span>)</span>
<span>df</span> <span>&lt;-</span> <span>cbind</span><span>(</span><span>iterations</span>,<span>df</span><span>)</span>
<span>names</span><span>(</span><span>df</span><span>)</span> <span>&lt;-</span> <span>c</span><span>(</span><span>&quot;iterations&quot;</span>,<span>&quot;4&quot;</span>,<span>&quot;6&quot;</span>,<span>&quot;9&quot;</span><span>)</span>
<span>library</span><span>(</span><span>reshape2</span><span>)
</span># Melt
<span>df1</span> <span>&lt;-</span> <span>melt</span><span>(</span><span>df</span>,<span>id</span><span>=</span><span>&quot;iterations&quot;</span><span>)</span> 
# Plot   
<span>ggplot</span><span>(</span><span>df1</span><span>)</span> <span>+</span> <span>geom_line</span><span>(</span><span>aes</span><span>(</span><span>x</span><span>=</span><span>iterations</span>,<span>y</span><span>=</span><span>value</span>,<span>colour</span><span>=</span><span>variable</span><span>)</span>,<span>size</span><span>=</span><span>1</span><span>)</span>  <span>+</span> 
    <span>xlab</span><span>(</span><span>&quot;Iterations&quot;</span><span>)</span> <span>+</span>
    <span>ylab</span><span>(</span><span>'Cost'</span><span>)</span> <span>+</span> <span>ggtitle</span><span>(</span><span>&quot;Cost vs No iterations for different number of hidden units&quot;</span><span>)</span></code></pre>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig3-1.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig3-1.png?w=300" data-attachment-id="6199" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig3-1.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/fig3-1-6/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig3-1.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig3-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6199" data-recalc-dims="1"></p>
</div>
<div>
<h2>10a. Performance of the Neural Network for different learning rates (Octave)</h2>
<p><code>source(&quot;DL-function2.m&quot;)<br>
plotLRCostVsIterations()<br>
print -djph figa.jpg</code></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/figb.jpg" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/01/figb.jpg?w=300" data-attachment-id="6200" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/figb.jpg?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/figb/" data-large-file="https://gigadom.files.wordpress.com/2018/01/figb.jpg?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="figb" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6200" data-recalc-dims="1"></p>
</div>
<div>
<h2>10b. Performance of the Neural Network for different number of hidden units (Octave)</h2>
<p><code>source(&quot;DL-function2.m&quot;)<br>
plotHiddenCostVsIterations()<br>
print -djph figa.jpg</code><br>
<img data-orig-file="https://gigadom.files.wordpress.com/2018/01/figd.jpg" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/01/figd.jpg?w=300" data-attachment-id="6202" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/figd.jpg?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/figd/" data-large-file="https://gigadom.files.wordpress.com/2018/01/figd.jpg?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="figd" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6202" data-recalc-dims="1"></p>
</div>
<div>
<h2>11. Turning the heat on the Neural Network</h2>
<p>In this 2nd part I create a a central region of positives and and the outside region as negatives. The points are generated using the equation of a circle (x – a)^{2} + (y -b) ^{2} = R^{2} . How does the 3 layer Neural Network perform on this?  Here’s a look! <strong>Note</strong>: <em>The same dataset is also used for R and Octave Neural Network constructions</em></p>
</div>
<div>
<h2>12. Manually creating a circular central region</h2>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> matplotlib.colors
<span>import</span> sklearn.linear_model

<span>from</span> sklearn.model_selection <span>import</span> train_test_split
<span>from</span> sklearn.datasets <span>import</span> make_classification, make_blobs
<span>from</span> matplotlib.colors <span>import</span> ListedColormap
<span>import</span> sklearn
<span>import</span> sklearn.datasets

colors=[<span>'black'</span>,<span>'gold'</span>]
cmap = matplotlib.colors.ListedColormap(colors)
x1=np.random.uniform(<span>0</span>,<span>10</span>,<span>800</span>).reshape(<span>800</span>,<span>1</span>)
x2=np.random.uniform(<span>0</span>,<span>10</span>,<span>800</span>).reshape(<span>800</span>,<span>1</span>)
X=np.append(x1,x2,axis=<span>1</span>)
X.shape
# Create (x-a)^2 + (y-b)^2 = R^2
<span># Create a subset of values where squared is &lt;0,4. Perform ravel() to flatten this vector</span>
a=(np.power(X[:,<span>0</span>]-<span>5</span>,<span>2</span>) + np.power(X[:,<span>1</span>]-<span>5</span>,<span>2</span>) &lt;= <span>6</span>).ravel()
Y=a.reshape(<span>800</span>,<span>1</span>)

cmap = matplotlib.colors.ListedColormap(colors)

plt.figure()
plt.title(<span>'Non-linearly separable classes'</span>)
plt.scatter(X[:,<span>0</span>], X[:,<span>1</span>], c=Y,
           marker= <span>'o'</span>, s=<span>15</span>,cmap=cmap)
plt.savefig(<span>'fig6.png'</span>, bbox_inches=<span>'tight'</span>)</code></pre>
</div>
<div>
<h2><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig6.png?w=456" data-orig-size="546,435" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig6.png?w=456?w=300" data-attachment-id="6203" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig6.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/fig6-5/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig6.png?w=456?w=546" alt="" data-image-title="fig6" sizes="(max-width: 546px) 100vw, 546px" data-comments-opened="1" class="wp-image-6203" data-recalc-dims="1"></h2>
<h3>13a. Decision boundary with hidden units=4 and learning rate = 2.2 (Python)</h3>
<p>With the above hyper parameters the decision boundary is triangular</p>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> matplotlib.colors
<span>import</span> sklearn.linear_model
execfile(<span>&quot;./DLfunctions.py&quot;</span>)
x1=np.random.uniform(<span>0</span>,<span>10</span>,<span>800</span>).reshape(<span>800</span>,<span>1</span>)
x2=np.random.uniform(<span>0</span>,<span>10</span>,<span>800</span>).reshape(<span>800</span>,<span>1</span>)
X=np.append(x1,x2,axis=<span>1</span>)
X.shape

<span># Create a subset of values where squared is &lt;0,4. Perform ravel() to flatten this vector</span>
a=(np.power(X[:,<span>0</span>]-<span>5</span>,<span>2</span>) + np.power(X[:,<span>1</span>]-<span>5</span>,<span>2</span>) &lt;= <span>6</span>).ravel()
Y=a.reshape(<span>800</span>,<span>1</span>)

X2=X.T
Y2=Y.T

parameters,costs = computeNN(X2, Y2, numHidden = <span>4</span>, learningRate=<span>2.2</span>, numIterations = <span>10000</span>)
plot_decision_boundary(<span>lambda</span> x: predict(parameters, x.T), X2, Y2,str(<span>4</span>),str(<span>2.2</span>),<span>&quot;fig7.png&quot;</span>)
</code></pre>
<pre><code>## Cost after iteration 0: 0.692836
## Cost after iteration 1000: 0.331052
## Cost after iteration 2000: 0.326428
## Cost after iteration 3000: 0.474887
## Cost after iteration 4000: 0.247989
## Cost after iteration 5000: 0.218009
## Cost after iteration 6000: 0.201034
## Cost after iteration 7000: 0.197030
## Cost after iteration 8000: 0.193507
## Cost after iteration 9000: 0.191949</code></pre>
</div>
<div>
<h2><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig7.png?w=456" data-orig-size="576,455" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig7.png?w=456?w=300" data-attachment-id="6204" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig7.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/fig7-8/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig7.png?w=456?w=576" alt="" data-image-title="fig7" sizes="(max-width: 576px) 100vw, 576px" data-comments-opened="1" class="wp-image-6204" data-recalc-dims="1"></h2>
<h3>13b. Decision boundary with hidden units=12 and learning rate = 2.2 (Python)</h3>
<p>With the above hyper parameters the decision boundary is triangular</p>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> matplotlib.colors
<span>import</span> sklearn.linear_model
execfile(<span>&quot;./DLfunctions.py&quot;</span>)
x1=np.random.uniform(<span>0</span>,<span>10</span>,<span>800</span>).reshape(<span>800</span>,<span>1</span>)
x2=np.random.uniform(<span>0</span>,<span>10</span>,<span>800</span>).reshape(<span>800</span>,<span>1</span>)
X=np.append(x1,x2,axis=<span>1</span>)
X.shape

<span># Create a subset of values where squared is &lt;0,4. Perform ravel() to flatten this vector</span>
a=(np.power(X[:,<span>0</span>]-<span>5</span>,<span>2</span>) + np.power(X[:,<span>1</span>]-<span>5</span>,<span>2</span>) &lt;= <span>6</span>).ravel()
Y=a.reshape(<span>800</span>,<span>1</span>)

X2=X.T
Y2=Y.T

parameters,costs = computeNN(X2, Y2, numHidden = <span>12</span>, learningRate=<span>2.2</span>, numIterations = <span>10000</span>)
plot_decision_boundary(<span>lambda</span> x: predict(parameters, x.T), X2, Y2,str(<span>12</span>),str(<span>2.2</span>),<span>&quot;fig8.png&quot;</span>)
</code></pre>
<pre><code>## Cost after iteration 0: 0.693291
## Cost after iteration 1000: 0.383318
## Cost after iteration 2000: 0.298807
## Cost after iteration 3000: 0.251735
## Cost after iteration 4000: 0.177843
## Cost after iteration 5000: 0.130414
## Cost after iteration 6000: 0.152400
## Cost after iteration 7000: 0.065359
## Cost after iteration 8000: 0.050921
## Cost after iteration 9000: 0.039719</code></pre>
</div>
<h2><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig81.png?w=456" data-orig-size="584,455" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig81.png?w=456?w=300" data-attachment-id="6206" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig81.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/fig8-5/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig81.png?w=456?w=584" alt="" data-image-title="fig8" sizes="(max-width: 584px) 100vw, 584px" data-comments-opened="1" class="wp-image-6206" data-recalc-dims="1"></h2>
<div>
<h3>14a. Decision boundary with hidden units=9 and learning rate = 0.5 (R)</h3>
<p>When the number of hidden units is 6 and the learning rate is 0,1, is also a triangular shape in R</p>
<pre><code><span>source</span><span>(</span><span>&quot;DLfunctions2_1.R&quot;</span><span>)</span>
<span>z</span> <span>&lt;-</span> <span>as.matrix</span><span>(</span><span>read.csv</span><span>(</span><span>&quot;data1.csv&quot;</span>,<span>header</span><span>=</span><span>FALSE</span><span>)</span><span>)</span> <span># N</span>
<span>x</span> <span>&lt;-</span> <span>z</span><span>[</span>,<span>1</span><span>:</span><span>2</span><span>]</span>
<span>y</span> <span>&lt;-</span> <span>z</span><span>[</span>,<span>3</span><span>]</span>
<span>x1</span> <span>&lt;-</span> <span>t</span><span>(</span><span>x</span><span>)</span>
<span>y1</span> <span>&lt;-</span> <span>t</span><span>(</span><span>y</span><span>)</span>
<span>nn</span> <span>&lt;-</span><span>computeNN</span><span>(</span><span>x1</span>, <span>y1</span>, <span>9</span>, <span>learningRate</span><span>=</span><span>0.5</span>,<span>numIterations</span><span>=</span><span>10000</span><span>)</span> <span># Triangular</span></code></pre>
<pre><code>## [1] 0.8398838
## [1] 0.3303621
## [1] 0.3127731
## [1] 0.3012791
## [1] 0.3305543
## [1] 0.3303964
## [1] 0.2334615
## [1] 0.1920771
## [1] 0.2341225
## [1] 0.2188118
## [1] 0.2082687</code></pre>
<pre><code><span>plotDecisionBoundary</span><span>(</span><span>z</span>,<span>nn</span>,<span>6</span>,<span>0.1</span><span>)</span></code></pre>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig4-1.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig4-1.png?w=300" data-attachment-id="6207" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig4-1.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/fig4-1-6/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig4-1.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig4-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6207" data-recalc-dims="1"></p>
</div>
<div>
<h3>14b. Decision boundary with hidden units=8 and learning rate = 0.1 (R)</h3>
<pre><code><span>source</span><span>(</span><span>&quot;DLfunctions2_1.R&quot;</span><span>)</span>
<span>z</span> <span>&lt;-</span> <span>as.matrix</span><span>(</span><span>read.csv</span><span>(</span><span>&quot;data1.csv&quot;</span>,<span>header</span><span>=</span><span>FALSE</span><span>)</span><span>)</span> <span># N</span>
<span>x</span> <span>&lt;-</span> <span>z</span><span>[</span>,<span>1</span><span>:</span><span>2</span><span>]</span>
<span>y</span> <span>&lt;-</span> <span>z</span><span>[</span>,<span>3</span><span>]</span>
<span>x1</span> <span>&lt;-</span> <span>t</span><span>(</span><span>x</span><span>)</span>
<span>y1</span> <span>&lt;-</span> <span>t</span><span>(</span><span>y</span><span>)</span>
<span>nn</span> <span>&lt;-</span><span>computeNN</span><span>(</span><span>x1</span>, <span>y1</span>, <span>8</span>, <span>learningRate</span><span>=</span><span>0.1</span>,<span>numIterations</span><span>=</span><span>10000</span><span>)</span> <span># Hemisphere</span></code></pre>
<pre><code>## [1] 0.7273279
## [1] 0.3169335
## [1] 0.2378464
## [1] 0.1688635
## [1] 0.1368466
## [1] 0.120664
## [1] 0.111211
## [1] 0.1043362
## [1] 0.09800573
## [1] 0.09126161
## [1] 0.0840379</code></pre>
<pre><code><span>plotDecisionBoundary</span><span>(</span><span>z</span>,<span>nn</span>,<span>8</span>,<span>0.1</span><span>)</span></code><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig5-1.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig5-1.png?w=300" data-attachment-id="6208" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig5-1.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/fig5-1-4/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig5-1.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig5-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6208" data-recalc-dims="1"></pre>
</div>
<div>
<h3>15a. Decision boundary with hidden units=12 and learning rate = 1.5 (Octave)</h3>
<p><code>source(&quot;DL-function2.m&quot;)<br>
data=csvread(&quot;data1.csv&quot;);<br>
X=data(:,1:2);<br>
Y=data(:,3);<br>
# Make sure that the model parameters are correct. Take the transpose of X &amp; Y<br>
[W1,b1,W2,b2,costs]= computeNN(X', Y',12, learningRate=1.5, numIterations = 10000);<br>
plotDecisionBoundary(data, W1,b1,W2,b2)<br>
print -djpg fige.jpg</code><br>
<img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fige.jpg" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fige.jpg?w=300" data-attachment-id="6210" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fige.jpg?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/fige/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fige.jpg?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="fige" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6210" data-recalc-dims="1"></p>
<p><strong>Conclusion</strong>: This post implemented a 3 layer Neural Network to create non-linear boundaries while performing classification. Clearly the Neural Network performs very well when the number of hidden units and learning rate are varied.</p>
<p>To be continued…<br>
Watch this space!!</p>
<p>References<br>
1. <a rel="nofollow" href="https://www.coursera.org/specializations/deep-learning" target="_blank">Deep Learning Specialization</a><br>
2. <a rel="nofollow" href="https://www.coursera.org/learn/neural-networks" target="_blank">Neural Networks for Machine Learning</a><br>
3. <a rel="nofollow" href="http://www.deeplearningbook.org/" target="_blank">Deep Learning, Ian Goodfellow, Yoshua Bengio and Aaron Courville</a><br>
4. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/01/21/neural-networks-the-mechanics-of-backpropagation/" target="_blank">Neural Networks: The mechanics of backpropagation</a><br>
5. <a rel="nofollow" href="https://www.coursera.org/learn/machine-learning" target="_blank">Machine Learning</a></p>
<p>Also see<br>
1. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/12/05/my-book-practical-machine-learning-with-r-and-python-on-amazon/" target="_blank">My book ‘Practical Machine Learning with R and Python’ on Amazon</a><br>
2. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/01/05/googlyplus-yorkr-analyzes-ipl-players-teams-matches-with-plots-and-tables/" target="_blank">GooglyPlus: yorkr analyzes IPL players, teams, matches with plots and tables</a><br>
3. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/12/16/the-3rd-paperback-edition-of-my-books-on-cricket-on-amazon/" target="_blank">The 3rd paperback &amp; kindle editions of my books on Cricket, now on Amazon</a><br>
4. <a rel="nofollow" href="https://gigadom.wordpress.com/2016/06/05/exploring-quantum-gate-operations-with-qcsimulator/" target="_blank">Exploring Quantum Gate operations with QCSimulator</a><br>
5. <a rel="nofollow" href="https://gigadom.wordpress.com/2013/06/11/simulating-a-web-joint-in-android/" target="_blank">Simulating a Web Joint in Android</a><br>
6. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/09/11/my-travels-through-the-realms-of-data-science-machine-learning-deep-learning-and-ai/" target="_blank">My travels through the realms of Data Science, Machine Learning, Deep Learning and (AI)</a><br>
7. <a rel="nofollow" href="https://gigadom.wordpress.com/2013/07/24/presentation-on-wireless-technologies-part-1/" target="_blank">Presentation on Wireless Technologies – Part 1</a></p>
<p>To see all posts check <a rel="nofollow" href="https://gigadom.wordpress.com/aa-2/" target="_blank">Index of posts</a></p>
</div>
<p> <a rel="nofollow" href="https://feeds.wordpress.com/1.0/gocomments/gigadom.wordpress.com/6158/" target="_blank"><img border="0" alt="" src="https://feeds.wordpress.com/1.0/comments/gigadom.wordpress.com/6158/"></a> <img border="0" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" alt="" width="1" data-recalc-dims="1" height="1"></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/"> R – Giga thoughts …</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/YTVBznyLuQw" height="1"></a></div>